Asynchronous Python: What You Need to Know
The Development Process of Python Coroutines and an In-depth Analysis of Old and New Coroutines 1. The Historical Evolution of Python Coroutines Throughout the long development of Python, the implementation of coroutines has undergone several significant changes. Understanding these changes helps us better grasp the essence of Python asynchronous programming. 1.1 Early Exploration and Introduction of Basic Functions Python 2.5: This version introduced the .send(), .throw(), and .close() methods to generators. The appearance of these methods made generators more than just simple iterators; they began to possess more complex interaction capabilities, laying a certain foundation for the development of coroutines. For example, the .send() method can send data into the generator, breaking the previous limitation that generators could only output unidirectionally. Python 3.3: The yield from syntax was introduced, which was an important milestone. It enabled generators to receive return values and allowed coroutines to be defined directly using yield from. This feature simplified the writing of coroutines and enhanced the readability and maintainability of the code. For instance, with yield from, a complex generator operation can be conveniently delegated to another generator, achieving code reuse and logical separation. 1.2 Standard Library Support and Syntax Refinement Python 3.4: The asyncio module was added, which was a crucial step for Python towards modern asynchronous programming. asyncio provides an event-loop-based asynchronous programming framework, enabling developers to write highly efficient asynchronous code more conveniently. It offers a powerful infrastructure for running coroutines, including core functions such as event loops and task management. Python 3.5: The async and await keywords were added, providing more direct and clearer support for asynchronous programming at the syntax level. The async keyword is used to define an asynchronous function, indicating that the function is a coroutine; the await keyword is used to pause the execution of a coroutine and wait for the completion of an asynchronous operation. This syntactic sugar makes the writing style of asynchronous code closer to that of synchronous code, greatly reducing the threshold for asynchronous programming. 1.3 Maturity and Optimization Stages Python 3.7: The way of defining coroutines using async def + await was formally established. This method is more concise and straightforward and has become the standard way to define coroutines in Python. It further strengthened the syntactic structure of asynchronous programming, enabling developers to write asynchronous code more naturally. Python 3.10: Defining coroutines with yield from was removed, marking a new stage in the development of Python coroutines. It became more focused on the new coroutine system based on async and await, reducing the confusion caused by different implementation methods. 1.4 Concepts and Impacts of Old and New Coroutines Old coroutines are implemented based on generator syntax such as yield and yield from. New coroutines, on the other hand, are based on keywords like asyncio, async, and await. In the history of coroutine development, the two implementation methods had an intersection period. However, the generator-based syntax of old coroutines made it easy to confuse the concepts of generators and coroutines, causing some difficulties for learners. Therefore, deeply understanding the differences between the two implementation methods of coroutines is crucial for mastering Python asynchronous programming. 2. Review of Old Coroutines 2.1 Core Mechanism: The Magic of the yield Keyword The core of old coroutines lies in the yield keyword, which endows functions with powerful capabilities, including pausing and resuming code execution, alternating execution between functions, and transferring CPU resources. 2.2 Analysis of Code Examples import time def consume(): r = '' while True: n = yield r print(f'[consumer] Starting to consume {n}...') time.sleep(1) r = f'{n} consumed' def produce(c): next(c) n = 0 while n
The Development Process of Python Coroutines and an In-depth Analysis of Old and New Coroutines
1. The Historical Evolution of Python Coroutines
Throughout the long development of Python, the implementation of coroutines has undergone several significant changes. Understanding these changes helps us better grasp the essence of Python asynchronous programming.
1.1 Early Exploration and Introduction of Basic Functions
-
Python 2.5: This version introduced the
.send()
,.throw()
, and.close()
methods to generators. The appearance of these methods made generators more than just simple iterators; they began to possess more complex interaction capabilities, laying a certain foundation for the development of coroutines. For example, the.send()
method can send data into the generator, breaking the previous limitation that generators could only output unidirectionally. -
Python 3.3: The
yield from
syntax was introduced, which was an important milestone. It enabled generators to receive return values and allowed coroutines to be defined directly usingyield from
. This feature simplified the writing of coroutines and enhanced the readability and maintainability of the code. For instance, withyield from
, a complex generator operation can be conveniently delegated to another generator, achieving code reuse and logical separation.
1.2 Standard Library Support and Syntax Refinement
-
Python 3.4: The
asyncio
module was added, which was a crucial step for Python towards modern asynchronous programming.asyncio
provides an event-loop-based asynchronous programming framework, enabling developers to write highly efficient asynchronous code more conveniently. It offers a powerful infrastructure for running coroutines, including core functions such as event loops and task management. -
Python 3.5: The
async
andawait
keywords were added, providing more direct and clearer support for asynchronous programming at the syntax level. Theasync
keyword is used to define an asynchronous function, indicating that the function is a coroutine; theawait
keyword is used to pause the execution of a coroutine and wait for the completion of an asynchronous operation. This syntactic sugar makes the writing style of asynchronous code closer to that of synchronous code, greatly reducing the threshold for asynchronous programming.
1.3 Maturity and Optimization Stages
-
Python 3.7: The way of defining coroutines using
async def + await
was formally established. This method is more concise and straightforward and has become the standard way to define coroutines in Python. It further strengthened the syntactic structure of asynchronous programming, enabling developers to write asynchronous code more naturally. -
Python 3.10: Defining coroutines with
yield from
was removed, marking a new stage in the development of Python coroutines. It became more focused on the new coroutine system based onasync
andawait
, reducing the confusion caused by different implementation methods.
1.4 Concepts and Impacts of Old and New Coroutines
Old coroutines are implemented based on generator syntax such as yield
and yield from
. New coroutines, on the other hand, are based on keywords like asyncio
, async
, and await
. In the history of coroutine development, the two implementation methods had an intersection period. However, the generator-based syntax of old coroutines made it easy to confuse the concepts of generators and coroutines, causing some difficulties for learners. Therefore, deeply understanding the differences between the two implementation methods of coroutines is crucial for mastering Python asynchronous programming.
2. Review of Old Coroutines
2.1 Core Mechanism: The Magic of the yield
Keyword
The core of old coroutines lies in the yield
keyword, which endows functions with powerful capabilities, including pausing and resuming code execution, alternating execution between functions, and transferring CPU resources.
2.2 Analysis of Code Examples
import time
def consume():
r = ''
while True:
n = yield r
print(f'[consumer] Starting to consume {n}...')
time.sleep(1)
r = f'{n} consumed'
def produce(c):
next(c)
n = 0
while n < 5:
n = n + 1
print(f'[producer] Produced {n}...')
r = c.send(n)
print(f'[producer] Consumer return: {r}')
c.close()
if __name__ == '__main__':
c = consume()
produce(c)
In this example, the consume
function is a consumer coroutine, and the produce
function is a producer function. The while True
loop in the consume
function allows it to keep running. The line n = yield r
is crucial. When the execution reaches this line, the execution flow of the consume
function pauses, returns the value of r
to the caller (i.e., the produce
function), and saves the current state of the function.
The produce
function starts the consume
coroutine via next(c)
and then enters its own while
loop. In each loop, it produces a piece of data (n
) and sends the data to the consume
coroutine through c.send(n)
. c.send(n)
not only sends data to the consume
coroutine but also resumes the execution of the consume
coroutine, making it continue from the line n = yield r
.
2.3 Execution Results and Analysis
Execution results:
[producer] Produced 1...
[consumer] Starting to consume 1...
[producer] Consumer return: 1 consumed
[producer] Produced 2...
[consumer] Starting to consume 2...
[producer] Consumer return: 2 consumed
[producer] Produced 3...
[consumer] Starting to consume 3...
[producer] Consumer return: 3 consumed
[producer] Produced 4...
[consumer] Starting to consume 4...
[producer] Consumer return: 4 consumed
[producer] Produced 5...
[consumer] Starting to consume 5...
[producer] Consumer return: 5 consumed
When the consumer consume
executes to n = yield r
, the process pauses and returns the CPU to the caller produce
. In this example, the while
loops in consume
and produce
work together to simulate a simple event loop function. And the yield
and send
methods implement the pausing and resuming of tasks.
To summarize the implementation of old coroutines:
-
Event loop: Implemented by manually writing
while
loop code. Although this method is relatively basic, it gives developers a deeper understanding of the principle of the event loop. -
Code pausing and resuming: Achieved with the help of the characteristics of the
yield
generator.yield
can not only pause the function execution but also save the function state, enabling the function to continue from where it was paused when resumed.
3. Review of New Coroutines
3.1 A Powerful System Based on the Event Loop
New coroutines are implemented based on keywords such as asyncio
, async
, and await
, with the event loop mechanism at its core. This mechanism provides more powerful and efficient asynchronous programming capabilities, including event loops, task management, and callback mechanisms.
3.2 Analysis of the Functions of Key Components
- asyncio: Provides an event loop, which is the foundation for running new coroutines. The event loop is responsible for managing all asynchronous tasks, scheduling their execution, and ensuring that each task is executed at the appropriate time.
-
async: Used to mark a function as a coroutine function. When a function is defined as
async def
, it becomes a coroutine, and theawait
keyword can be used to handle asynchronous operations. -
await: Provides the ability to suspend the process. In a coroutine function, when
await
is executed, the execution of the current coroutine pauses, waits for the completion of the asynchronous operation followingawait
, and then resumes execution.
3.3 Detailed Explanation of Code Examples
import asyncio
async def coro1():
print("start coro1")
await asyncio.sleep(2)
print("end coro1")
async def coro2():
print("start coro2")
await asyncio.sleep(1)
print("end coro2")
# Create an event loop
loop = asyncio.get_event_loop()
# Create tasks
task1 = loop.create_task(coro1())
task2 = loop.create_task(coro2())
# Run coroutines
loop.run_until_complete(asyncio.gather(task1, task2))
# Close the event loop
loop.close()
In this example, two coroutine functions coro1
and coro2
are defined. After printing "start coro1", the coro1
function pauses for 2 seconds via await asyncio.sleep(2)
. Here, await
suspends the execution of coro1
and returns the CPU to the event loop. The event loop will schedule other executable tasks, such as coro2
, within these 2 seconds. After printing "start coro2", the coro2
function pauses for 1 second via await asyncio.sleep(1)
and then prints "end coro2". When coro2
is paused, if there are no other executable tasks in the event loop, it will wait for the pause time of coro2
to end and continue to execute the remaining code of coro2
. When both coro1
and coro2
are executed, the event loop ends.
3.4 Execution Results and Analysis
Results:
start coro1
start coro2
end coro2
end coro1
When coro1
executes to await asyncio.sleep(2)
, the process is suspended, and the CPU is returned to the event loop, waiting for the next scheduling of the event loop. At this time, the event loop schedules coro2
to continue execution.
To summarize the implementation of new coroutines:
-
Event loop: Achieved through the
loop
provided byasyncio
, which is more efficient and flexible and can manage a large number of asynchronous tasks. -
Program suspension: Achieved through the
await
keyword, making the asynchronous operations of coroutines more intuitive and easier to understand.
4. Comparison of Old and New Coroutine Implementations
4.1 Differences in Implementation Mechanisms
-
yield: It is a keyword for generator (Generator) functions. When a function contains a
yield
statement, it returns a generator object. The generator object can be iterated step by step to obtain the values in the generator function by calling thenext()
method or using afor
loop. Throughyield
, a function can be divided into multiple code blocks, and the execution can be switched between these blocks, thus achieving the pausing and resuming of function execution. -
asyncio: It is a standard library provided by Python for writing asynchronous code. It is based on the event loop (Event Loop) pattern, allowing multiple concurrent tasks to be processed in a single thread.
asyncio
uses theasync
andawait
keywords to define coroutine functions. In a coroutine function, theawait
keyword is used to pause the execution of the current coroutine, wait for the completion of an asynchronous operation, and then resume execution.
4.2 Summary of Differences
-
Old coroutines: Mainly achieve coroutines through the ability of the
yield
keyword to pause and resume execution. Its advantage is that, for developers familiar with generators, it is easy to understand based on the generator syntax; its disadvantages are that it is easy to confuse with the generator concept, and the way of manually writing the event loop is not flexible and efficient enough. -
New coroutines: Achieve coroutines through the event loop mechanism combined with the ability of the
await
keyword to suspend the process. Its advantages are that it provides more powerful and flexible asynchronous programming capabilities, the code structure is clearer, and it better meets the needs of modern asynchronous programming; its disadvantage is that for beginners, the concepts of the event loop and asynchronous programming may be relatively abstract and require some time to understand and master.
5. The Relationship between await
and yield
5.1 Similarities
-
Control flow pause and resume: Both
await
andyield
have the ability to pause the code execution at a certain point and continue it at a later time. This characteristic plays a crucial role in asynchronous programming and generator programming. - Coroutine support: Both are closely related to coroutines (Coroutine). They can be used to define and manage coroutines, making the writing of asynchronous code simpler and more readable. Whether it is old or new coroutines, they rely on these two keywords to achieve the core functions of coroutines.
5.2 Differences
-
Syntax differences: The
await
keyword was introduced in Python 3.5 and is specifically used to pause execution in an asynchronous function, waiting for the completion of an asynchronous operation. Theyield
keyword is for early coroutines and is mainly used in generator (Generator) functions to create iterators and implement lazy evaluation. Early coroutines were realized through the capabilities of generators. -
Semantics:
-
await
means that the current coroutine needs to wait for the completion of an asynchronous operation and suspend execution, giving other tasks a chance to execute. It emphasizes waiting for the result of an asynchronous operation and is a waiting mechanism in asynchronous programming. -
yield
hands over the control of execution to the caller while saving the state of the function so that it can resume execution from the paused position in the next iteration. It focuses more on the control and state preservation of function execution. -
await
suspends the program and lets the event loop schedule new tasks;yield
suspends the program and waits for the next instruction from the caller.
-
-
Context:
await
must be used in an asynchronous context, such as in an asynchronous function or in anasync with
block. Whileyield
can be used in an ordinary function, as long as the function is defined as a generator function, even without the context of using coroutines. -
Return values:
yield
returns a generator object, and the values in the generator can be iterated step by step by calling thenext()
method or using afor
loop. Theawait
keyword returns an awaitable object (Awaitable), which can be aFuture
,Task
,Coroutine
, etc., representing the result or state of an asynchronous operation.
5.3 Summary
await
does not implement program pausing and execution through yield
. Although they have similar capabilities, they have no calling relationship at all and are both Python keywords. await
is suitable for asynchronous programming scenarios, used to wait for the completion of asynchronous operations, and supports more flexible coroutine management; while yield
is mainly used in generator functions to implement iterators and lazy evaluation. There are some differences in their application scenarios and syntax, but both provide the ability to pause and resume the control flow.
By reviewing the development process of Python coroutines, comparing the implementation methods of old and new coroutines, and analyzing the relationship between await
and yield
in depth, we have a more comprehensive and in-depth understanding of the principles of Python coroutines. Although these contents are somewhat difficult to understand, mastering them will lay a solid foundation for our exploration in the field of Python asynchronous programming.
Leapcell: The Best Serverless Platform for Python app Hosting
Finally, let me introduce Leapcell, the most suitable platform for deploying Python applications.
1. Multi-Language Support
- Develop with JavaScript, Python, Go, or Rust.
2. Deploy unlimited projects for free
- pay only for usage — no requests, no charges.
3. Unbeatable Cost Efficiency
- Pay-as-you-go with no idle charges.
- Example: $25 supports 6.94M requests at a 60ms average response time.
4. Streamlined Developer Experience
- Intuitive UI for effortless setup.
- Fully automated CI/CD pipelines and GitOps integration.
- Real-time metrics and logging for actionable insights.
5. Effortless Scalability and High Performance
- Auto-scaling to handle high concurrency with ease.
- Zero operational overhead — just focus on building.
Explore more in the documentation!
Leapcell Twitter: https://x.com/LeapcellHQ