Quantcast
Channel: Active questions tagged python - Stack Overflow
Viewing all articles
Browse latest Browse all 23131

Starting or resuming async method/coroutine without awaiting it in Python

$
0
0

In JavaScript (and other languages) an async function is immediately executed upon its invocation until the first await is encountered.This is not the case in Python. From the asyncio documentation:

Note that simply calling a coroutine will not schedule it to be executed:

I find this very confusing, because my mental model of asynchronous programming is like this:

  • Setup/Initialization/Queuing of asynchronous task as soon as possible (usually quick and nonblocking)
  • Asynchronous task happens in background
  • Do other stuff
  • Waiting for task to complete as late as possible. Maybe its already completed, so no waiting occurs.
| S | AAAAAAAAAAAAAAAAAAAAA | W |        | S | AAAAAAAAAAAAA | W |           | S | AAAAAAAAAA | WWWWWW |

To reduce waiting time the Setup should be executed as soon as possible.But if the method is async in Python the "Setup" part is not executed by calling the method at all. Only if await (or gather, etc.) is used is it actually started. That makes it very hard to start a bunch of coroutines and wait for all of them or batches to finish, because if you use asyncio.gather they are all started at the very end and at the same time, which can waste a lot of time.

One way to execute the setup part immediately without awaiting the method is this:

async def work():    print("Setup...")    await asyncio.sleep(1)async def main():    task = asyncio.create_task(work())    await asyncio.sleep(0)    # do something else    await task

But that is quite cumbersome (and ugly) and also puts me at the whim of the event loop to actually choose the newly created task and not just swap back to the main method or something entirely different.

I managed to come up with this method decorator that immediately executes an async method by using send(None) and wraps the async method into a normal method that returns an Awaitable:

def js_like_await(coro):    class JSLikeCoroutine:        def __init__(self, first_return, coro_obj):            self.first_return = first_return            self.coro_obj = coro_obj        def __await__(self):            try:                yield self.first_return                while True:                    yield self.coro_obj.send(None)            except StopIteration as e:                return e.value    def coroutine_with_js_like_await(*args, **kwargs):        coro_obj = coro(*args, **kwargs)        first_return = None        try:            first_return = coro_obj.send(None)            return JSLikeCoroutine(first_return, coro_obj)        except StopIteration as e:            result = asyncio.Future()            result.set_result(e.value)            return result    return coroutine_with_js_like_await

But for this to have an effect it would also have to wrap possible inner asynchronous function calls as well, which might not be modifiable for outside callers of the method.

I thought I could make this a lot simpler if there was a way to "attempt" a coroutine to start or resume it.So switching control flow to the coroutine (similar to what send() does), but only if the async function is currently not waiting for something. Returning immediately, if it does.

So I want a method attempt that can be used like this to control the execution flow:

def attempt(task):    while not task.is_waiting:        task.continue_to_next_await()async def work():    print("Setup")    await asyncio.sleep(10)    print("After first sleep")    await asyncio.sleep(10)    print("Done")    return "Result"task = work()  # Nothing is printed, because the coroutine is only created.attempt(task)  # "Setup" is printed and method returns immediatelyattempt(task)  # Nothing happens, because task is still sleepingtime.sleep(15) # Block CPU to wait for sleeping to finishattempt(task)  # "After first sleep" is printed, because the sleep time is overattempt(task)  # Nothing happens, because task is still sleepingtime.sleep(15) # Block CPU to wait for second sleeping to finishattempt(task)  # "Done" is printed.attempt(task)  # Nothing happens, because task is finishedr = await task # Get result.

Is there a way to achieve something like this?

I couldn't get send(None) to work multiple times, because after the second send it crashes with await wasn't used with future if i discard the Futures that are returned from send(None):

async def work():    print("Setup")    await asyncio.sleep(10)    print("Done")async def main():    task = work()    _ = task.send(None) # Returns <Future pending> of type <class '_asyncio.Future'>    _ = task.send(None) # RuntimeError: await wasn't used with future

I feel like I have to somehow give the returned Future back to the event loop, but I don't know how. If I make attempt an __await__-object that just yields the Future from send and await it, it will wait for the sleep and the attempt becomes blocking.

Note, that I'm aware of the obvious solution of "not make your method async". But sadly, it is too late, to make everyone adhere to that.


Viewing all articles
Browse latest Browse all 23131

Trending Articles