Django Async DEP

classic Classic list List threaded Threaded
39 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Django Async DEP

Andrew Godwin-3
Hello everyone,

While the ASGI patch (https://github.com/django/django/pull/11209) is maybe the first small step in a long road to async, it's the only real one we could do without a DEP as it purely pulls on existing specs and bugs.

To that end, I have drafted a DEP (provisionally #0009), "Async-capable Django". It is a summary - and I use that word very lightly - of the rest of the work to be done to make a version of Django that one could call "async capable".


It is approximately 7,000 words long - while it could probably do with some editing, the subject matter means it has to be quite in-depth. Even then, it does not propose an exact plan to follow; instead, it proposes the overall strategy and the high-level ideas about what needs to be done.

If you are short on time, please read the Foreword and the High-Level Summary, and then jump around using the table of contents to the sections that interest you.

Feedback on this is very much appreciated; you can either reply here or, if you have comments that would benefit from precise location on the diff, use the pull request here: https://github.com/django/deps/pull/56

This is a very complex topic, and I appreciate feedback might take a while; at least a couple of weeks to get everyone's thoughts. If you want a more private channel to discuss concerns or design questions you're not happy to talk about publicly, feel free to email me directly.

This proposal does not quite fit the DEP template - we never really planned for something of this scale - but I have done my best to make it work; I think we still benefit from discussing and voting on it in the DEP format, but suggestions on how to make it work better are welcome, provided they fit within how Django operates as a community now.

Finally, if there is positive feedback on this, we have the separate question of funding it that I will discuss separately, if and when we get that far. The DEP covers some of the topics, but it's notable that the implementation speed of this varies considerably; if we could get, say, me and someone else on it full-time (which is not a cheap or likely prospect), we can probably be done inside of a single release. If we rely just on volunteer time, it may take years. There is hopefully somewhere between those two that works.

Thanks for taking the time to read through!

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1uqav3JXQA3m%2Bajf0Bd15QQX3JTZTfmH3Hc3ECWyRR3CVg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

Patryk Zawadzki-2
Hey Andrew,

Great work on the DEP, the task at hand is humongous.

Do you think it's worth it to try and make the ORM async? It contains tons of magic that is inherently incompatible with explicit I/O that is required for async/await to work, things like silently fetching relations on first access, silently fetching deferred fields, queryset laziness etc. Those need to be explicitly awaited now and having to await regular field access for the sake of making deferred fields work is (I believe) not worth it.

Conveniently, all of these magic properties are things that silently break performance of applications leading to the n+1 queries problem so I'm more than happy to get rid of them ;)

Seeing that we can't get rid of anything in the old synchronous ORM, one idea would be to just depend on a separate, asyncio-based ORM and configure it to use the same tables and relations that the old ORM uses. It could work like model forms or formset factories, give it a "classic" model class, a list of overrides (we can't hope to automatically match old field types to anything understood by another engine) and receive an asyncio-capable table. There's an obvious candidate that Tom Christie is working on.

We could even improve the querying syntax to use the SQL Alchemy's filtering operators with the hope of eventually reaching full static type safety[2] (one can only dream).

The new ORM would only be usable in new async views and other features could be implemented gradually. You don't need to async forms (but with Tom Christie's typesystem[3] you don't really need async forms) and async template loading to achieve great things.

You can already use things like requests-async[4] to access external resources, you can use libraries like Ariadne[5] to implement a GraphQL API with WebSocket-based subscriptions without having any of those (disclaimer: I work on Ariadne). It's okay if some things can't be ported initially or if certain features are only available in their sync flavor. The community will pave the way and provide many missing pieces before we get to implementing them. It's okay if new async counterparts have different APIs if these APIs are either better or are a closer match to how async/await works.



PS: If Django provided abstractions for common event loop operations such as "await this collections of futures" and "ensure this task is executed even if not explicitly awaited" then I think it may be possible to avoid explicitly depending on asyncio.

All the best

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/da0d4c65-562b-44cc-abe0-e5c31e28d91d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

J. Pic
Hi Patryk,

I'm not sure but for me the "What is Django" section answers the question. For me Django is full of philosophy that seeds a great ecosystem of apps of all sorts with a growing user base nonetheless, and a bunch of brilliant hackers to look up to and inspire for more. Of course if you're into fixing a particular technical issue such as "I want to make an efficient GraphQL server", then even Django works even if it's not necessarily the most fit for X reason. But for "I want to get a web page going with some forms and a database and see where that goes" then Django is definitely enjoyable, and at this point in time where tons of more sophisticated frameworks are born it's clear that Django is still in the game and will be for the next 10 years. So, why not also get more features out of it while we've not been abandoned by all the talented contributors ? It's not because we like Django that we cant haz nice things ;)

Great move Andrew and very exciting, keep it up and best luck to you !

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAC6Op19DRYB%2BtUJX-9fG_9EOeo0oqGZayn5SHHBdec9qweq8ng%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

J. Pic
Oops too fast, if it's possible to split the DEP and delay the ORM as advised by Patrick it could make it a lot easier to distribute the work, I don't know really sry. Have a great day !

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAC6Op1-tDvM-raHfOLGCQYFFk7Aoz2CAa%3DY_bXcaj30Y887wzw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

Patryk Zawadzki-2
In reply to this post by J. Pic
I'm not sure but for me the "What is Django" section answers the question. For me Django is full of philosophy that seeds a great ecosystem of apps of all sorts with a growing user base nonetheless, and a bunch of brilliant hackers to look up to and inspire for more. Of course if you're into fixing a particular technical issue such as "I want to make an efficient GraphQL server", then even Django works even if it's not necessarily the most fit for X reason. But for "I want to get a web page going with some forms and a database and see where that goes" then Django is definitely enjoyable, and at this point in time where tons of more sophisticated frameworks are born it's clear that Django is still in the game and will be for the next 10 years. So, why not also get more features out of it while we've not been abandoned by all the talented contributors ? It's not because we like Django that we cant haz nice things ;)

I'm arguing the opposite, that by limiting the scope of the MVP to asynchronous views (even if middleware stays synchronous for now) and using an existing ORM we can make Django a viable solution for many new applications. And some of them don't need the full stack.

I'd go as far as arguing that fewer and fewer applications benefit from Django providing all of the pieces (as many new applications only implement an API and render in the client) so treating them all as blockers here may not be beneficial.

Best regards

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/314d7b71-1bf1-441b-ae49-e4fbeb5e1891%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

J. Pic
In one project I really enjoyed using channels just as a background worker instead of celery, and the DEP does not talk much in the Email section: use the async variant, can be tackled separately, low priority.

For now the ORM works, but if sending an email fails (ie. SMTP down) it will raise an exception that can end up in a 500 error, it would be great if Django could retry so that emails leave when its SMTP is up again, would that maybe weight in priority ?

Also, you would probably like to view the queue in the admin, maybe make an unmanaged Model for tasks. So, as simple as Email would look like, I think efforts in it could benefit in a much wider way too.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAC6Op18rkxnz04hVciJs_pHEvmfG2ir_bMafPaYQ7d9HGTJhoQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

Andrew Godwin-3
In reply to this post by Patryk Zawadzki-2
I would agree with both of you - I think the most important thing is to get the view layer async-capable, as that then lets sites use any manner of asynchronous libraries that already exist to get experiments and unique things going. Tom Christie, for example, has already started work on an asynchronous ORM. Some of Django's biggest sites don't use the ORM - for example, Instagram.

That said, I also think it's important to allow the ORM to support both modes in the long term. I truly believe the best way to be able to write async code is to _have the choice to write it_, rather than being made to all the time; if we make people use a separate, async ORM, then we force them to write every view asynchronously, with all the extra danger and thinking that requires. It's much better for Django to do the hard work, and say "hey, if you want to write asynchronously or synchronously, that's fine - it takes literally zero extra effort to go either way".

This is why I propose in the DEP that we do the view layer first, and then move onto the ORM as a second wave.

Andrew

On Thu, May 9, 2019 at 12:29 PM Patryk Zawadzki <[hidden email]> wrote:
I'm not sure but for me the "What is Django" section answers the question. For me Django is full of philosophy that seeds a great ecosystem of apps of all sorts with a growing user base nonetheless, and a bunch of brilliant hackers to look up to and inspire for more. Of course if you're into fixing a particular technical issue such as "I want to make an efficient GraphQL server", then even Django works even if it's not necessarily the most fit for X reason. But for "I want to get a web page going with some forms and a database and see where that goes" then Django is definitely enjoyable, and at this point in time where tons of more sophisticated frameworks are born it's clear that Django is still in the game and will be for the next 10 years. So, why not also get more features out of it while we've not been abandoned by all the talented contributors ? It's not because we like Django that we cant haz nice things ;)

I'm arguing the opposite, that by limiting the scope of the MVP to asynchronous views (even if middleware stays synchronous for now) and using an existing ORM we can make Django a viable solution for many new applications. And some of them don't need the full stack.

I'd go as far as arguing that fewer and fewer applications benefit from Django providing all of the pieces (as many new applications only implement an API and render in the client) so treating them all as blockers here may not be beneficial.

Best regards

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/314d7b71-1bf1-441b-ae49-e4fbeb5e1891%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1urQVr%3DZ55Xqn7E%2B5-AeZU-egJAZ_%2BooAp_3nSSYnbNX7w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

Patryk Zawadzki-2
That said, I also think it's important to allow the ORM to support both modes in the long term. I truly believe the best way to be able to write async code is to _have the choice to write it_, rather than being made to all the time; if we make people use a separate, async ORM, then we force them to write every view asynchronously, with all the extra danger and thinking that requires. It's much better for Django to do the hard work, and say "hey, if you want to write asynchronously or synchronously, that's fine - it takes literally zero extra effort to go either way".

Slightly off-topic but once we have an async ORM, making it synchronous is not impossible (I believe either Channels or Daphne already have shims that use a worker thread to spin the event loop until a future is fulfilled).

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/402dbe77-44fc-485b-b058-e619a1576ddb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

Aymeric Augustin
In reply to this post by Andrew Godwin-3
Hello Andrew,

Thanks for your work putting together this plan. Within our constraints, it's a good plan.

Regarding templating, I would say it isn't a priority because a developer who knows how to parallelize I/O bound operations will prefer (or at least accept) to perform these operations in the view, not in the template.

I'm on the fence about the convention for async APIs. I'm not super excited by spraying async code with _async prefixes. The namespacing approach would allow for cleaner async code. Most Python modules should be either sync or async, not a mix of both styles. But I might be underestimating the importance of explicitness...

I don't have much else to say on the DEP. It makes a lot of sense. I'm happy to see this happening!

Cheers,

-- 
Aymeric.



On 9 May 2019, at 18:10, Andrew Godwin <[hidden email]> wrote:

Hello everyone,

While the ASGI patch (https://github.com/django/django/pull/11209) is maybe the first small step in a long road to async, it's the only real one we could do without a DEP as it purely pulls on existing specs and bugs.

To that end, I have drafted a DEP (provisionally #0009), "Async-capable Django". It is a summary - and I use that word very lightly - of the rest of the work to be done to make a version of Django that one could call "async capable".


It is approximately 7,000 words long - while it could probably do with some editing, the subject matter means it has to be quite in-depth. Even then, it does not propose an exact plan to follow; instead, it proposes the overall strategy and the high-level ideas about what needs to be done.

If you are short on time, please read the Foreword and the High-Level Summary, and then jump around using the table of contents to the sections that interest you.

Feedback on this is very much appreciated; you can either reply here or, if you have comments that would benefit from precise location on the diff, use the pull request here: https://github.com/django/deps/pull/56

This is a very complex topic, and I appreciate feedback might take a while; at least a couple of weeks to get everyone's thoughts. If you want a more private channel to discuss concerns or design questions you're not happy to talk about publicly, feel free to email me directly.

This proposal does not quite fit the DEP template - we never really planned for something of this scale - but I have done my best to make it work; I think we still benefit from discussing and voting on it in the DEP format, but suggestions on how to make it work better are welcome, provided they fit within how Django operates as a community now.

Finally, if there is positive feedback on this, we have the separate question of funding it that I will discuss separately, if and when we get that far. The DEP covers some of the topics, but it's notable that the implementation speed of this varies considerably; if we could get, say, me and someone else on it full-time (which is not a cheap or likely prospect), we can probably be done inside of a single release. If we rely just on volunteer time, it may take years. There is hopefully somewhere between those two that works.

Thanks for taking the time to read through!

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1uqav3JXQA3m%2Bajf0Bd15QQX3JTZTfmH3Hc3ECWyRR3CVg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/4B8D50C5-7550-45CE-974F-F0A0B8A1E6C7%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

Andrew Godwin-3
In reply to this post by Patryk Zawadzki-2
> Slightly off-topic but once we have an async ORM, making it synchronous is not impossible (I believe either Channels or Daphne already have shims that use a worker thread to spin the event loop until a future is fulfilled).

This is in fact the proposed way to make the ORM work with both sync and async worlds once the interior is fully-async.

> In one project I really enjoyed using channels just as a background worker instead of celery, and the DEP does not talk much in the Email section: use the async variant, can be tackled separately, low priority.
> For now the ORM works, but if sending an email fails (ie. SMTP down) it will raise an exception that can end up in a 500 error, it would be great if Django could retry so that emails leave when its SMTP is up again, would that maybe weight in priority ?

If you want guaranteed email delivery, that's a task for something like Celery or a third-party API; any method of sending emails in the background in the same process, be it threads or async coroutines, is going to be unreliable as the server may die any time. I don't think it's sensible for Django to try and solve this problem internally, at least not as part of this async push.

Andrew

On Thu, May 9, 2019 at 12:56 PM Patryk Zawadzki <[hidden email]> wrote:
That said, I also think it's important to allow the ORM to support both modes in the long term. I truly believe the best way to be able to write async code is to _have the choice to write it_, rather than being made to all the time; if we make people use a separate, async ORM, then we force them to write every view asynchronously, with all the extra danger and thinking that requires. It's much better for Django to do the hard work, and say "hey, if you want to write asynchronously or synchronously, that's fine - it takes literally zero extra effort to go either way".

Slightly off-topic but once we have an async ORM, making it synchronous is not impossible (I believe either Channels or Daphne already have shims that use a worker thread to spin the event loop until a future is fulfilled).

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/402dbe77-44fc-485b-b058-e619a1576ddb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1urOnhEJDb6Sm45U59iF5hCZmN%3Dw-J%2B0er4UCnqh3qsmCw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

Andrew Godwin-3
In reply to this post by Aymeric Augustin


On Thu, May 9, 2019 at 1:04 PM Aymeric Augustin <[hidden email]> wrote:
Hello Andrew,

Thanks for your work putting together this plan. Within our constraints, it's a good plan.

Regarding templating, I would say it isn't a priority because a developer who knows how to parallelize I/O bound operations will prefer (or at least accept) to perform these operations in the view, not in the template.

I'm on the fence about the convention for async APIs. I'm not super excited by spraying async code with _async prefixes. The namespacing approach would allow for cleaner async code. Most Python modules should be either sync or async, not a mix of both styles. But I might be underestimating the importance of explicitness...


I am also not a fan of the approach, but I did err towards being explicit. Jannis suggested what I think might be a nicer approach on Twitter, which is to add an async "proxy object" to access methods with, e.g.:

cache.get("foo")
cache.async.get("foo")

This is still explicit but looks less ugly, and gives us a way in future to make some modules transition to async-by-default (by e.g. supplying cache.sync.get). What do you think?

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1uowvL3FH9BR80hpOoBdRw-qUwpu5bA%3DBDLXmhY0WzBM%2Bg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

J. Pic
In reply to this post by Andrew Godwin-3
Hi Andrew,

On Thu, May 9, 2019 at 10:04 PM Andrew Godwin <[hidden email]> wrote:

If you want guaranteed email delivery, that's a task for something like Celery or a third-party API; any method of sending emails in the background in the same process, be it threads or async coroutines, is going to be unreliable as the server may die any time. I don't think it's sensible for Django to try and solve this problem internally, at least not as part of this async push.

I'm a bit confused here, what benefit are you getting from async emails if you're already retrying emails in the background in production ?

Thanks for your kind reply,

Have a great day

--

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAC6Op18AA3Q7wguoZn%3D%2BfqHTAquXU0cESxF2i0GYzp%2BjB_ft%3Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

Patryk Zawadzki-2
In reply to this post by Andrew Godwin-3
I am also not a fan of the approach, but I did err towards being explicit. Jannis suggested what I think might be a nicer approach on Twitter, which is to add an async "proxy object" to access methods with, e.g.:

cache.get("foo")
cache.async.get("foo")

This is still explicit but looks less ugly, and gives us a way in future to make some modules transition to async-by-default (by e.g. supplying cache.sync.get). What do you think?

I think I'd personally prefer to have to do `from django.asyncore.cache import cache` and if the synchronous cache raised an exception when attempting to use it with an event loop active. Like Aymeric, I don't see myself mixing async and sync code within one module.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/d2c879cb-fad7-4e56-a1d9-1a31d5134e96%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

Aymeric Augustin
In reply to this post by Andrew Godwin-3
> On 9 May 2019, at 22:06, Andrew Godwin <[hidden email]> wrote:
>
> Jannis suggested what I think might be a nicer approach on Twitter, which is to add an async "proxy object" to access methods with, e.g.:
>
> cache.get("foo")
> cache.async.get("foo")
>
> This is still explicit but looks less ugly, and gives us a way in future to make some modules transition to async-by-default (by e.g. supplying cache.sync.get). What do you think?


I don't think it makes a significant difference from a readability perspective in this example. It does have some advantages, though:

- It could be easier to implement a sync version based on the async one, or vice-versa, if each one has its own class. It will probably be more DRY.
- It could also be a bit more usable by developers, especially in IDEs

Also it could provide an alternative solution for the async attribute access problem: `model_instance.async.related_field` could be a Future that, when awaited, resolves to the related field. So you could `await model_instance.async.related_field` instead of having to `select_related`. To be honest I didn't investigate all the consequences of this idea. It seemed worth mentioning, though, even if it turns out to be impractical :-)

We've always considered that implicit queries on attribute access were an intractable problem. I said it on stage an DjangoCon US 2013. I'm now wondering if I was wrong all along! In an async ORM context, every time we traverse a relation, we could create a Future that would execute the query and resolve to its result. This would require one await per SQL query so we'd still get the benefit of making queries explicit, although to a lesser extent than with an explicit select/prefetch_related.

--
Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/258CBCB2-2456-439F-82DF-E8760AE5EEF0%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

J. Pic
In reply to this post by J. Pic
Nevermind my question you will get a lot more out of the workers, that Django 3.0 is going to be really blazing fast like channels that calls for a celebration xD

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAC6Op1-1apTUFZq9z5%3DEanpL%2B%2BA9nx9%3D3zS6_C-ZGNtJyBacog%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

Patryk Zawadzki-2
In reply to this post by J. Pic
W dniu czwartek, 9 maja 2019 22:47:48 UTC+2 użytkownik J. Pic napisał:
I'm a bit confused here, what benefit are you getting from async emails if you're already retrying emails in the background in production ?

Anything that uses I/O should be async to unblock the worker to process other things while for example waiting for an SMTP server to return. Network I/O like that often requires tens to hundreds of milliseconds, in this time you could process another request from another browser.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/2ce12106-b423-4dc1-a95a-56f112e63a0f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

Patryk Zawadzki-2
In reply to this post by Aymeric Augustin
I don't think it makes a significant difference from a readability perspective in this example. It does have some advantages, though:

- It could be easier to implement a sync version based on the async one, or vice-versa, if each one has its own class. It will probably be more DRY.
- It could also be a bit more usable by developers, especially in IDEs

Speaking about convenience, it would be great if we could make the sync versions raise exceptions when used with an event loop attached and the async ones fail when there isn't a loop running. I assume synchronous views would be executed within a worker thread with the event loop explicitly set to None. 

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5b6222c3-9f76-4eb3-8619-e3f44a17d6a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

KimSia Sim
In reply to this post by Andrew Godwin-3
Hi Andrew, 

I joined this group and chat because I saw the twitter post you made about this DEP. I  find that interesting.


On Friday, May 10, 2019 at 3:49:30 AM UTC+8, Andrew Godwin wrote:
 Tom Christie, for example, has already started work on an asynchronous ORM. Some of Django's biggest sites don't use the ORM - for example, Instagram.


I have a question that strays slightly away from the main topic. I have looked at Tom's repos. Is it encode/databases  https://github.com/encode/databases that you're referring to?

Or do you mean Tom's working on an async ORM that works within Django?

Thank you. I think Django will be even more awesome with async features

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/521a0e83-a23d-4d0b-a40f-45ecd30c4ef6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

Andrew Godwin-3
In reply to this post by Aymeric Augustin


I don't think it makes a significant difference from a readability perspective in this example. It does have some advantages, though:

- It could be easier to implement a sync version based on the async one, or vice-versa, if each one has its own class. It will probably be more DRY.
- It could also be a bit more usable by developers, especially in IDEs

Right. It could also be a sub-class on the class in question to make the implementation easier - e.g. "core.cache.cache.sync" is a literal passthrough, and "core.cache.cache.async" is the async wrapper implementation.
 

Also it could provide an alternative solution for the async attribute access problem: `model_instance.async.related_field` could be a Future that, when awaited, resolves to the related field. So you could `await model_instance.async.related_field` instead of having to `select_related`. To be honest I didn't investigate all the consequences of this idea. It seemed worth mentioning, though, even if it turns out to be impractical :-)

I think this might be feasible? I'd want to probably push it to have its own research and proposal, probably, but it might be a nice way out of the initial thing of requiring select_related. I just don't know enough about how that might cascade down the ORM internals to judge it at this point!

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1upEsCpnL-81guSw_vKLn%3DWmfM3pamuFY%2BSZWLMerNC0yA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django Async DEP

Andrew Godwin-3
In reply to this post by Patryk Zawadzki-2


On Thu, May 9, 2019 at 2:15 PM Patryk Zawadzki <[hidden email]> wrote:
I don't think it makes a significant difference from a readability perspective in this example. It does have some advantages, though:

- It could be easier to implement a sync version based on the async one, or vice-versa, if each one has its own class. It will probably be more DRY.
- It could also be a bit more usable by developers, especially in IDEs

Speaking about convenience, it would be great if we could make the sync versions raise exceptions when used with an event loop attached and the async ones fail when there isn't a loop running. I assume synchronous views would be executed within a worker thread with the event loop explicitly set to None. 

My ASGI patch already does this with the @async_unsafe decorator all over the ORM so people can't screw up by accident. It would be quite easy to extend this to enforcement on both the sync and async versions - there's maybe an edge case that you can call an async function from a thread you have not started an event loop in _yet_, but I'd rather see if and when that happens and provide a workaround, maybe.

Andrew 

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1uq4HX%3DHQa9Gmymckv-T_qXbGgA%3DtUrHn2pG8ejVSML3GA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
12