Django 2.2 and the watchman reloader

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Django 2.2 and the watchman reloader

Claude Paroz
Hi,

The new watchman-based autoreloader in Django 2.2 is a nice improvement. However, the main issue in my opinion is that there are no Debian/Ubuntu official packaging of watchman to my knowledge.
I've been able to compile it without difficulty, but I'm especially concerned about newcomers and non-experienced programmers which will be faced to a startup warning message and a rather heavy system load when they start developing with Django. Telling them they have to compile from sources to get rid of that is not very friendly at least.

Tom (Forbes), as the author of this refactoring, could you tell us if it would be much work to create a pyinotify reloader to fallback when watchman is not installed?

Claude

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/23ecaa54-a148-4439-a16b-16937da9e934%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django 2.2 and the watchman reloader

Dan Davis
Claude,

I've tested a Django based application on 2.2b1 without watchman on Windows, it does tell you about watchman, but it doesn't fail to run.  Apparently, it falls back to the old way of reloading.   Is this not the behavior on Debian/Ubuntu? 

On Thu, Feb 21, 2019 at 12:28 PM Claude Paroz <[hidden email]> wrote:
Hi,

The new watchman-based autoreloader in Django 2.2 is a nice improvement. However, the main issue in my opinion is that there are no Debian/Ubuntu official packaging of watchman to my knowledge.
I've been able to compile it without difficulty, but I'm especially concerned about newcomers and non-experienced programmers which will be faced to a startup warning message and a rather heavy system load when they start developing with Django. Telling them they have to compile from sources to get rid of that is not very friendly at least.

Tom (Forbes), as the author of this refactoring, could you tell us if it would be much work to create a pyinotify reloader to fallback when watchman is not installed?

Claude

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/23ecaa54-a148-4439-a16b-16937da9e934%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFzonYbE4fSy7wsO_wDgHy1X%2BRaWppXejQXB-tnbPtC5RjwwoA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django 2.2 and the watchman reloader

Claude Paroz
Le jeudi 21 février 2019 19:48:31 UTC+1, Dan Davis a écrit :
Claude,

I've tested a Django based application on 2.2b1 without watchman on Windows, it does tell you about watchman, but it doesn't fail to run.  Apparently, it falls back to the old way of reloading.   Is this not the behavior on Debian/Ubuntu?

Yes it is. I would say this is still a slight regression in two ways:

- no messages told you the reload method was not optimal before. So now people will try to "fix" their system, more than before.
- for Debian-based systems, you could improve the reloading performance by installing system or pip packages in 30 seconds. Now you have to spend 30 minutes to search how watchman can be installed and to compile the package (+ you have to care yourself for any security issue).

Claude

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/b059553a-e25e-4d90-beed-bf7e0f797305%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django 2.2 and the watchman reloader

Claude Paroz
In reply to this post by Claude Paroz
I have a POC patch here: https://github.com/django/django/pull/11014

Claude

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/0e5965c9-0e1c-42ed-8d58-55a71ff6e91c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django 2.2 and the watchman reloader

Tom Forbes
In reply to this post by Claude Paroz
Hey Claude,
Thanks for your feedback on the feature, I fully agree with you. I think we should remove that warning message about the missing package. I will make a PR to do that.

Regarding creating another reloader: it should not be that difficult to do at all since we have all the other pieces in place. In theory it's just implementing a class with single generator method.

If people agree I would like to use the 'watchdog' package for this rather than the pyinotify library as it would be quicker to implement, a lot nicer to work with and is easier to test.

On Thu, 21 Feb 2019, 19:56 Claude Paroz, <[hidden email]> wrote:
Le jeudi 21 février 2019 19:48:31 UTC+1, Dan Davis a écrit :
Claude,

I've tested a Django based application on 2.2b1 without watchman on Windows, it does tell you about watchman, but it doesn't fail to run.  Apparently, it falls back to the old way of reloading.   Is this not the behavior on Debian/Ubuntu?

Yes it is. I would say this is still a slight regression in two ways:

- no messages told you the reload method was not optimal before. So now people will try to "fix" their system, more than before.
- for Debian-based systems, you could improve the reloading performance by installing system or pip packages in 30 seconds. Now you have to spend 30 minutes to search how watchman can be installed and to compile the package (+ you have to care yourself for any security issue).

Claude

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/b059553a-e25e-4d90-beed-bf7e0f797305%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJMK_MCbKByWkVKC7QY8_As30%2BnKA4rR1c9UNqWLhUX60g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django 2.2 and the watchman reloader

Claude Paroz
Le jeudi 21 février 2019 21:43:43 UTC+1, Tom Forbes a écrit :
Hey Claude,
Thanks for your feedback on the feature, I fully agree with you. I think we should remove that warning message about the missing package. I will make a PR to do that.

I'm not completely sure it's a good idea to entirely remove the message. Maybe just telling the used reloader would be fine.
 
Regarding creating another reloader: it should not be that difficult to do at all since we have all the other pieces in place. In theory it's just implementing a class with single generator method.

If people agree I would like to use the 'watchdog' package for this rather than the pyinotify library as it would be quicker to implement, a lot nicer to work with and is easier to test.

++1, watchdog is better maintained, looks like pyinotify is dead. Tell me if you need help, even if you seems a lot more knowledgeable than me on the subject.
 
Thanks.

Claude


On Thu, 21 Feb 2019, 19:56 Claude Paroz, <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="C46J-4mPAgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">cla...@...> wrote:
Le jeudi 21 février 2019 19:48:31 UTC+1, Dan Davis a écrit :
Claude,

I've tested a Django based application on 2.2b1 without watchman on Windows, it does tell you about watchman, but it doesn't fail to run.  Apparently, it falls back to the old way of reloading.   Is this not the behavior on Debian/Ubuntu?

Yes it is. I would say this is still a slight regression in two ways:

- no messages told you the reload method was not optimal before. So now people will try to "fix" their system, more than before.
- for Debian-based systems, you could improve the reloading performance by installing system or pip packages in 30 seconds. Now you have to spend 30 minutes to search how watchman can be installed and to compile the package (+ you have to care yourself for any security issue).

Claude

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" rel="nofollow" target="_blank" gdf-obfuscated-mailto="C46J-4mPAgAJ" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-develop...@googlegroups.com.
To post to this group, send email to <a href="javascript:" rel="nofollow" target="_blank" gdf-obfuscated-mailto="C46J-4mPAgAJ" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-d...@googlegroups.com.
Visit this group at <a href="https://groups.google.com/group/django-developers" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/b059553a-e25e-4d90-beed-bf7e0f797305%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/b059553a-e25e-4d90-beed-bf7e0f797305%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/b059553a-e25e-4d90-beed-bf7e0f797305%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/django-developers/b059553a-e25e-4d90-beed-bf7e0f797305%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6de70f64-9bbb-4017-bd10-c85e94ae08d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django 2.2 and the watchman reloader

Tom Forbes

I have a small PR here to remove the “watchman unavailable” message, whilst keeping the one that specifies which reloader we are using: https://github.com/django/django/pull/11025.




On 21 February 2019 at 20:52:29, Claude Paroz ([hidden email]) wrote:

Le jeudi 21 février 2019 21:43:43 UTC+1, Tom Forbes a écrit :
Hey Claude,
Thanks for your feedback on the feature, I fully agree with you. I think we should remove that warning message about the missing package. I will make a PR to do that.

I'm not completely sure it's a good idea to entirely remove the message. Maybe just telling the used reloader would be fine.
 
Regarding creating another reloader: it should not be that difficult to do at all since we have all the other pieces in place. In theory it's just implementing a class with single generator method.

If people agree I would like to use the 'watchdog' package for this rather than the pyinotify library as it would be quicker to implement, a lot nicer to work with and is easier to test.

++1, watchdog is better maintained, looks like pyinotify is dead. Tell me if you need help, even if you seems a lot more knowledgeable than me on the subject.
 
Thanks.

Claude


On Thu, 21 Feb 2019, 19:56 Claude Paroz, <<a href="" target="_blank" rel="nofollow" onmousedown="this.href=&#39;&#39;;return true;" onclick="this.href=&#39;&#39;;return true;">cla...@...> wrote:
Le jeudi 21 février 2019 19:48:31 UTC+1, Dan Davis a écrit :
Claude,

I've tested a Django based application on 2.2b1 without watchman on Windows, it does tell you about watchman, but it doesn't fail to run.  Apparently, it falls back to the old way of reloading.   Is this not the behavior on Debian/Ubuntu?

Yes it is. I would say this is still a slight regression in two ways:

- no messages told you the reload method was not optimal before. So now people will try to "fix" their system, more than before.
- for Debian-based systems, you could improve the reloading performance by installing system or pip packages in 30 seconds. Now you have to spend 30 minutes to search how watchman can be installed and to compile the package (+ you have to care yourself for any security issue).

Claude
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="" rel="nofollow" target="_blank" onmousedown="this.href=&#39;&#39;;return true;" onclick="this.href=&#39;&#39;;return true;">django-develop...@googlegroups.com.
To post to this group, send email to <a href="" rel="nofollow" target="_blank" onmousedown="this.href=&#39;&#39;;return true;" onclick="this.href=&#39;&#39;;return true;">django-d...@googlegroups.com.
Visit this group at <a href="https://groups.google.com/group/django-developers" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;"> https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/b059553a-e25e-4d90-beed-bf7e0f797305%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/b059553a-e25e-4d90-beed-bf7e0f797305%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/b059553a-e25e-4d90-beed-bf7e0f797305%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;"> https://groups.google.com/d/msgid/django-developers/b059553a-e25e-4d90-beed-bf7e0f797305%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6de70f64-9bbb-4017-bd10-c85e94ae08d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJOOELOoSEGGx9LrMryrYFhqm%2Bzd_PPZ-ztarjiZyR%2Bvnw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Django 2.2 and the watchman reloader

Tom Forbes
Just an update to this: I've made a WIP PR to add watchdog support here: https://github.com/django/django/pull/11480

I'd like to disable it on Windows and MacOS, and there are some tricky issues to resolve around how events are triggered (watchdog uses a number of threads internally, and we need to ensure only the main thread triggers the file changed notification), but otherwise it seems to be working OK locally.

On Mon, Feb 25, 2019 at 10:22 PM Tom Forbes <[hidden email]> wrote:

I have a small PR here to remove the “watchman unavailable” message, whilst keeping the one that specifies which reloader we are using: https://github.com/django/django/pull/11025.




On 21 February 2019 at 20:52:29, Claude Paroz ([hidden email]) wrote:

Le jeudi 21 février 2019 21:43:43 UTC+1, Tom Forbes a écrit :
Hey Claude,
Thanks for your feedback on the feature, I fully agree with you. I think we should remove that warning message about the missing package. I will make a PR to do that.

I'm not completely sure it's a good idea to entirely remove the message. Maybe just telling the used reloader would be fine.
 
Regarding creating another reloader: it should not be that difficult to do at all since we have all the other pieces in place. In theory it's just implementing a class with single generator method.

If people agree I would like to use the 'watchdog' package for this rather than the pyinotify library as it would be quicker to implement, a lot nicer to work with and is easier to test.

++1, watchdog is better maintained, looks like pyinotify is dead. Tell me if you need help, even if you seems a lot more knowledgeable than me on the subject.
 
Thanks.

Claude


On Thu, 21 Feb 2019, 19:56 Claude Paroz, <[hidden email]> wrote:
Le jeudi 21 février 2019 19:48:31 UTC+1, Dan Davis a écrit :
Claude,

I've tested a Django based application on 2.2b1 without watchman on Windows, it does tell you about watchman, but it doesn't fail to run.  Apparently, it falls back to the old way of reloading.   Is this not the behavior on Debian/Ubuntu?

Yes it is. I would say this is still a slight regression in two ways:

- no messages told you the reload method was not optimal before. So now people will try to "fix" their system, more than before.
- for Debian-based systems, you could improve the reloading performance by installing system or pip packages in 30 seconds. Now you have to spend 30 minutes to search how watchman can be installed and to compile the package (+ you have to care yourself for any security issue).

Claude
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/b059553a-e25e-4d90-beed-bf7e0f797305%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6de70f64-9bbb-4017-bd10-c85e94ae08d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJMLobYmyXQh2vV93%3DP8OCSmKi29-BM9Z1FPiw-YM7FXkw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.