Quantcast

Database connection retry

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Database connection retry

James Pic-2
Hi all,

It seems like runserver won't retry to connect to the database after a failing connection. Once the db server is up, it looks like I have to restart runserver manually.

If this is correct, may I suggest that we make runserver retry connecting to the database if it fails ?

Thanks

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAC6Op19Q3nGUU1wCCsmdJbHhUvqEuaS-wNc9htAE1QXLQocsBw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Database connection retry

Tim Graham-2
Could you explain the use case a bit more? Why is your database failing on a regular basis?

On Wednesday, March 1, 2017 at 4:00:12 PM UTC-5, James Pic wrote:
Hi all,

It seems like runserver won't retry to connect to the database after a failing connection. Once the db server is up, it looks like I have to restart runserver manually.

If this is correct, may I suggest that we make runserver retry connecting to the database if it fails ?

Thanks

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5015961b-21c4-4d1f-83b4-cb962fa2330b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Database connection retry

James Pic
Sometimes it's not started because some modern orchestration tools such as ansible-container and docker-compose (perhaps more) start everything at once, and django might be faster than the db, or I have to fix something with the db orchestration tool.

I noted we might have the same issue with redis+channels (if I've not waited long enough before taking action instead of waiting for channels to retry). It seems like something reasonable to improve the development experience with Django while keeping up with the orchestration tools because I've heard about other users making tools in python on top of docker-compose just to add the orchestration that their Django project needed (I swear) already two years ago.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CALC3Kae-iWpWD3ZNFRu_GF9M0601bg8%2BpBGPR229%2BxgHrSfZ%2BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Database connection retry

Tim Graham-2
I don't know. Can you propose a patch so we can see what's involved? How would a "production" web server (nginx, apache, etc.) handle the issue?

I'm more interested in moving runserver toward using gunicorn [0] (Windows support seems the main blocker to proceeding there) than adding more features to our own webserver, although it's not clear if the fix would actually involve the webserver.

[0] https://code.djangoproject.com/ticket/21978

On Wednesday, March 1, 2017 at 6:52:48 PM UTC-5, is_null wrote:
Sometimes it's not started because some modern orchestration tools such as ansible-container and docker-compose (perhaps more) start everything at once, and django might be faster than the db, or I have to fix something with the db orchestration tool.

I noted we might have the same issue with redis+channels (if I've not waited long enough before taking action instead of waiting for channels to retry). It seems like something reasonable to improve the development experience with Django while keeping up with the orchestration tools because I've heard about other users making tools in python on top of docker-compose just to add the orchestration that their Django project needed (I swear) already two years ago.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/d2d72cc5-3b18-4d6e-bbed-833fdff02226%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Database connection retry

Aymeric Augustin
In reply to this post by James Pic-2
Hello James,

If I understand correctly, the problem is that runserver fails to boot because it cannot perform checks, some of which connect to the database (e.g. the check that all migrations were run).

Is that what you're talking about?

I suppose the fix involves something like:

while True:
    try:
        perform_system_checks()  # I made this up
    except Exception:
        log_exception()  # ditto
        time.sleep(1)
    else:
        break
    
Assuming the exception is clearly reported, that would be more beginner-friendly, which is a design goal of runserver. I think it's worth exploring, provided it doesn't create too much complexity.

Best regards,

-- 
Aymeric.



On 1 Mar 2017, at 21:59, James Pic <[hidden email]> wrote:

Hi all,

It seems like runserver won't retry to connect to the database after a failing connection. Once the db server is up, it looks like I have to restart runserver manually.

If this is correct, may I suggest that we make runserver retry connecting to the database if it fails ?

Thanks

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAC6Op19Q3nGUU1wCCsmdJbHhUvqEuaS-wNc9htAE1QXLQocsBw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/7F4FC0B0-8A47-4219-83FC-978A37F10D84%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Database connection retry

James Pic-2
Thanks for your feedback, the use case i was talking about is not quite valid anymore, since docker-stack and docker-compose v3 ​do handle dependencies.

However, perhaps runserver could just exit if checks don't pass, which makes sense I think, allowing the optional use of a shell loop. I'd prefer that, what do you think is better, exit or retry ?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAC6Op18z%3D2nAfxEqiJPczWz5r%3DLBCEXjrcJuO2G3LK5M%2BQJVzA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Database connection retry

Shai Berger
On Sunday 05 March 2017 12:32:34 James Pic wrote:
>
> However, perhaps runserver could just exit if checks don't pass, which
> makes sense I think, allowing the optional use of a shell loop. I'd prefer
> that, what do you think is better, exit or retry ?

... then you can do that yourself already by running

        manage.py check && manage.py runserver

My 2 cents,
        Shai.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Database connection retry

James Pic-2
until manage check; do sleep 1; done; manage runserver would work for me then, thanks Shai !

However, I'm still a bit puzzled by having a process that's just stuck when checks fail (if I understand correctly) is there any particular reason why it is this way ? If not, perhaps a retry or exit could improve the developer experience

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAC6Op1-d8s5D-56PWQACEyfsv%2Bx1cS_R%2Bt9auuRVx6-e-S-Q0g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Database connection retry

Aymeric Augustin
On 5 Mar 2017, at 12:27, James Pic <[hidden email]> wrote:

However, I'm still a bit puzzled by having a process that's just stuck when checks fail (if I understand correctly) is there any particular reason why it is this way ? If not, perhaps a retry or exit could improve the developer experience

To me this looks like a bug in the implementation of the auto-reloader.

Handling correctly both the initial invocation and subsequent invocations (after reloads) isn't as easy as it seems.

-- 
Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9CD08426-CC6A-4BA7-A9D4-AB8451FA47B0%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Database connection retry

James Pic-2
Thanks for sharing some of your insight Aymeric, if I'm not mistaken then the auto-reload feature/case invalidates Shai's suggestion: would you recommend that the runserver process exits with non-zero when a check fails rather than being stuck waiting for another code change to trigger a reload, so that we could wrap it in an until loop in bash ?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAC6Op190sX%3DnHepPOGikcFRAtvnNcEt8J-xMyDkeCjJPq-GZqQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Database connection retry

Tim Graham-2
The behavior of runserver hanging on a check error seems fine to me. That gives you an opportunity to fix the error without having to manually restart the server afterward -- just the same as if you had a SyntaxError. Am I missing the reason why the behavior is problematic?

On Tuesday, March 7, 2017 at 6:32:45 AM UTC-5, James Pic wrote:
Thanks for sharing some of your insight Aymeric, if I'm not mistaken then the auto-reload feature/case invalidates Shai's suggestion: would you recommend that the runserver process exits with non-zero when a check fails rather than being stuck waiting for another code change to trigger a reload, so that we could wrap it in an until loop in bash ?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/52269166-9d24-478a-95d2-f820786aabbe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Database connection retry

James Pic-2
It works on SyntaxErrors because updating the code triggers a reload, but if the check fails for something that's not related to code (db conn, redis conn...) then it's stuck and we have to manually interrupt runserver to start it again, unless we touch some code just to trigger the reload as you mention. My question is: is there anything we can do to automate this ?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAC6Op1_bS5AF-ukvdPmqTjn-moxz05wSOgEktkYt9_nc10VLDQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Database connection retry

Aymeric Augustin
Hello,

On 7 Mar 2017, at 13:30, James Pic <[hidden email]> wrote:

My question is: is there anything we can do to automate this ?

I'm not seeing an obvious solution to this problem.

Django has no way to tell it's a temporary issue.

-- 
Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/7AFCDB65-08B4-48F6-944F-8F5C5BA1EB98%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Database connection retry

James Pic-2
It seems like we have 2 kind of issues:

- code broke runserver,
- network broke runserver.

In the first case, runserver waits for a code reload event which is perfect ;)
In the second case, runserver also waits for a code reload event, which is not very intuitive after fixing a network error.

So if we want to handle both case, we indeed need to detect if an error is caused by code or networking, which is defined by CACHES, DATABASES and CHANNEL_LAYERS.

Perhaps we could add a special attribute to the exception, so DatabaseWrapper.get_new_connection()'s call of:

    connection = Database.connect(**conn_params) 

Would become something like:

    try:
        connection = Database.connect(**conn_params)
    except Exception as e:
        e.network_error = True
        raise

Another way would be to inspect exc info or have a pre-defined list of exceptions that are to be considered as network error, which involves referencing to exceptions potentially defined by other packages such as redis.

While that may seem a lot for runserver, I've restrained myself from talking about what this could look like in production so far in the discussion, but I feel like even production deployment could somehow benefit from this at some point, so that might be worth the effort after all.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAC6Op1-xvvwDHpo-Qzc2ycQQ6q1H7NO5FFJbZQqzphOGfPrZvA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Database connection retry

Chris Foresman
I'll chime in to say I've had a similar problem related to the shell and I couldn't sort out how to address it.

Our database servers will drop connections that last longer than 10 minutes. So basically can never do a task I might otherwise use the shell for that would take longer than 10 minutes of typing things in the shell. The workaround I eventually arrived at was copying all the data I pulled from previous runs into a doc in my text editor, and restart the shell every time the database connection dropped. Eventually I was able to just copy and paste enough stuff from that doc to get everything done in the 10 minute limit. Is there a way (e.g. the DatabaseWrapper mentioned above) to get the shell to reconnect without stopping the shell and restarting from scratch every time?

On Tuesday, March 7, 2017 at 7:30:27 AM UTC-6, James Pic wrote:
It seems like we have 2 kind of issues:

- code broke runserver,
- network broke runserver.

In the first case, runserver waits for a code reload event which is perfect ;)
In the second case, runserver also waits for a code reload event, which is not very intuitive after fixing a network error.

So if we want to handle both case, we indeed need to detect if an error is caused by code or networking, which is defined by CACHES, DATABASES and CHANNEL_LAYERS.

Perhaps we could add a special attribute to the exception, so DatabaseWrapper.get_new_connection()'s call of:

    connection = Database.connect(**conn_params) 

Would become something like:

    try:
        connection = Database.connect(**conn_params)
    except Exception as e:
        e.network_error = True
        raise

Another way would be to inspect exc info or have a pre-defined list of exceptions that are to be considered as network error, which involves referencing to exceptions potentially defined by other packages such as redis.

While that may seem a lot for runserver, I've restrained myself from talking about what this could look like in production so far in the discussion, but I feel like even production deployment could somehow benefit from this at some point, so that might be worth the effort after all.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/cc47bf21-9bbd-4728-9f2f-968405d128ad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Database connection retry

Adam Johnson-2
Chris, whilst I'm sure you could work something out, it probably wouldn't generally work as database connections contain a lot of state, such whether or not we're in a transaction and variables.

On 8 March 2017 at 20:23, Chris Foresman <[hidden email]> wrote:
I'll chime in to say I've had a similar problem related to the shell and I couldn't sort out how to address it.

Our database servers will drop connections that last longer than 10 minutes. So basically can never do a task I might otherwise use the shell for that would take longer than 10 minutes of typing things in the shell. The workaround I eventually arrived at was copying all the data I pulled from previous runs into a doc in my text editor, and restart the shell every time the database connection dropped. Eventually I was able to just copy and paste enough stuff from that doc to get everything done in the 10 minute limit. Is there a way (e.g. the DatabaseWrapper mentioned above) to get the shell to reconnect without stopping the shell and restarting from scratch every time?

On Tuesday, March 7, 2017 at 7:30:27 AM UTC-6, James Pic wrote:
It seems like we have 2 kind of issues:

- code broke runserver,
- network broke runserver.

In the first case, runserver waits for a code reload event which is perfect ;)
In the second case, runserver also waits for a code reload event, which is not very intuitive after fixing a network error.

So if we want to handle both case, we indeed need to detect if an error is caused by code or networking, which is defined by CACHES, DATABASES and CHANNEL_LAYERS.

Perhaps we could add a special attribute to the exception, so DatabaseWrapper.get_new_connection()'s call of:

    connection = Database.connect(**conn_params

Would become something like:

    try:
        connection = Database.connect(**conn_params)
    except Exception as e:
        e.network_error = True
        raise

Another way would be to inspect exc info or have a pre-defined list of exceptions that are to be considered as network error, which involves referencing to exceptions potentially defined by other packages such as redis.

While that may seem a lot for runserver, I've restrained myself from talking about what this could look like in production so far in the discussion, but I feel like even production deployment could somehow benefit from this at some point, so that might be worth the effort after all.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/cc47bf21-9bbd-4728-9f2f-968405d128ad%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Adam

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMyDDM0kFw%3Dr5ZvS-h7548dGSX1XgGEYweZOqS2XrRn%3DpdrD6Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Database connection retry

Aymeric Augustin
In reply to this post by Chris Foresman
Hello,

On 8 Mar 2017, at 21:23, Chris Foresman <[hidden email]> wrote:

I'll chime in to say I've had a similar problem related to the shell and I couldn't sort out how to address it.

In such situations, AFAIK, the following works:

from django.db import connection
connection.close()

Then Django will reopen the connection for the next database query.

HTH,

-- 
Aymeric.


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/E5F0989A-97C8-4E7E-8C69-FA08E60F76B6%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Database connection retry

Chris Foresman
Thanks Aymeric! I'll give that a try next time!


On Wednesday, March 8, 2017 at 3:20:31 PM UTC-6, Aymeric Augustin wrote:
Hello,

On 8 Mar 2017, at 21:23, Chris Foresman <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="KCOPuxrlBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">fore...@...> wrote:

I'll chime in to say I've had a similar problem related to the shell and I couldn't sort out how to address it.

In such situations, AFAIK, the following works:

from django.db import connection
connection.close()

Then Django will reopen the connection for the next database query.

HTH,

-- 
Aymeric.


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/e2cee051-7c7b-4224-99b1-908b21f968a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...