[Django] #24810: Reopen database connection automatically when no transaction is active

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[Django] #24810: Reopen database connection automatically when no transaction is active

Django
#24810: Reopen database connection automatically when no transaction is active
-------------------------------------+-------------------------------------
               Reporter:  aaugustin  |          Owner:  nobody
                   Type:  New        |         Status:  new
  feature                            |
              Component:  Database   |        Version:  master
  layer (models, ORM)                |
               Severity:  Normal     |       Keywords:
           Triage Stage:  Accepted   |      Has patch:  0
    Needs documentation:  0          |    Needs tests:  0
Patch needs improvement:  0          |  Easy pickings:  0
                  UI/UX:  0          |
-------------------------------------+-------------------------------------
 This is a follow-up of #15802.

 When the database closes the connection, for instance because the database
 server was restarted, an exception will be raised the next time user code
 attempts to use the database. Django doesn't attempt to do anything about
 it.

 This is expected to result in at most one 500 error for each thread that
 serves requests, since database connections are thread-local. This doesn't
 sound horrific. After all the database was restarted while the website was
 serving requests.

 It can also cause management commands to crash, which isn't the end of the
 world either since management commands should fall into one of the
 following categories:
 - short-lived, idempotent and scheduled to run regularly
 - long-lived and supervised to run constantly
 - run manually

 If the connection was in autocommit mode and an operation fails because
 the database has closed the connection, theoretically, it's safe the
 reopen the connection and retry the operation.

 In that case, Django could continue to operate transparently instead of
 raising an exception.

 This may not be easy to implement.

--
Ticket URL: <https://code.djangoproject.com/ticket/24810>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/052.169dff053fe2864d216a7fae7d207a1e%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #24810: Reopen database connection automatically when no transaction is active

Django
#24810: Reopen database connection automatically when no transaction is active
-------------------------------------+-------------------------------------
     Reporter:  aaugustin            |                    Owner:  nobody
         Type:  New feature          |                   Status:  new
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by akaariai):

 It must be noted that SQL connections can have state even when not in a
 transaction. These could be for example session variables, per-connection
 configuration (SET search_path = xxx), temporary tables etc. While Django
 doesn't use any of these directly, breaking Django for users who do use
 connection state should be avoided.

 If we are going to reopen connections automatically we need some safe-
 guards against above problems. As we can't detect automatically when users
 have session state in their connections, the users should tell us about
 this. I think opt-out (or maybe opt-in?) flag in the connection's settings
 could work. Something like set `OPTIONS['stateful_connection'] = True`,
 and Django will not try to reopen failed connections.

 Other than the above remark I don't see a problem with this approach.
 Still, without further evidence that these kinds of failures are common, I
 believe this will require too much complexity compared to what is gained.

--
Ticket URL: <https://code.djangoproject.com/ticket/24810#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/067.57457ee0875981eaf2e5f9fd9055fd8e%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #24810: Reopen database connection automatically when no transaction is active

Django
In reply to this post by Django
#24810: Reopen database connection automatically when no transaction is active
-------------------------------------+-------------------------------------
     Reporter:  aaugustin            |                    Owner:  nobody
         Type:  New feature          |                   Status:  new
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by depaolim):

 * cc: depaolim@… (added)


--
Ticket URL: <https://code.djangoproject.com/ticket/24810#comment:2>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/067.1e7aba4db667066114e1ec17a91b503f%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #24810: Reopen database connection automatically when no transaction is active

Django
In reply to this post by Django
#24810: Reopen database connection automatically when no transaction is active
-------------------------------------+-------------------------------------
     Reporter:  aaugustin            |                    Owner:  nobody
         Type:  New feature          |                   Status:  new
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by binary):

 Yes, I have the same problem in the daemon tasks that access DB through
 django models.

 I don't see a problem with the connection state, actually, as we can say
 to do all the necessary preparations in the connection_created signal and
 then newly created connection can be set up correctly?

--
Ticket URL: <https://code.djangoproject.com/ticket/24810#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/067.0c1138be876ddea296953939080c94dc%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #24810: Reopen database connection automatically when no transaction is active

Django
In reply to this post by Django
#24810: Reopen database connection automatically when no transaction is active
-------------------------------------+-------------------------------------
     Reporter:  aaugustin            |                    Owner:  nobody
         Type:  New feature          |                   Status:  new
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by binary):

 * cc: binary@… (added)


--
Ticket URL: <https://code.djangoproject.com/ticket/24810#comment:4>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/067.825f6c6d15c0661bc85a7a2e17e0d1f5%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #24810: Reopen database connection automatically when no transaction is active

Django
In reply to this post by Django
#24810: Reopen database connection automatically when no transaction is active
-------------------------------------+-------------------------------------
     Reporter:  aaugustin            |                    Owner:  nobody
         Type:  New feature          |                   Status:  new
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by noky):

 It should be noted that in the case where a daemon is accessing the DB via
 the ORM, if the db connection is closed for some reason (usually the db
 server restarting), then ALL subsequent DB read queries FAIL. At that
 point, the process can no longer perform any read query since the db
 connection is closed. I'm not concerned with retrying statements that
 failed during the time the db was down, but it does seem like a bug that
 the ORM completely fails to reconnect AT ALL. The funny thing is: if and
 only if the process happens to perform a DB write after the DB connection
 gets closed, then the ORM will open a new connection.

--
Ticket URL: <https://code.djangoproject.com/ticket/24810#comment:5>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/067.b4bb1cd9fd690e383aee9f873a7cf316%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #24810: Reopen database connection automatically when no transaction is active

Django
In reply to this post by Django
#24810: Reopen database connection automatically when no transaction is active
-------------------------------------+-------------------------------------
     Reporter:  aaugustin            |                    Owner:  nobody
         Type:  New feature          |                   Status:  new
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by aaugustin):

 Yes, these are the reasons why this ticket exists. Patches *that do not
 ignore transactional integrity* welcome.

--
Ticket URL: <https://code.djangoproject.com/ticket/24810#comment:6>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/067.793dad914f562573a69002b91dd91eb4%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #24810: Reopen database connection automatically when no transaction is active

Django
In reply to this post by Django
#24810: Reopen database connection automatically when no transaction is active
-------------------------------------+-------------------------------------
     Reporter:  aaugustin            |                    Owner:  nobody
         Type:  New feature          |                   Status:  new
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by binary):

 Replying to [comment:6 aaugustin]:
 > Yes, these are the reasons why this ticket exists. Patches *that do not
 ignore transactional integrity* welcome.
 So I gave it a humble try.

 There is no problem in case the trouble happened inside a transaction
 because in case error happened and was caught by the Atomic, the
 connection will be closed in one of the places like
 [https://github.com/django/django/blob/master/django/db/transaction.py#L230
 this]. And next transaction will work without any problem as connection
 will be re-established because django is now aware that the current one is
 not an option any more.

 It's still a problem in autocommit mode though because in this case
 connection won't be closed and will remain in a broken state, so in case
 we try to obtain a cursor again, we'll get the InterfaceError exception.
 So the code like:

 {{{
 def do_query():
     return Model.objects.first()

 def loop():
     while True:
         try:
             print do_query()
         except:
             print 'shit happened'
         sleep(5)
 }}}

 will stop working after first disconnect.

 At first I tried to intercept InterfaceError at the moment we are
 obtaining a cursor and OperationalErorr at the execute method, but then I
 realized that such a problem may happen in other methods, like fetchone or
 anywhere else, so eventually I came to just intercept it in the
 DatabaseErrorWrapper and close the connection if one of those errors
 occured but only in case we are not inside a transaction, as this case is
 covered already and it may break that code. Here's the commit:
 https://github.com/jbinary/django/commit/568a4eb573c7fce2fdec2d1578b2db4f3d0e51fd

 After that, that code above starts to work even after disconnect.

 I have no any idea if this is the right approach, and I have no idea how
 to write tests for this yet. But if anyone will tell this one is a worthy
 approach, I'll try to think about tests too. (but advices are welcome,
 ofc)

--
Ticket URL: <https://code.djangoproject.com/ticket/24810#comment:7>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/067.2f71b8e271d2b77c0f501e22d21a073d%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #24810: Reopen database connection automatically when no transaction is active

Django
In reply to this post by Django
#24810: Reopen database connection automatically when no transaction is active
-------------------------------------+-------------------------------------
     Reporter:  aaugustin            |                    Owner:  nobody
         Type:  New feature          |                   Status:  new
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  1                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by binary):

 * has_patch:  0 => 1
 * needs_tests:  0 => 1


--
Ticket URL: <https://code.djangoproject.com/ticket/24810#comment:8>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/067.e41cd2bbac0e89bd3cb8eb1e58442580%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #24810: Reopen database connection automatically when no transaction is active

Django
In reply to this post by Django
#24810: Reopen database connection automatically when no transaction is active
-------------------------------------+-------------------------------------
     Reporter:  Aymeric Augustin     |                    Owner:  nobody
         Type:  New feature          |                   Status:  new
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  1                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Marti Raudsepp):

 * cc: Marti Raudsepp (added)


--
Ticket URL: <https://code.djangoproject.com/ticket/24810#comment:9>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/067.e1ba60aa3e9e87054ad3aef50e4834b4%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.