[Django] #31373: Django 3.x / uwsgi is leaking connections

classic Classic list List threaded Threaded
38 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[Django] #31373: Django 3.x / uwsgi is leaking connections

Django
#31373: Django 3.x / uwsgi is leaking connections
-----------------------------------------+------------------------
               Reporter:  Marcin Nowak   |          Owner:  nobody
                   Type:  Uncategorized  |         Status:  new
              Component:  Uncategorized  |        Version:  3.0
               Severity:  Normal         |       Keywords:
           Triage Stage:  Unreviewed     |      Has patch:  0
    Needs documentation:  0              |    Needs tests:  0
Patch needs improvement:  0              |  Easy pickings:  0
                  UI/UX:  0              |
-----------------------------------------+------------------------
 Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
 connections.
 ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0

 After few hours all posgtress connections are used (too many connections
 error).
 pg_stat_activity is reporting idle connections with part of queries
 visible.

 I would try to deliver more details later.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/050.93a8d0fc4e89003bd67a304a1c5cacb2%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------
     Reporter:  Marcin Nowak   |                    Owner:  nobody
         Type:  Uncategorized  |                   Status:  new
    Component:  Uncategorized  |                  Version:  3.0
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Unreviewed
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+--------------------------------------
Changes (by Marcin Nowak):

 * Attachment "Firefox_Screenshot_2020-03-17T15-05-35.104Z.png" added.


--
Ticket URL: <https://code.djangoproject.com/ticket/31373>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.292c20fc9f47ac9dbfc66c9dfa6f75af%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
In reply to this post by Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------
     Reporter:  Marcin Nowak   |                    Owner:  nobody
         Type:  Uncategorized  |                   Status:  new
    Component:  Uncategorized  |                  Version:  3.0
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Unreviewed
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:

Old description:

> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
>
> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.
New description:

 Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
 connections.
 ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0

 After few hours all posgtress connections are used (too many connections
 error).
 pg_stat_activity is reporting idle connections with part of queries
 visible.

 I would try to deliver more details later.

 [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

--

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.560d5e3e4779ee6fdbdadf0742e72c6a%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
In reply to this post by Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------
     Reporter:  Marcin Nowak   |                    Owner:  nobody
         Type:  Uncategorized  |                   Status:  new
    Component:  Uncategorized  |                  Version:  3.0
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Unreviewed
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+--------------------------------------
Changes (by Marcin Nowak):

 * Attachment "Firefox_Screenshot_2020-03-17T19-42-34.977Z.png" added.


--
Ticket URL: <https://code.djangoproject.com/ticket/31373>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.9a3b91b708d73c3ad99e5b6e5658e4cc%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
In reply to this post by Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------
     Reporter:  Marcin Nowak   |                    Owner:  nobody
         Type:  Uncategorized  |                   Status:  new
    Component:  Uncategorized  |                  Version:  3.0
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Unreviewed
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:

Old description:

> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
>
> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.
>
> [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]
New description:

 Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
 connections.
 ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0

 After few hours all posgtress connections are used (too many connections
 error).
 pg_stat_activity is reporting idle connections with part of queries
 visible.

 I would try to deliver more details later.

 [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

 Facts:
 - setting up CONN_MAX_AGE > 0 does not help
 - starting `runserver` and making many calls with `ab -c 10 -n 100
 http://127.0.0.1:8000/` uses all available connections and ends with error
 500 (due to no available connections)
 - starting `runserver --nothreading` and making calls with  `ab -c 100 -n
 100 http://127.0.0.1:8000/` works fine

--

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:2>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.bbb390f5dc09064f134c082e92121e2d%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
In reply to this post by Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------
     Reporter:  Marcin Nowak   |                    Owner:  nobody
         Type:  Uncategorized  |                   Status:  new
    Component:  Uncategorized  |                  Version:  3.0
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Unreviewed
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:

Old description:

> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
>
> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.
>
> [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]
>
> Facts:
> - setting up CONN_MAX_AGE > 0 does not help
> - starting `runserver` and making many calls with `ab -c 10 -n 100
> http://127.0.0.1:8000/` uses all available connections and ends with
> error 500 (due to no available connections)
> - starting `runserver --nothreading` and making calls with  `ab -c 100 -n
> 100 http://127.0.0.1:8000/` works fine
New description:

 Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
 connections.
 ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
 UWSGi threads are enabled (`--enable-threads`)

 After few hours all posgtress connections are used (too many connections
 error).
 pg_stat_activity is reporting idle connections with part of queries
 visible.

 I would try to deliver more details later.

 [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

 Facts:
 - setting up CONN_MAX_AGE > 0 does not help
 - starting `runserver` and making many calls with `ab -c 10 -n 100
 http://127.0.0.1:8000/` uses all available connections and ends with error
 500 (due to no available connections)
 - starting `runserver --nothreading` and making calls with  `ab -c 100 -n
 100 http://127.0.0.1:8000/` works fine

--

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.48fe2a24c0654db690c0d67e0e0f26aa%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
In reply to this post by Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------
     Reporter:  Marcin Nowak   |                    Owner:  nobody
         Type:  Uncategorized  |                   Status:  new
    Component:  Uncategorized  |                  Version:  3.0
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Unreviewed
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:

Old description:

> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
> UWSGi threads are enabled (`--enable-threads`)
>
> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.
>
> [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]
>
> Facts:
> - setting up CONN_MAX_AGE > 0 does not help
> - starting `runserver` and making many calls with `ab -c 10 -n 100
> http://127.0.0.1:8000/` uses all available connections and ends with
> error 500 (due to no available connections)
> - starting `runserver --nothreading` and making calls with  `ab -c 100 -n
> 100 http://127.0.0.1:8000/` works fine
New description:

 Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
 connections.
 ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
 UWSGi threads are enabled (`--enable-threads`)
 Python 3.8.1 and 3.7.3

 After few hours all posgtress connections are used (too many connections
 error).
 pg_stat_activity is reporting idle connections with part of queries
 visible.

 I would try to deliver more details later.

 [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

 Facts:
 - setting up CONN_MAX_AGE > 0 does not help
 - starting `runserver` and making many calls with `ab -c 10 -n 100
 http://127.0.0.1:8000/` uses all available connections and ends with error
 500 (due to no available connections)
 - starting `runserver --nothreading` and making calls with  `ab -c 100 -n
 100 http://127.0.0.1:8000/` works fine

--

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:4>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.99ee8b0478a96595ce94c54babee6e33%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
In reply to this post by Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------
     Reporter:  Marcin Nowak   |                    Owner:  nobody
         Type:  Uncategorized  |                   Status:  new
    Component:  Uncategorized  |                  Version:  3.0
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Unreviewed
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:

Old description:

> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
> UWSGi threads are enabled (`--enable-threads`)
> Python 3.8.1 and 3.7.3
>
> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.
>
> [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]
>
> Facts:
> - setting up CONN_MAX_AGE > 0 does not help
> - starting `runserver` and making many calls with `ab -c 10 -n 100
> http://127.0.0.1:8000/` uses all available connections and ends with
> error 500 (due to no available connections)
> - starting `runserver --nothreading` and making calls with  `ab -c 100 -n
> 100 http://127.0.0.1:8000/` works fine
New description:

 Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
 connections.
 ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
 UWSGi threads are enabled (`--enable-threads`)
 Python 3.8.1 and 3.7.3

 After few hours all posgtress connections are used (too many connections
 error).
 pg_stat_activity is reporting idle connections with part of queries
 visible.

 I would try to deliver more details later.

 [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

 Facts:
 - setting up CONN_MAX_AGE > 0 does not help
 -  CONN_MAX_AGE = 30, starting `runserver` and making many calls with `ab
 -c 10 -n 100 http://127.0.0.1:8000/` uses all available connections and
 ends with error 500 (due to no available connections)
 - CONN_MAX_AGE = 30, starting `runserver --nothreading` and making calls
 with  `ab -c 100 -n 100 http://127.0.0.1:8000/` works fine


 CONN_MAX_AGE=30 + runserver + ab scenario is reproducible on clean
 install. View must call database.
 CONN_MAX_AGE=0 + runserver + ab -> cannot reproduce.

--

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:5>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.64dbb3c0168afdbdeaf86d335971d2ad%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
In reply to this post by Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------
     Reporter:  Marcin Nowak   |                    Owner:  nobody
         Type:  Uncategorized  |                   Status:  new
    Component:  Uncategorized  |                  Version:  3.0
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Unreviewed
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:

Old description:

> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
> UWSGi threads are enabled (`--enable-threads`)
> Python 3.8.1 and 3.7.3
>
> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.
>
> [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]
>
> Facts:
> - setting up CONN_MAX_AGE > 0 does not help
> -  CONN_MAX_AGE = 30, starting `runserver` and making many calls with `ab
> -c 10 -n 100 http://127.0.0.1:8000/` uses all available connections and
> ends with error 500 (due to no available connections)
> - CONN_MAX_AGE = 30, starting `runserver --nothreading` and making calls
> with  `ab -c 100 -n 100 http://127.0.0.1:8000/` works fine
>

> CONN_MAX_AGE=30 + runserver + ab scenario is reproducible on clean
> install. View must call database.
> CONN_MAX_AGE=0 + runserver + ab -> cannot reproduce.

New description:

 Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
 connections.
 ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
 UWSGi threads are enabled (`--enable-threads`)
 Python 3.8.1 and 3.7.3

 After few hours all posgtress connections are used (too many connections
 error).
 pg_stat_activity is reporting idle connections with part of queries
 visible.

 I would try to deliver more details later.

 [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

 Facts:
 - setting up CONN_MAX_AGE > 0 does not help
 -  CONN_MAX_AGE = 30, starting `runserver` and making many calls with `ab
 -c 10 -n 100 http://127.0.0.1:8000/` uses all available connections and
 ends with error 500 (due to no available connections)
 - CONN_MAX_AGE = 30, starting `runserver --nothreading` and making calls
 with  `ab -c 100 -n 100 http://127.0.0.1:8000/` works fine
 - CONN_MAX_AGE=30 + runserver + ab scenario is reproducible on clean
 install. Called view must access database.
 - CONN_MAX_AGE=0 + runserver + ab -> cannot reproduce.
 - Django 2.x has similar issue for CONN_MAX_AGE=30, but can reuse
 connections (still there are several errors on high traffic, but app
 server can "recover" and handle bunch of incoming requests)
 - Looks like Django 3.x cannot reuse connections

 `ab -c 10 -n 200 ` Django 3.0.4:
 Complete requests:      200
 Failed requests:        108

 `ab -c 10 -n 1000` Django 2.2.11:
 Complete requests:      1000
 Failed requests:        37

--

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:6>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.44394bba299af967e75f2b2ff0673608%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
In reply to this post by Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------
     Reporter:  Marcin Nowak   |                    Owner:  nobody
         Type:  Uncategorized  |                   Status:  new
    Component:  Uncategorized  |                  Version:  3.0
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Unreviewed
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:

Old description:

> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
> UWSGi threads are enabled (`--enable-threads`)
> Python 3.8.1 and 3.7.3
>
> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.
>
> [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]
>
> Facts:
> - setting up CONN_MAX_AGE > 0 does not help
> -  CONN_MAX_AGE = 30, starting `runserver` and making many calls with `ab
> -c 10 -n 100 http://127.0.0.1:8000/` uses all available connections and
> ends with error 500 (due to no available connections)
> - CONN_MAX_AGE = 30, starting `runserver --nothreading` and making calls
> with  `ab -c 100 -n 100 http://127.0.0.1:8000/` works fine
> - CONN_MAX_AGE=30 + runserver + ab scenario is reproducible on clean
> install. Called view must access database.
> - CONN_MAX_AGE=0 + runserver + ab -> cannot reproduce.
> - Django 2.x has similar issue for CONN_MAX_AGE=30, but can reuse
> connections (still there are several errors on high traffic, but app
> server can "recover" and handle bunch of incoming requests)
> - Looks like Django 3.x cannot reuse connections
>
> `ab -c 10 -n 200 ` Django 3.0.4:
> Complete requests:      200
> Failed requests:        108
>
> `ab -c 10 -n 1000` Django 2.2.11:
> Complete requests:      1000
> Failed requests:        37
New description:

 Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
 connections.
 ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
 UWSGi threads are enabled (`--enable-threads`)
 Python 3.8.1 and 3.7.3

 After few hours all posgtress connections are used (too many connections
 error).
 pg_stat_activity is reporting idle connections with part of queries
 visible.

 I would try to deliver more details later.

 [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

 ----

 Facts:
 - setting up CONN_MAX_AGE > 0 does not help
 -  CONN_MAX_AGE = 30, starting `runserver` and making many calls with `ab
 -c 10 -n 100 http://127.0.0.1:8000/` uses all available connections and
 ends with error 500 (due to no available connections)
 - CONN_MAX_AGE = 30, starting `runserver --nothreading` and making calls
 with  `ab -c 100 -n 100 http://127.0.0.1:8000/` works fine
 - CONN_MAX_AGE=30 + runserver + ab scenario is reproducible on clean
 install. Called view must access database.
 - CONN_MAX_AGE=0 + runserver + ab -> cannot reproduce.
 - Django 2.x has similar issue for CONN_MAX_AGE=30, but can reuse
 connections (still there are several errors on high traffic, but app
 server can "recover" and handle bunch of incoming requests)
 - Looks like Django 3.x cannot reuse connections

 ----

 Runserver (with threads) test results:

 `ab -c 10 -n 200 ` Django 3.0.4:
 Complete requests:      200
 Failed requests:        108

 `ab -c 10 -n 1000` Django 2.2.11:
 Complete requests:      1000
 Failed requests:        37

 ----

 There may be two separate cases:
 1. not reusing connection when CONN_MAX_AGE is greater than 0
 2. leaking (not closing?) connections on uwsgi with enabled threads and
 CONN_MAX_AGE set to 0

--

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:7>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.719111c7a477342a64cbb8afd2409889%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
In reply to this post by Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------
     Reporter:  Marcin Nowak   |                    Owner:  nobody
         Type:  Uncategorized  |                   Status:  new
    Component:  Uncategorized  |                  Version:  3.0
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Unreviewed
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:

Old description:

> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
> UWSGi threads are enabled (`--enable-threads`)
> Python 3.8.1 and 3.7.3
>
> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.
>
> [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]
>
> ----
>
> Facts:
> - setting up CONN_MAX_AGE > 0 does not help
> -  CONN_MAX_AGE = 30, starting `runserver` and making many calls with `ab
> -c 10 -n 100 http://127.0.0.1:8000/` uses all available connections and
> ends with error 500 (due to no available connections)
> - CONN_MAX_AGE = 30, starting `runserver --nothreading` and making calls
> with  `ab -c 100 -n 100 http://127.0.0.1:8000/` works fine
> - CONN_MAX_AGE=30 + runserver + ab scenario is reproducible on clean
> install. Called view must access database.
> - CONN_MAX_AGE=0 + runserver + ab -> cannot reproduce.
> - Django 2.x has similar issue for CONN_MAX_AGE=30, but can reuse
> connections (still there are several errors on high traffic, but app
> server can "recover" and handle bunch of incoming requests)
> - Looks like Django 3.x cannot reuse connections
>
> ----
>
> Runserver (with threads) test results:
>
> `ab -c 10 -n 200 ` Django 3.0.4:
> Complete requests:      200
> Failed requests:        108
>
> `ab -c 10 -n 1000` Django 2.2.11:
> Complete requests:      1000
> Failed requests:        37
>
> ----
>
> There may be two separate cases:
> 1. not reusing connection when CONN_MAX_AGE is greater than 0
> 2. leaking (not closing?) connections on uwsgi with enabled threads and
> CONN_MAX_AGE set to 0
New description:

 Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
 connections.
 ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
 UWSGi threads are enabled (`--enable-threads`)
 Python 3.8.1 and 3.7.3

 After few hours all posgtress connections are used (too many connections
 error).
 pg_stat_activity is reporting idle connections with part of queries
 visible.

 I would try to deliver more details later.

 [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

 ----

 Facts:
 - setting up CONN_MAX_AGE > 0 does not help
 -  CONN_MAX_AGE = 30, starting `runserver` and making many calls with `ab
 -c 10 -n 100 http://127.0.0.1:8000/` uses all available connections and
 ends with error 500 (due to no available connections)
 - CONN_MAX_AGE = 30, starting `runserver --nothreading` and making calls
 with  `ab -c 100 -n 100 http://127.0.0.1:8000/` works fine
 - CONN_MAX_AGE=30 + runserver + ab scenario is reproducible on clean
 install. Called view must access database.
 - CONN_MAX_AGE=0 + runserver + ab -> cannot reproduce.
 - Django 2.x has similar issue for CONN_MAX_AGE=30, but can reuse
 connections (still there are several errors on high traffic, but app
 server can "recover" and handle bunch of incoming requests)
 - Looks like Django 3.x cannot reuse connections
 - uwsgi + threads + CONN_MAX_AGE=0 on localhost -> cannot reproduce

 ----

 Runserver (with threads) test results:

 `ab -c 10 -n 200 ` Django 3.0.4:
 Complete requests:      200
 Failed requests:        108

 `ab -c 10 -n 1000` Django 2.2.11:
 Complete requests:      1000
 Failed requests:        37

 ----

 There may be two separate cases:
 1. not reusing connection when CONN_MAX_AGE is greater than 0
 2. leaking (not closing?) connections on uwsgi (?) with enabled threads
 (?) and CONN_MAX_AGE set to 0

--

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:8>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.b8b4652e1d087176f11e86872222afb4%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
In reply to this post by Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------
     Reporter:  Marcin Nowak   |                    Owner:  nobody
         Type:  Uncategorized  |                   Status:  new
    Component:  Uncategorized  |                  Version:  3.0
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Unreviewed
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+--------------------------------------
Description changed by Marcin Nowak:

Old description:

> Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
> connections.
> ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
> UWSGi threads are enabled (`--enable-threads`)
> Python 3.8.1 and 3.7.3
>
> After few hours all posgtress connections are used (too many connections
> error).
> pg_stat_activity is reporting idle connections with part of queries
> visible.
>
> I would try to deliver more details later.
>
> [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]
>
> ----
>
> Facts:
> - setting up CONN_MAX_AGE > 0 does not help
> -  CONN_MAX_AGE = 30, starting `runserver` and making many calls with `ab
> -c 10 -n 100 http://127.0.0.1:8000/` uses all available connections and
> ends with error 500 (due to no available connections)
> - CONN_MAX_AGE = 30, starting `runserver --nothreading` and making calls
> with  `ab -c 100 -n 100 http://127.0.0.1:8000/` works fine
> - CONN_MAX_AGE=30 + runserver + ab scenario is reproducible on clean
> install. Called view must access database.
> - CONN_MAX_AGE=0 + runserver + ab -> cannot reproduce.
> - Django 2.x has similar issue for CONN_MAX_AGE=30, but can reuse
> connections (still there are several errors on high traffic, but app
> server can "recover" and handle bunch of incoming requests)
> - Looks like Django 3.x cannot reuse connections
> - uwsgi + threads + CONN_MAX_AGE=0 on localhost -> cannot reproduce
>
> ----
>
> Runserver (with threads) test results:
>
> `ab -c 10 -n 200 ` Django 3.0.4:
> Complete requests:      200
> Failed requests:        108
>
> `ab -c 10 -n 1000` Django 2.2.11:
> Complete requests:      1000
> Failed requests:        37
>
> ----
>
> There may be two separate cases:
> 1. not reusing connection when CONN_MAX_AGE is greater than 0
> 2. leaking (not closing?) connections on uwsgi (?) with enabled threads
> (?) and CONN_MAX_AGE set to 0
New description:

 Project upgraded from 2.x to 3.0.4 is leaking PostgreSQL database
 connections.
 ATOMIC_REQUESTS=False, AUTOCOMMIT=True, CONN_MAX_AGE=0
 UWSGi threads are enabled (`--enable-threads`)
 Python 3.8.1 and 3.7.3

 After few hours all posgtress connections are used (too many connections
 error).
 pg_stat_activity is reporting idle connections with part of queries
 visible.


 [[Image(Firefox_Screenshot_2020-03-17T15-05-35.104Z.png)]]

 ----

 Facts:
 - setting up CONN_MAX_AGE > 0 does not help
 -  CONN_MAX_AGE = 30, starting `runserver` and making many calls with `ab
 -c 10 -n 100 http://127.0.0.1:8000/` uses all available connections and
 ends with error 500 (due to no available connections)
 - CONN_MAX_AGE = 30, starting `runserver --nothreading` and making calls
 with  `ab -c 100 -n 100 http://127.0.0.1:8000/` works fine
 - CONN_MAX_AGE=30 + runserver + ab scenario is reproducible on clean
 install. Called view must access database.
 - CONN_MAX_AGE=0 + runserver + ab -> cannot reproduce.
 - Django 2.x has similar issue for CONN_MAX_AGE=30, but can reuse
 connections (still there are several errors on high traffic, but app
 server can "recover" and handle bunch of incoming requests)
 - Looks like Django 3.x cannot reuse connections
 - uwsgi + threads + CONN_MAX_AGE=0 on localhost -> cannot reproduce

 ----

 Runserver (with threads) test results:

 `ab -c 10 -n 200 ` Django 3.0.4:
 Complete requests:      200
 Failed requests:        108

 `ab -c 10 -n 1000` Django 2.2.11:
 Complete requests:      1000
 Failed requests:        37

 ----

 There may be two separate cases:
 1. not reusing connection when CONN_MAX_AGE is greater than 0
 2. leaking (not closing?) connections on uwsgi (?) with enabled threads
 (?) and CONN_MAX_AGE set to 0

 I'm not sure about the 2nd case.

--

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:9>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.cbd08eb7689b2c65486d0b0e40f98bae%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
In reply to this post by Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------
     Reporter:  Marcin Nowak   |                    Owner:  nobody
         Type:  Uncategorized  |                   Status:  closed
    Component:  Uncategorized  |                  Version:  3.0
     Severity:  Normal         |               Resolution:  needsinfo
     Keywords:                 |             Triage Stage:  Unreviewed
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+--------------------------------------
Changes (by Simon Charette):

 * status:  new => closed
 * resolution:   => needsinfo


Comment:

 I understand your desire to see this addressed but without a simplified
 test project that reproduces your issue it's impossible to determine
 whether or not Django is at fault and not one of your projects third-party
 app is.

 Please reopen this issue if you can provide a reproduction case else it's
 simply not actionable by triagers that don't have access to the project
 you're trying to upgrade. Thanks!

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:10>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.f980d95e26729daf7a3ab22a17c8df0b%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
In reply to this post by Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------
     Reporter:  Marcin Nowak   |                    Owner:  nobody
         Type:  Uncategorized  |                   Status:  closed
    Component:  Uncategorized  |                  Version:  3.0
     Severity:  Normal         |               Resolution:  needsinfo
     Keywords:                 |             Triage Stage:  Unreviewed
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+--------------------------------------

Comment (by Marcin Nowak):

 Replying to [comment:10 Simon Charette]:
 > I understand your desire to see this addressed but without a simplified
 test project that reproduces your issue it's impossible to determine
 whether or not Django is at fault and its not one of your projects third-
 party app that is.
 >
 > Please reopen this issue if you can provide a reproduction case else
 it's simply not actionable by triagers that don't have access to the
 project you're trying to upgrade. Thanks!

 The case about not reusing db connections is reproducible.
 1. Just create a new project
 2. Setup postgresql_psycopg2 db, set CONN_MAX_AGE=30,
 3. create view hitting the database (User.objects.count() for example),
 bind it to "/" route
 4. make sure your postgres has connection limit set to quite low value
 (mine is 100)
 5. manage.py runserver
 6. run `ab -c 10 -n 200 https://127.0.0.1:8000/`

 My case from production server is quite different,  maybe related somehow.
 There is a chance that my project is causing the issue, but it looks that
 Django 3.x has own issue too, which maybe increases my problem.   The most
 significant change in my project was upgrade of Django, from 2.x to 3.x

 Sorry for chaotic description. It's been 16 hour of my today's work, for
 last few hours I was trying to solve the issue, find the cause (I gave up,
 sorry). Now I'm working on getting Django 2.x back, backporting some 3.x
 features to 2.x, so I cannot focus anymore on reproducing my issue in
 isolated environment (I know how important it is).

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:11>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.85a0ab3c6925be2290ed15f98b84dffe%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
In reply to this post by Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------
     Reporter:  Marcin Nowak   |                    Owner:  nobody
         Type:  Uncategorized  |                   Status:  closed
    Component:  Uncategorized  |                  Version:  3.0
     Severity:  Normal         |               Resolution:  needsinfo
     Keywords:                 |             Triage Stage:  Unreviewed
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+--------------------------------------
Changes (by Marcin Nowak):

 * Attachment "connmaxage.zip" added.

 Example project with CONN_MAX_AGE issue

--
Ticket URL: <https://code.djangoproject.com/ticket/31373>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.ea4a7d1e84c13972087ad0c9eabab0c6%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
In reply to this post by Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------
     Reporter:  Marcin Nowak   |                    Owner:  nobody
         Type:  Uncategorized  |                   Status:  closed
    Component:  Uncategorized  |                  Version:  3.0
     Severity:  Normal         |               Resolution:  needsinfo
     Keywords:                 |             Triage Stage:  Unreviewed
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+--------------------------------------

Comment (by Marcin Nowak):

 {{{
 createdb connmaxage
 python -m venv /tmp/env
 . /tmp/env/bin/activate
 pip install django==3.0.4
 pip install psycopg2-binary
 cd /tmp/
 unzip connmaxage.zip
 cd connmaxage
 ./manage.py runserver
 }}}

 then (on the 2nd terminal)

 {{{
 ab -c 10 -n 1000 http://127.0.0.1:8000/
 }}}

 Result:

 {{{
 [17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
 [17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
 [17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
 [17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
 [17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
 [17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
 [17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
 [17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
 [17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
 [17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
 [17/Mar/2020 23:20:23] "GET / HTTP/1.0" 200 1
 Internal Server Error: /
 Traceback (most recent call last):
   File "/tmp/env/lib/python3.8/site-
 packages/django/db/backends/base/base.py", line 220, in ensure_connection
     self.connect()
   File "/tmp/env/lib/python3.8/site-packages/django/utils/asyncio.py",
 line 26, in inner
     return func(*args, **kwargs)
   File "/tmp/env/lib/python3.8/site-
 packages/django/db/backends/base/base.py", line 197, in connect
     self.connection = self.get_new_connection(conn_params)
   File "/tmp/env/lib/python3.8/site-packages/django/utils/asyncio.py",
 line 26, in inner
     return func(*args, **kwargs)
   File "/tmp/env/lib/python3.8/site-
 packages/django/db/backends/postgresql/base.py", line 185, in
 get_new_connection
     connection = Database.connect(**conn_params)
   File "/tmp/env/lib/python3.8/site-packages/psycopg2/__init__.py", line
 126, in connect
     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
 psycopg2.OperationalError: FATAL:  sorry, too many clients already


 The above exception was the direct cause of the following exception:

 Traceback (most recent call last):
   File "/tmp/env/lib/python3.8/site-
 packages/django/core/handlers/exception.py", line 34, in inner
     response = get_response(request)
   File "/tmp/env/lib/python3.8/site-
 packages/django/core/handlers/base.py", line 115, in _get_response
     response = self.process_exception_by_middleware(e, request)
   File "/tmp/env/lib/python3.8/site-
 packages/django/core/handlers/base.py", line 113, in _get_response
     response = wrapped_callback(request, *callback_args,
 **callback_kwargs)
   File "/tmp/connmaxage/connmaxage/main/views.py", line 7, in main
     return HttpResponse(str(User.objects.count()))
   File "/tmp/env/lib/python3.8/site-packages/django/db/models/manager.py",
 line 82, in manager_method
     return getattr(self.get_queryset(), name)(*args, **kwargs)
   File "/tmp/env/lib/python3.8/site-packages/django/db/models/query.py",
 line 397, in count
     return self.query.get_count(using=self.db)
   File "/tmp/env/lib/python3.8/site-
 packages/django/db/models/sql/query.py", line 517, in get_count
     number = obj.get_aggregation(using, ['__count'])['__count']
   File "/tmp/env/lib/python3.8/site-
 packages/django/db/models/sql/query.py", line 502, in get_aggregation
     result = compiler.execute_sql(SINGLE)
   File "/tmp/env/lib/python3.8/site-
 packages/django/db/models/sql/compiler.py", line 1149, in execute_sql
     cursor = self.connection.cursor()
   File "/tmp/env/lib/python3.8/site-packages/django/utils/asyncio.py",
 line 26, in inner
     return func(*args, **kwargs)
   File "/tmp/env/lib/python3.8/site-
 packages/django/db/backends/base/base.py", line 260, in cursor
     return self._cursor()
   File "/tmp/env/lib/python3.8/site-
 packages/django/db/backends/base/base.py", line 236, in _cursor
     self.ensure_connection()
   File "/tmp/env/lib/python3.8/site-packages/django/utils/asyncio.py",
 line 26, in inner
     return func(*args, **kwargs)
   File "/tmp/env/lib/python3.8/site-
 packages/django/db/backends/base/base.py", line 220, in ensure_connection
     self.connect()
   File "/tmp/env/lib/python3.8/site-packages/django/db/utils.py", line 90,
 in __exit__
     raise dj_exc_value.with_traceback(traceback) from exc_value
   File "/tmp/env/lib/python3.8/site-
 packages/django/db/backends/base/base.py", line 220, in ensure_connection
     self.connect()
   File "/tmp/env/lib/python3.8/site-packages/django/utils/asyncio.py",
 line 26, in inner
     return func(*args, **kwargs)
   File "/tmp/env/lib/python3.8/site-
 packages/django/db/backends/base/base.py", line 197, in connect
     self.connection = self.get_new_connection(conn_params)
   File "/tmp/env/lib/python3.8/site-packages/django/utils/asyncio.py",
 line 26, in inner
     return func(*args, **kwargs)
   File "/tmp/env/lib/python3.8/site-
 packages/django/db/backends/postgresql/base.py", line 185, in
 get_new_connection
     connection = Database.connect(**conn_params)
   File "/tmp/env/lib/python3.8/site-packages/psycopg2/__init__.py", line
 126, in connect
     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
 django.db.utils.OperationalError: FATAL:  sorry, too many clients already

 [17/Mar/2020 23:20:23] "GET / HTTP/1.0" 500 166202
 }}}

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:12>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.9a1fc2fb87f06ab891b2a0eb368f122c%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
In reply to this post by Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------+--------------------------------------
     Reporter:  Marcin Nowak   |                    Owner:  nobody
         Type:  Uncategorized  |                   Status:  closed
    Component:  Uncategorized  |                  Version:  3.0
     Severity:  Normal         |               Resolution:  needsinfo
     Keywords:                 |             Triage Stage:  Unreviewed
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+--------------------------------------

Comment (by Marcin Nowak):

 Ab results for Django 3:

 {{{
 Complete requests:      1000
 Failed requests:        901
 }}}

 Re-test with Django 2:

 1. in the same env `pip install django==2.2.11`
 2. do same `ab` test:

 {{{
 Complete requests:      1000
 Failed requests:        19
 }}}

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:13>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.9127e08d1968aede506da035a5af8e0a%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
In reply to this post by Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------------+-------------------------------------
     Reporter:  Marcin Nowak         |                    Owner:  nobody
         Type:  Bug                  |                   Status:  new
    Component:  Database layer       |                  Version:  3.0
  (models, ORM)                      |
     Severity:  Release blocker      |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Simon Charette):

 * status:  closed => new
 * severity:  Normal => Release blocker
 * cc: Andrew Godwin (added)
 * type:  Uncategorized => Bug
 * component:  Uncategorized => Database layer (models, ORM)
 * resolution:  needsinfo =>
 * stage:  Unreviewed => Accepted


Comment:

 Thanks for the sample project.

 I managed to bisect the regression at
 a415ce70bef6d91036b00dd2c8544aed7aeeaaed which introduced async support in
 Django.

 I didn't investigate why connections kept piling up but I suspect
 [https://github.com/django/django/commit/a415ce70bef6d91036b00dd2c8544aed7aeeaaed
 #diff-c50ed26e574fb41592a832e0b37c5713R143-R148 these lines] are to blame.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:14>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.8e0b47e4105f1030430619cc719e0b38%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
In reply to this post by Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------------+-------------------------------------
     Reporter:  Marcin Nowak         |                    Owner:  nobody
         Type:  Bug                  |                   Status:  new
    Component:  Database layer       |                  Version:  3.0
  (models, ORM)                      |
     Severity:  Release blocker      |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Andrew Godwin):

 That would not surprise me - next things to investigate are:

 - Of course, reverting that one change and seeing if it fixes it.
 - Does uwsgi override any part of the threading module? This happens with
 gevent and, I think, mod_wsgi, and it causes havoc as they ruin thread
 IDs.
 - Checking a couple of recent major versions of asgiref within Django's
 compatibility range, to see if the change was there (doubtful)

 Likely cause is that the close_old_connections() task is never being run
 in what Local thinks is the same thread as before. I have a potential
 workaround we can do if it _is_ overriding thread IDs, but if not, we'll
 have to dive deeper and see what we can fix about it.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:15>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.7b0b28c32f418fc4c208eb9241b446d2%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #31373: Django 3.x / uwsgi is leaking connections

Django
In reply to this post by Django
#31373: Django 3.x / uwsgi is leaking connections
-------------------------------------+-------------------------------------
     Reporter:  Marcin Nowak         |                    Owner:  nobody
         Type:  Bug                  |                   Status:  new
    Component:  Database layer       |                  Version:  3.0
  (models, ORM)                      |
     Severity:  Release blocker      |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Simon Charette):

 * cc: Simon Charette (added)


Comment:

 Andrew what about `runserver`?

 I understand it's not meant to be used as a production http server but the
 fact the issue can be reproduced pretty easily makes me believe something
 might be wrong with this commit or `asgiref.local.Local`.

--
Ticket URL: <https://code.djangoproject.com/ticket/31373#comment:16>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.cbc27eb5d4ead7a069fbade5e4e34a93%40djangoproject.com.
12