[Django] #30727: Pickling a Query object evaluates querysets in Subquery expressions

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Django] #30727: Pickling a Query object evaluates querysets in Subquery expressions

Django
#30727: Pickling a Query object evaluates querysets in Subquery expressions
-------------------------------------+-------------------------------------
               Reporter:  Andrew     |          Owner:  nobody
  Brown                              |
                   Type:             |         Status:  new
  Uncategorized                      |
              Component:  Database   |        Version:  2.2
  layer (models, ORM)                |
               Severity:  Normal     |       Keywords:
           Triage Stage:             |      Has patch:  0
  Unreviewed                         |
    Needs documentation:  0          |    Needs tests:  0
Patch needs improvement:  0          |  Easy pickings:  0
                  UI/UX:  0          |
-------------------------------------+-------------------------------------
 I wrote a test case for `tests/queryset_pickle/tests.py` modeled after the
 test from bug #27499 which is very similar.

 {{{#!python
     def test_pickle_subquery_queryset_not_evaluated(self):
         """
         Verifies that querysets passed into Subquery expressions
         are not evaluated when pickled
         """
         groups = Group.objects.annotate(
 has_event=models.Exists(Event.objects.filter(group_id=models.OuterRef('id')))
         )
         with self.assertNumQueries(0):
             pickle.loads(pickle.dumps(groups.query))
 }}}

 The Subquery class (which is the base for `Exists`) only stores the
 underlying query object and throws the QuerySet away (as of this
 [https://github.com/django/django/commit/96b6ad94d9ebbd57b77b44e185ee215b5b899ac8
 commit], although I don't think it worked before that). So in theory it
 shouldn't be pickling the QuerySet.

 However, the QuerySet is still stored on the instance within the
 `_constructor_args` attribute added by the `@deconstructible` decorator on
 the `BaseExpression` base class. So when the inner query object gets
 pickled, so does the QuerySet, which attempts to evaluate it. In this
 case, it gets the error "ValueError: This queryset contains a reference to
 an outer query and may only be used in a subquery." since the inner
 queryset is being evaluated independently when called from pickle: it
 calls `QuerySet.__getstate__()`.

 I'm not sure what the best solution is here. I made a patch that adds the
 following override to `__getstate__` to the SubQuery class, which fixes
 the problem and passes all other Django tests on my machine. I'll submit a
 PR shortly, but welcome any better approaches since I'm not sure what else
 that may effect.

 {{{#!python
 class Subquery(Expression):
     ...
     def __getstate__(self):
         obj_dict = super().__getstate__()
         obj_dict.pop('_constructor_args', None)
         return obj_dict
 }}}

--
Ticket URL: <https://code.djangoproject.com/ticket/30727>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/050.2d080b5ea491b61632b11fb45a7055c4%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30727: Pickling a Query object evaluates querysets in Subquery expressions

Django
#30727: Pickling a Query object evaluates querysets in Subquery expressions
-------------------------------------+-------------------------------------
     Reporter:  Andrew Brown         |                    Owner:  nobody
         Type:  Uncategorized        |                   Status:  new
    Component:  Database layer       |                  Version:  2.2
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:
                                     |  Unreviewed
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Andrew Brown):

 * has_patch:  0 => 1


Comment:

 PR: https://github.com/django/django/pull/11707

--
Ticket URL: <https://code.djangoproject.com/ticket/30727#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.659bed27c0f8fd3104ca1d2c98c970a0%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30727: Pickling a Query object evaluates querysets in Subquery expressions

Django
In reply to this post by Django
#30727: Pickling a Query object evaluates querysets in Subquery expressions
-------------------------------------+-------------------------------------
     Reporter:  Andrew Brown         |                    Owner:  nobody
         Type:  Bug                  |                   Status:  new
    Component:  Database layer       |                  Version:  2.2
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:
                                     |  Unreviewed
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Andrew Brown):

 * type:  Uncategorized => Bug


--
Ticket URL: <https://code.djangoproject.com/ticket/30727#comment:2>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.91d623245bb27d1af90de06f291afed5%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30727: Pickling a QuerySet evaluates the querysets given to Subquery in annotate. (was: Pickling a Query object evaluates querysets in Subquery expressions)

Django
In reply to this post by Django
#30727: Pickling a QuerySet evaluates the querysets given to Subquery in annotate.
-------------------------------------+-------------------------------------
     Reporter:  Andrew Brown         |                    Owner:  Andrew
                                     |  Brown
         Type:  Bug                  |                   Status:  assigned
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by felixxm):

 * status:  new => assigned
 * owner:  nobody => Andrew Brown
 * version:  2.2 => master
 * stage:  Unreviewed => Accepted


--
Ticket URL: <https://code.djangoproject.com/ticket/30727#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.e860f8e616d8f33e6dc9409b9e4762c4%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30727: Pickling a QuerySet evaluates the querysets given to Subquery in annotate.

Django
In reply to this post by Django
#30727: Pickling a QuerySet evaluates the querysets given to Subquery in annotate.
-------------------------------------+-------------------------------------
     Reporter:  Andrew Brown         |                    Owner:  Andrew
                                     |  Brown
         Type:  Bug                  |                   Status:  closed
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:  fixed
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Mariusz Felisiak <felisiak.mariusz@…>):

 * status:  assigned => closed
 * resolution:   => fixed


Comment:

 In [changeset:"691def10a0197d83d2d108bd9043b0916d0f09b4" 691def1]:
 {{{
 #!CommitTicketReference repository=""
 revision="691def10a0197d83d2d108bd9043b0916d0f09b4"
 Fixed #30727 -- Made Subquery pickle without evaluating their QuerySet.

 Subquery expression objects, when pickled, were evaluating the QuerySet
 objects saved in its _constructor_args attribute.
 }}}

--
Ticket URL: <https://code.djangoproject.com/ticket/30727#comment:4>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.8d35a9ee52af16411bbe0ded08475b3d%40djangoproject.com.