[Django] #30687: GIS distance lookups fail within subqueries using OuterRef

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Django] #30687: GIS distance lookups fail within subqueries using OuterRef

Django
#30687: GIS distance lookups fail within subqueries using OuterRef
-------------------------------------+-------------------------------------
               Reporter:  Andrew     |          Owner:  nobody
  Brown                              |
                   Type:  Bug        |         Status:  new
              Component:  Database   |        Version:  2.2
  layer (models, ORM)                |
               Severity:  Normal     |       Keywords:
           Triage Stage:             |      Has patch:  0
  Unreviewed                         |
    Needs documentation:  0          |    Needs tests:  0
Patch needs improvement:  0          |  Easy pickings:  0
                  UI/UX:  0          |
-------------------------------------+-------------------------------------
 I discovered this when trying to make a query of this form:

 {{{#!python
 from django.db import models
 from django.contrib.gis.db.models.fields import PointField, RasterField

 class ModelA(models.Model):
     pointA = PointField()

 class ModelB(models.Model):
     pointB = PointField()


 def query1():
     return ModelA.objects.annotate(
         has_value=Exists(ModelB.objects.filter(
             pointB__dwithin=(OuterRef("pointA"), 10)
         ))
     ).filter(has_value=True)
 }}}

 This fails with "ValueError: This queryset contains a reference to an
 outer query and may only be used in a subquery"

 I dug into things, and while I may not fully understand how queries are
 processed, I think I've identified the problem in
 `Query.resolve_lookup_value()`:

 {{{#!python
 def resolve_lookup_value(self, value, can_reuse, allow_joins, simple_col):
     if hasattr(value, 'resolve_expression'):
         kwargs = {'reuse': can_reuse, 'allow_joins': allow_joins}
         if isinstance(value, F):
             kwargs['simple_col'] = simple_col
         value = value.resolve_expression(self, **kwargs)
     elif isinstance(value, (list, tuple)):
         # The items of the iterable may be expressions and therefore need
         # to be resolved independently.
         for sub_value in value:
             if hasattr(sub_value, 'resolve_expression'):
                 if isinstance(sub_value, F):
                     sub_value.resolve_expression(
                         self, reuse=can_reuse, allow_joins=allow_joins,
                         simple_col=simple_col,
                     )
                 else:
                     sub_value.resolve_expression(self, reuse=can_reuse,
 allow_joins=allow_joins)
     return value
 }}}

 This resolves the value passed in as the rhs of a filter. For single
 objects, it calls `resolve_expression()` and then returns the result. But
 for multiple objects in a list or tuple, it calls `resolve_expression()`
 on each but doesn't return the resolved objects. Rather it returns the
 original list.

 As a consequence, during the call to filter, the passed in `OuterRef`
 object doesn't get resolved to a `ResolvedOuterRef` object, since the
 `__dwithin` lookup takes a tuple of `(value, distance)`, triggering that
 second code path above. Later during the call to annotate, the `OuterRef`
 does resolve to a `ResolvedOuterRef` but when the query is compiled,
 `ResolvedOuterRef.as_sql()` is called which raises the error.

 I modified `resolve_lookup_value()` to return the list or tuple of
 resolved values, and wrote a test for the query above, which I will submit
 in a PR shortly.

--
Ticket URL: <https://code.djangoproject.com/ticket/30687>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/050.6521171f7af93695fe73c74dbb0b5fdd%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30687: GIS distance lookups fail within subqueries using OuterRef

Django
#30687: GIS distance lookups fail within subqueries using OuterRef
-------------------------------------+-------------------------------------
     Reporter:  Andrew Brown         |                    Owner:  nobody
         Type:  Bug                  |                   Status:  new
    Component:  Database layer       |                  Version:  2.2
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:
                                     |  Unreviewed
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Andrew Brown):

 * has_patch:  0 => 1


Comment:

 PR submitted: https://github.com/django/django/pull/11634

--
Ticket URL: <https://code.djangoproject.com/ticket/30687#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.b29b2bf693f35e316a985f87055493a5%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30687: GIS distance lookups fail within subqueries using OuterRef

Django
In reply to this post by Django
#30687: GIS distance lookups fail within subqueries using OuterRef
-------------------------------------+-------------------------------------
     Reporter:  Andrew Brown         |                    Owner:  nobody
         Type:  Bug                  |                   Status:  new
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Carlton Gibson):

 * version:  2.2 => master
 * stage:  Unreviewed => Accepted


Comment:

 OK, thanks for the report and the test case Andrew.

--
Ticket URL: <https://code.djangoproject.com/ticket/30687#comment:2>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.00d08353f915a3da9c76b019fd166202%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30687: GIS distance lookups fail within subqueries using OuterRef

Django
In reply to this post by Django
#30687: GIS distance lookups fail within subqueries using OuterRef
-------------------------------------+-------------------------------------
     Reporter:  Andrew Brown         |                    Owner:  nobody
         Type:  Bug                  |                   Status:  new
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Ready for
                                     |  checkin
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Simon Charette):

 * stage:  Accepted => Ready for checkin


Comment:

 I left a few comments for improvement but I'm marking as '''RFC''' because
 they are really minor and could be addressed by the committer.

 Thanks for the patch and tests Andrew, that was a breeze to review.

--
Ticket URL: <https://code.djangoproject.com/ticket/30687#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.0031f0f8b7cd29ca6adbcda2c968bc6d%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30687: GIS distance lookups fail within subqueries using OuterRef

Django
In reply to this post by Django
#30687: GIS distance lookups fail within subqueries using OuterRef
-------------------------------------+-------------------------------------
     Reporter:  Andrew Brown         |                    Owner:  nobody
         Type:  Bug                  |                   Status:  closed
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:  fixed
     Keywords:                       |             Triage Stage:  Ready for
                                     |  checkin
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Mariusz Felisiak <felisiak.mariusz@…>):

 * status:  new => closed
 * resolution:   => fixed


Comment:

 In [changeset:"8a281aa7fe76a9da2284f943964a9413697cff1f" 8a281aa7]:
 {{{
 #!CommitTicketReference repository=""
 revision="8a281aa7fe76a9da2284f943964a9413697cff1f"
 Fixed #30687 -- Fixed using of OuterRef() expressions in distance lookups.
 }}}

--
Ticket URL: <https://code.djangoproject.com/ticket/30687#comment:4>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/065.1ef3b5ab85099a6c4b206c00a161b528%40djangoproject.com.