Prefixing Q Objects

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Prefixing Q Objects

Robert Schindler
Hi all,

I've been redirected to the mailing list from the ticked I opened [0]. Therein, I proposed to add a new method to the Q object API for prefixing Q objects with a related field's name in order to be able to create reusable Q objects.

I think the use case I pointed out might have been misunderstood. Such an API would allow writing a manager like this one, let's say for a Subscription model:

class SubscriptionManager(Manager):
    def active_q(self):
        return Q(cancelled=False) & Q(end_date__lt=datetime.datetime.now())

    # This one is just for convenience and compatibility with the evolved practice
    def active(self):
        return self.filter(self.active_q())

We could then perform the following lookups:

# All active subscriptions, nothing special here
Subscription.objects.active()
# Given that a user has a ManyToManyField subscriptions, get all users with active subscriptions, without repeating any business logic
User.objects.filter(Subscription.objects.active_q().prefix("subscriptions"))

The traditional approach would be to implement methods for filtering the queryset on both the managers for Subscription and User and thus duplicating logic, or use sub-queries like so:

User.objects.filter(subscriptions__in=Subscription.objects.active())

which is clearly not wanted because of performance.

What are your opinions?

Best regards
Robert

[0] https://code.djangoproject.com/ticket/30631


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/30020900-7a4a-442e-9cce-af3d75a27c15%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

RE: Prefixing Q Objects

Matthew Pava

I had a use for such a feature, though I used an implementation more specific to my application.

Basically, I have a model and then I have several other models that are directly related to that model. So in the base model I would have a field to determine if the instance was marked as deleted. I don’t have this field in the sub-models, so I would pass in the related name of the base model to create a queryset of all instances of the submodels that should be marked as deleted. Actually, that’s not exactly what I was doing, but the concept is there. Saying that, I am only doing that on one field on one model in my project.

 

From: 'Robert Schindler' via Django developers (Contributions to Django itself) [mailto:[hidden email]]
Sent: Wednesday, July 10, 2019 2:13 PM
To: Django developers (Contributions to Django itself)
Subject: Prefixing Q Objects

 

Hi all,

 

I've been redirected to the mailing list from the ticked I opened [0]. Therein, I proposed to add a new method to the Q object API for prefixing Q objects with a related field's name in order to be able to create reusable Q objects.

 

I think the use case I pointed out might have been misunderstood. Such an API would allow writing a manager like this one, let's say for a Subscription model:

 

class SubscriptionManager(Manager):

    def active_q(self):

        return Q(cancelled=False) & Q(end_date__lt=datetime.datetime.now())

 

    # This one is just for convenience and compatibility with the evolved practice

    def active(self):

        return self.filter(self.active_q())

 

We could then perform the following lookups:

 

# All active subscriptions, nothing special here

Subscription.objects.active()

# Given that a user has a ManyToManyField subscriptions, get all users with active subscriptions, without repeating any business logic

User.objects.filter(Subscription.objects.active_q().prefix("subscriptions"))

 

The traditional approach would be to implement methods for filtering the queryset on both the managers for Subscription and User and thus duplicating logic, or use sub-queries like so:

 

User.objects.filter(subscriptions__in=Subscription.objects.active())

 

which is clearly not wanted because of performance.

 

What are your opinions?

 

Best regards

Robert

 

[0] https://code.djangoproject.com/ticket/30631

 

 

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/30020900-7a4a-442e-9cce-af3d75a27c15%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/0c0a9f8a6dae49388bae789b926b9477%40iss2.ISS.LOCAL.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Prefixing Q Objects

Robert Schindler
Hi Matthew,

Am 11.07.19 um 04:13 schrieb Matthew Pava:

> I had a use for such a feature, though I used an implementation more
> specific to my application.
>
> Basically, I have a model and then I have several other models that are
> directly related to that model. So in the base model I would have a
> field to determine if the instance was marked as deleted. I don’t have
> this field in the sub-models, so I would pass in the related name of the
> base model to create a queryset of all instances of the submodels that
> should be marked as deleted. Actually, that’s not exactly what I was
> doing, but the concept is there. Saying that, I am only doing that on
> one field on one model in my project.

Yes, that would be a use-case indeed.

I'm using it for filtering a model's QuerySet for those objects a user
is authorized to see, which has to be implemented for all models in the
project so that a custom view mixin can then filter the queryset
regardless of the model. However, these queries heavily depend on
related fields.

For convenience, I created this module [0], which makes declaring the
query logic as part of a model's manager really straightforward and also
provides the well-known API for getting filtered querysets (aka
Fruit.objects.green()).

Actually, this helped me a lot declaring the needed queries DRY-free.

[0] https://github.com/efficiosoft/QProvider/blob/master/QProvider.py

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/35029311-3b9e-0f99-2752-2147415a305e%40efficiosoft.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Prefixing Q Objects

Robert Schindler
Hi again,

Am 11.07.19 um 09:54 schrieb Robert Schindler:
 > For convenience, I created this module [0], which makes declaring the
 > query logic as part of a model's manager really straightforward and also
 > provides the well-known API for getting filtered querysets (aka
 > Fruit.objects.green()).

Sorry for the dead link, here [0] is the right one.

Best regards
Robert

[0] https://github.com/efficiosoft/django-qprovider

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/8c5f9db8-a6bd-22e7-be4b-62a8421ed785%40efficiosoft.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Prefixing Q Objects

Shai Berger
In reply to this post by Robert Schindler
Hi Robert,

On Wed, 10 Jul 2019 12:13:08 -0700 (PDT)
Robert Schindler wrote:

> The traditional approach would be to implement methods for filtering
> the queryset on both the managers for Subscription and User and thus
> duplicating logic, or use sub-queries like so:
>
> User.objects.filter(subscriptions__in=Subscription.objects.active())
>
> which is clearly not wanted because of performance.
>
Not so clearly. Querysets are lazy, so the above actually only performs
one database roundtrip, and whether a join is more or less performant
than an equivalent query-with-a-sub-query can depend on many details --
mostly details of the database engine implementations. If database
engines were ideal, equivalent queries would have equivalent
performance.

More generally: The idea does seem interesting, but note that the
prefix() method doesn't need to be a member of the Q class -- it seems
to be just as easy to write an external function to do the same thing:

    def add_prefix(q_obj, prefix):
        """Recursively copies the Q object, prefixing all lookup keys.

        The prefix and the existing filter key are delimited by the
        lookup separator __. Use this feature to delegate existing
        query constraints to a related field. """
        return Q(
            *(
                add_prefix(child, prefix)
                if isinstance(child, Q)
                else (prefix + LOOKUP_SEP + child[0], child[1])
                for child in q_obj.children
            ),
            _connector=q_obj.connector,
            _negated=q_obj.negated,
        )

So, this can be added as a utility function to your project, or even
published in a package on PyPI, without being in Django.

Given this, the question is no longer just "is this useful", but rather
"does this promote use-patterns we wish to encourage". On the ticket,
Mariusz gave an example of a possible use-case we'd actually want to
discourage. Here on the list, better use-cases were shown.

One use case you didn't bring up is using a Q object with prefix as a
"template" -- consider querying for the "first Tuesday of the month";
if there's a date field `d`, it would need to be something like

        ...filter(d__week_day=3, d__day__lte=7)

But with a prefix, we could say something like

        def is_first_tuesday(field)
                return Q(week_day=3, day__lte=7).prefix(field)

and then use it in

        ... filter(is_first_tuesday('d'),...)

Which, at a very first glance, looks like an excellent use-case. Except
that Django already has a way to support this, using custom lookups,
which blend better into the QuerySet API -- and so, have technical
advantages beyond the fact that they already exist and "There should be
one obvious way to do it".

So, I, at least, remain unconvinced that overall, this is an
improvement to Django's API.

Shai

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20190711195702.71eb809b.shai%40platonix.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Prefixing Q Objects

Robert Schindler
Hi Shai,

Am 11.07.19 um 18:57 schrieb Shai Berger:
> Not so clearly. Querysets are lazy, so the above actually only performs
> one database roundtrip, and whether a join is more or less performant
> than an equivalent query-with-a-sub-query can depend on many details --
> mostly details of the database engine implementations. If database
> engines were ideal, equivalent queries would have equivalent
> performance.

I'm no database expert and you are probably right. At least that was
what I often heard, but I didn't take any benchmarks myself.

> More generally: The idea does seem interesting, but note that the
> prefix() method doesn't need to be a member of the Q class -- it seems
> to be just as easy to write an external function to do the same thing:

Sure it is, but I, personally, find it more appealing to have the
functionality as part of the Q object itself, because it operates on a
single Q object as input and returns another one, which makes this an
ideal candidate for an object member, imho. Not to say that it must be
in Django directly.
>
> So, this can be added as a utility function to your project, or even
> published in a package on PyPI, without being in Django.

That's the way I planned to do it. When I like the API, I'm going to add
unittests and publish the package, the structure is already there in the
repo.
>
> Given this, the question is no longer just "is this useful", but rather
> "does this promote use-patterns we wish to encourage". On the ticket,
> Mariusz gave an example of a possible use-case we'd actually want to
> discourage. Here on the list, better use-cases were shown.
 >

> One use case you didn't bring up is using a Q object with prefix as a
> "template" -- consider querying for the "first Tuesday of the month";
> if there's a date field `d`, it would need to be something like
>
> ...filter(d__week_day=3, d__day__lte=7)
>
> But with a prefix, we could say something like
>
> def is_first_tuesday(field)
> return Q(week_day=3, day__lte=7).prefix(field)
>
> and then use it in
>
> ... filter(is_first_tuesday('d'),...)
>
> Which, at a very first glance, looks like an excellent use-case. Except
> that Django already has a way to support this, using custom lookups,
> which blend better into the QuerySet API -- and so, have technical
> advantages beyond the fact that they already exist and "There should be
> one obvious way to do it".

Looks interesting, haven't thought of such a use case myself yet.

> So, I, at least, remain unconvinced that overall, this is an
> improvement to Django's API.
I agree that this makes most sense in conjunction with the additional
QProvider class for defining hybrid Q and QuerySet generators on
managers, which, even though very short and simple, is another addition
to be considered in this regard.

Thanks for taking the time.

Best regards
Robert

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/707036ac-987f-7cb1-1629-882e24d694c7%40efficiosoft.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Prefixing Q Objects

Robert Schindler
Am 11.07.19 um 19:20 schrieb Robert Schindler:
> Sure it is, but I, personally, find it more appealing to have the
> functionality as part of the Q object itself, because it operates on a
> single Q object as input and returns another one, which makes this an
> ideal candidate for an object member, imho. Not to say that it must be
> in Django directly.

Oops, should have been: "I don't want to say it must be in Django directly."
>>
Best regards
Robert

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/0a554a71-a9cd-c411-f417-213f2e7025c4%40efficiosoft.com.
For more options, visit https://groups.google.com/d/optout.