[Django] #28668: Add support for ON CONFLICT to bulk_create

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
40 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[Django] #28668: Add support for ON CONFLICT to bulk_create

Django
#28668: Add support for ON CONFLICT to bulk_create
-------------------------------------+-------------------------------------
               Reporter:  Tom        |          Owner:  nobody
                   Type:  New        |         Status:  new
  feature                            |
              Component:  Database   |        Version:  master
  layer (models, ORM)                |
               Severity:  Normal     |       Keywords:
           Triage Stage:             |      Has patch:  0
  Unreviewed                         |
    Needs documentation:  0          |    Needs tests:  0
Patch needs improvement:  0          |  Easy pickings:  0
                  UI/UX:  0          |
-------------------------------------+-------------------------------------
 When using bulk_create it would be nice to support `ON CONFLICT DO
 NOTHING`, which allows existing rows to be included in the `bulk_create`
 call, e.g:

 {{{#!python
 Comment.objects.create(name='test', text='test')
 Comment.objects.bulk_create(Comment(name='test', text='test'),
 on_conflict=IGNORE)  # Does not throw an exception
 }}}

 All the databases we support have syntax for this, as well as
 updating/replacing fields.

--
Ticket URL: <https://code.djangoproject.com/ticket/28668>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/046.adebe81a35602b7d6bca0b0b9e447097%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add support for ON CONFLICT to bulk_create

Django
#28668: Add support for ON CONFLICT to bulk_create
-------------------------------------+-------------------------------------
     Reporter:  Tom                  |                    Owner:  nobody
         Type:  New feature          |                   Status:  new
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:
                                     |  Unreviewed
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Description changed by Tom:

Old description:

> When using bulk_create it would be nice to support `ON CONFLICT DO
> NOTHING`, which allows existing rows to be included in the `bulk_create`
> call, e.g:
>
> {{{#!python
> Comment.objects.create(name='test', text='test')
> Comment.objects.bulk_create(Comment(name='test', text='test'),
> on_conflict=IGNORE)  # Does not throw an exception
> }}}
>
> All the databases we support have syntax for this, as well as
> updating/replacing fields.
New description:

 When using bulk_create it would be nice to support `ON CONFLICT DO
 NOTHING`, which allows existing rows to be included in the `bulk_create`
 call, e.g:

 {{{#!python
 Comment.objects.create(name='test', text='test')
 Comment.objects.bulk_create(Comment(name='test', text='test'),
 on_conflict=IGNORE)  # Does not throw an exception
 }}}

 All the databases we support have syntax for this, as well as
 updating/replacing fields.

 This is a spin-off of https://code.djangoproject.com/ticket/28641, just
 including the ON CONFLICT idea which is backwards-compatible and would be
 easier to implement than the others.

--

--
Ticket URL: <https://code.djangoproject.com/ticket/28668#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.98bd48667a58efb6303ee86f74ccf8f8%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add ON CONFLICT support to QuerySet.bulk_create() (was: Add support for ON CONFLICT to bulk_create)

Django
In reply to this post by Django
#28668: Add ON CONFLICT support to QuerySet.bulk_create()
-------------------------------------+-------------------------------------
     Reporter:  Tom                  |                    Owner:  nobody
         Type:  New feature          |                   Status:  new
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Tim Graham):

 * stage:  Unreviewed => Accepted


Old description:

> When using bulk_create it would be nice to support `ON CONFLICT DO
> NOTHING`, which allows existing rows to be included in the `bulk_create`
> call, e.g:
>
> {{{#!python
> Comment.objects.create(name='test', text='test')
> Comment.objects.bulk_create(Comment(name='test', text='test'),
> on_conflict=IGNORE)  # Does not throw an exception
> }}}
>
> All the databases we support have syntax for this, as well as
> updating/replacing fields.
>
> This is a spin-off of https://code.djangoproject.com/ticket/28641, just
> including the ON CONFLICT idea which is backwards-compatible and would be
> easier to implement than the others.
New description:

 When using bulk_create it would be nice to support `ON CONFLICT DO
 NOTHING`, which allows existing rows to be included in the `bulk_create`
 call, e.g:

 {{{#!python
 Comment.objects.create(name='test', text='test')
 Comment.objects.bulk_create(Comment(name='test', text='test'),
 on_conflict=IGNORE)  # Does not throw an exception
 }}}

 All the databases we support have syntax for this, as well as
 updating/replacing fields.

 This is a spin-off of #28641, just including the ON CONFLICT idea which is
 backwards-compatible and would be easier to implement than the others.

--

--
Ticket URL: <https://code.djangoproject.com/ticket/28668#comment:2>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.f6bc313a2102d980c2a439dac7ed2db6%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add ON CONFLICT support to QuerySet.bulk_create()

Django
In reply to this post by Django
#28668: Add ON CONFLICT support to QuerySet.bulk_create()
-------------------------------------+-------------------------------------
     Reporter:  Tom                  |                    Owner:  Tom
         Type:  New feature          |                   Status:  assigned
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Tom):

 * owner:  nobody => Tom
 * status:  new => assigned


Comment:

 If in the future we want to add support for `ON CONFLICT UPDATE` we should
 design the API with that in mind.

 Perhaps have `DO_NOTHING` as a sentinel object, so in the future we can
 pass in something else to describe updating (like a dictionary or
 somesuch?).

--
Ticket URL: <https://code.djangoproject.com/ticket/28668#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.bb13db609deb20d5600273ab29fe6750%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add ON CONFLICT support to QuerySet.bulk_create()

Django
In reply to this post by Django
#28668: Add ON CONFLICT support to QuerySet.bulk_create()
-------------------------------------+-------------------------------------
     Reporter:  Tom                  |                    Owner:  Tom
         Type:  New feature          |                   Status:  assigned
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Tom):

 * has_patch:  0 => 1


Comment:

 Initial PR: https://github.com/django/django/pull/9192

--
Ticket URL: <https://code.djangoproject.com/ticket/28668#comment:4>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.68af024ffc1af0b3e25713a509175267%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add ON CONFLICT support to QuerySet.bulk_create()

Django
In reply to this post by Django
#28668: Add ON CONFLICT support to QuerySet.bulk_create()
-------------------------------------+-------------------------------------
     Reporter:  Tom Forbes           |                    Owner:  Tom
                                     |  Forbes
         Type:  New feature          |                   Status:  assigned
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Дилян Палаузов):

 Add documentation what bulk_create() is supposed to return.

 I assume this not going to work, as Postgresql has
 can_return_ids_from_bulk_insert, but 'INSERT ON CONFLICT DO NOTHING
 RETURNING id' returns only the ids of the inserted rows, so the assert
 will fail.  In particular not inserted rows are not returned.
 {{{
 -                ids = self._batched_insert(objs_without_pk, fields,
 batch_size)
 +                ids = self._batched_insert(objs_without_pk, fields,
 batch_size, on_conflict=on_conflict)
                  if connection.features.can_return_ids_from_bulk_insert:
                      assert len(ids) == len(objs_without_pk)
 }}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28668#comment:5>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.092a76e2f2d7d438c39328227fd6d6fb%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add ON CONFLICT support to QuerySet.bulk_create()

Django
In reply to this post by Django
#28668: Add ON CONFLICT support to QuerySet.bulk_create()
-------------------------------------+-------------------------------------
     Reporter:  Tom Forbes           |                    Owner:  Tom
                                     |  Forbes
         Type:  New feature          |                   Status:  assigned
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Tom Forbes):

 Good point, I'll fix it up tomorrow. There are some other issues I've
 found tonight as well.

--
Ticket URL: <https://code.djangoproject.com/ticket/28668#comment:6>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.ac697a8ccaafaccbf8a1d5b0e92a5f25%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add ON CONFLICT support to QuerySet.bulk_create()

Django
In reply to this post by Django
#28668: Add ON CONFLICT support to QuerySet.bulk_create()
-------------------------------------+-------------------------------------
     Reporter:  Tom Forbes           |                    Owner:  Tom
                                     |  Forbes
         Type:  New feature          |                   Status:  assigned
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Дилян Палаузов):

 * cc: Дилян Палаузов (added)


--
Ticket URL: <https://code.djangoproject.com/ticket/28668#comment:7>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.2e9a1e44863d99066ff8365794a7c7bd%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add ON CONFLICT support to QuerySet.bulk_create()

Django
In reply to this post by Django
#28668: Add ON CONFLICT support to QuerySet.bulk_create()
-------------------------------------+-------------------------------------
     Reporter:  Tom Forbes           |                    Owner:  Tom
                                     |  Forbes
         Type:  New feature          |                   Status:  assigned
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Tom Forbes):

 I fixed that assertion by skipping returning ID's from the insert if
 `on_conflict` is used. They are incompatible, as you cannot tell which
 models where inserted and which where skipped.

--
Ticket URL: <https://code.djangoproject.com/ticket/28668#comment:8>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.d618a340b9e05aafc3e5bfcd863104e5%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add ON CONFLICT support to QuerySet.bulk_create()

Django
In reply to this post by Django
#28668: Add ON CONFLICT support to QuerySet.bulk_create()
-------------------------------------+-------------------------------------
     Reporter:  Tom Forbes           |                    Owner:  Tom
                                     |  Forbes
         Type:  New feature          |                   Status:  assigned
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Дилян Палаузов):

 For a table like {{{CREATE TABLE t (id SERIAL PRIMARY KEY, name
 VARCHAR(10) UNIQUE, comment VARCHAR(10));}}} it is possible to tell which
 models were inserted with a query like
 {{{
 WITH
   to_be_inserted AS (SELECT * FROM (VALUES ('name12', 'comment12'),
 ('name5', 'comment5'), ('name6', 'comment6')) as g(name, comment)),
   successfully_inserted AS (
       INSERT INTO t ("name", "comment" ) SELECT *
         FROM to_be_inserted ON CONFLICT DO NOTHING RETURNING *)
 SELECT s.id FROM to_be_inserted AS b
   LEFT JOIN successfully_inserted AS s ON (b.name = s.name AND b.comment =
 s.comment);
 }}}

 where {{{to_be_inserted}}} contains the values that are going into the
 database.  The returned column contains NULL for values that were
 presented in the database, and the id for the inserted rows.

 With the proposed changes
 {{{
 -                if connection.features.can_return_ids_from_bulk_insert:
 +                if connection.features.can_return_ids_from_bulk_insert
 and not on_conflict == 'ignore':
                       assert len(ids) == len(objs_without_pk)
                   for obj_without_pk, pk in zip(objs_without_pk, ids):
                       obj_without_pk.pk = pk
 }}}
 the for-loop will not work, as the amount of ids is not the same as the
 amount of obj_without_pk.

--
Ticket URL: <https://code.djangoproject.com/ticket/28668#comment:9>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.e50be14c0e4c2b49683b1c8840fc5ec8%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add ON CONFLICT support to QuerySet.bulk_create()

Django
In reply to this post by Django
#28668: Add ON CONFLICT support to QuerySet.bulk_create()
-------------------------------------+-------------------------------------
     Reporter:  Tom Forbes           |                    Owner:  Tom
                                     |  Forbes
         Type:  New feature          |                   Status:  assigned
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Chris Beck):

 Is there any consideration of this being included in a future 1.11.x
 release? I'd really like to be able to use bulk_create for a fire-hose of
 data that we're getting from a 3rd party who don't guarantee uniqueness so
 at the moment our only solution is to remove the unique_together attribute
 and enforce it at the reporting level.

--
Ticket URL: <https://code.djangoproject.com/ticket/28668#comment:10>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.fe1fe4ccd35370ed3b25f160ae090542%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add ON CONFLICT support to QuerySet.bulk_create()

Django
In reply to this post by Django
#28668: Add ON CONFLICT support to QuerySet.bulk_create()
-------------------------------------+-------------------------------------
     Reporter:  Tom Forbes           |                    Owner:  Tom
                                     |  Forbes
         Type:  New feature          |                   Status:  assigned
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Tim Graham):

 No, new features aren't backported to stable branches. See our
 [https://docs.djangoproject.com/en/dev/internals/release-process
 /#supported-versions supported versions policy].

--
Ticket URL: <https://code.djangoproject.com/ticket/28668#comment:11>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.fe8cad9778a6d3d78ad75891980dfb92%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add ON CONFLICT support to QuerySet.bulk_create()

Django
In reply to this post by Django
#28668: Add ON CONFLICT support to QuerySet.bulk_create()
-------------------------------------+-------------------------------------
     Reporter:  Tom Forbes           |                    Owner:  Tom
                                     |  Forbes
         Type:  New feature          |                   Status:  assigned
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Дилян Палаузов):

 * Attachment "on_conflict_postgresql.patch" added.

 this adds INSERT ON CONFLICT IGNORE to Postgresql, where only IDs of the
 actually inserted rows are returned

--
Ticket URL: <https://code.djangoproject.com/ticket/28668>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.ab1b15b298481c5b45068f5958c9f717%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add ON CONFLICT support to QuerySet.bulk_create()

Django
In reply to this post by Django
#28668: Add ON CONFLICT support to QuerySet.bulk_create()
-------------------------------------+-------------------------------------
     Reporter:  Tom Forbes           |                    Owner:  Tom
                                     |  Forbes
         Type:  New feature          |                   Status:  assigned
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Дилян Палаузов):

 Let's start with the most complex case: calling
 {{{.objects.bulk_create(..., on_conflict='ignore')}}} where the caller
 wants to determine which of the supplied objects were actually inserted
 (in order to call signals on them).  We can start with the attached
 minimalistic patch, which does this for Postgresql as only RDBMS allowing
 efficient handling of this most complex case, and see how to implement it
 in the other RDBMs.

 Unfortunately, the sequence counter is automatically increased for each
 object provided as parameter, even if at the end the object was already in
 the database.

 I propose extending the logic for bulk_create to send implicit post_save
 signals for Postgresql.

--
Ticket URL: <https://code.djangoproject.com/ticket/28668#comment:12>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.8f705b44f0a7b0e8bcc505c0e2eeca43%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add ON CONFLICT support to QuerySet.bulk_create()

Django
In reply to this post by Django
#28668: Add ON CONFLICT support to QuerySet.bulk_create()
-------------------------------------+-------------------------------------
     Reporter:  Tom Forbes           |                    Owner:  Tom
                                     |  Forbes
         Type:  New feature          |                   Status:  assigned
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Дилян Палаузов):

 * Attachment "on_conflict_postgresql.patch" added.

 this adds INSERT ON CONFLICT IGNORE to Postgresql, where only IDs of the
 actually inserted rows are returned

--
Ticket URL: <https://code.djangoproject.com/ticket/28668>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.ff633f254f53bd32f0be2df4ccfe0810%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add ON CONFLICT support to QuerySet.bulk_create()

Django
In reply to this post by Django
#28668: Add ON CONFLICT support to QuerySet.bulk_create()
-------------------------------------+-------------------------------------
     Reporter:  Tom Forbes           |                    Owner:  Tom
                                     |  Forbes
         Type:  New feature          |                   Status:  assigned
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Дилян Палаузов):

 * Attachment "on_conflict_postgresql.patch" added.

 this adds INSERT ON CONFLICT IGNORE to Postgresql, where only IDs of the
 actually inserted rows are returned

--
Ticket URL: <https://code.djangoproject.com/ticket/28668>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.37c31cf05425f550b22207223aca4823%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add ON CONFLICT support to QuerySet.bulk_create()

Django
In reply to this post by Django
#28668: Add ON CONFLICT support to QuerySet.bulk_create()
-------------------------------------+-------------------------------------
     Reporter:  Tom Forbes           |                    Owner:  Tom
                                     |  Forbes
         Type:  New feature          |                   Status:  assigned
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Дилян Палаузов):

 * Attachment "on_conflict_postgresql_signals_ids.patch" added.

 In addition to the other file, this one add support for sending post_save
 for the newly created objects and retrieving the ids of the objects
 provided as parameters to bulk_create, that already existed in the
 database.

--
Ticket URL: <https://code.djangoproject.com/ticket/28668>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.1f71c00361aae00bae7662c47080091d%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add ON CONFLICT support to QuerySet.bulk_create()

Django
In reply to this post by Django
#28668: Add ON CONFLICT support to QuerySet.bulk_create()
-------------------------------------+-------------------------------------
     Reporter:  Tom Forbes           |                    Owner:  Tom
                                     |  Forbes
         Type:  New feature          |                   Status:  assigned
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Tom Forbes):

 Hey, I recommend consolidating your patches into a single PR on Github and
 linking it here, that's the development process Django now follows.

--
Ticket URL: <https://code.djangoproject.com/ticket/28668#comment:13>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.0661cc670d464ef9215aa565a55275b2%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add ON CONFLICT support to QuerySet.bulk_create()

Django
In reply to this post by Django
#28668: Add ON CONFLICT support to QuerySet.bulk_create()
-------------------------------------+-------------------------------------
     Reporter:  Tom Forbes           |                    Owner:  Tom
                                     |  Forbes
         Type:  New feature          |                   Status:  assigned
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Дилян Палаузов):

 * Attachment "on_conflict_ignore.patch" added.

 The on_conflict_ignore.patch adds on_conflict='ignore' to
 QuerySet.bulk_create and empowers bulk_create optionally to send post_save
 signals, when the ID of the inserted object is known to bulk_create.  For
 postgresql bulk_create offers in addition to retrieve the IDs of the newly
 inserted objects, when using on_conflict='ignore' and, with a second
 query, to find the PKs pf the supplied objs that were already in the
 databse.  When supported by the backend (so not Oracle),
 on_conflict='ignore' is added to bulk_create() in
 django.db.models.fields.create_forward_many_to_many_manager.ManyRelatedManager._add_items,
 making ManyRelatedManager.add() thread-safe, possibly resolving #19544.  I
 have the feeling reset_sequence = True doesn't work as expected, as the
 IDs are random.  This patch integrated the changes of om Forbes and
 previous changes of me.

--
Ticket URL: <https://code.djangoproject.com/ticket/28668>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.eef8d2ba2e5ed1b30d31df9f05cd18ea%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #28668: Add ON CONFLICT support to QuerySet.bulk_create()

Django
In reply to this post by Django
#28668: Add ON CONFLICT support to QuerySet.bulk_create()
-------------------------------------+-------------------------------------
     Reporter:  Tom Forbes           |                    Owner:  Tom
                                     |  Forbes
         Type:  New feature          |                   Status:  assigned
    Component:  Database layer       |                  Version:  master
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Дилян Палаузов):

 * Attachment "on_conflict_ignore.patch" added.

 The on_conflict_ignore.patch adds on_conflict='ignore' to
 QuerySet.bulk_create and empowers bulk_create optionally to send post_save
 signals, when the ID of the inserted object is known to bulk_create. For
 postgresql bulk_create offers in addition to retrieve the IDs of the newly
 inserted objects, when using on_conflict='ignore' and, with a second
 query, to find the PKs pf the supplied objs that were already in the
 databse. When supported by the backend (so not Oracle),
 on_conflict='ignore' is added to bulk_create() in
 django.db.models.fields.create_forward_many_to_many_manager.ManyRelatedManager._add_items,
 making ManyRelatedManager.add() thread-safe, possibly resolving #19544.

--
Ticket URL: <https://code.djangoproject.com/ticket/28668>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/061.f895f69068dd432e85384491266b127f%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
12