problems with get_or_create

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

problems with get_or_create

Kenneth Gonsalves-2
hi,

I have a model like this:

name - unique
slno - not null
mode - not null

If the instance does not exist and I give all three values to get_or_create,
it works

if the instance exists, and I give values for slno and mode which are different
from the existing ones - I get a programming error as get_or_create tries to
create a new instance and falls foul of the unique constraint on name. It
works if I only give the value for name - it gets the instance and then I can
add the new values and save

if the instance does not exist and I only give the value for name, I get a
programming error that the not null constraints on slno and mode are violated.

so it looks to me that get_or_create will only work reliably when there are no
not null or unique constraints in the model. Or am I missing something?
--
regards
Kenneth Gonsalves
Senior Associate
NRC-FOSS
http://certificate.nrcfoss.au-kbc.org.in

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: problems with get_or_create

Tom Evans-3
On Thu, Mar 25, 2010 at 5:52 AM, Kenneth Gonsalves <[hidden email]> wrote:

> hi,
>
> I have a model like this:
>
> name - unique
> slno - not null
> mode - not null
>
> If the instance does not exist and I give all three values to get_or_create,
> it works
>
> if the instance exists, and I give values for slno and mode which are different
> from the existing ones - I get a programming error as get_or_create tries to
> create a new instance and falls foul of the unique constraint on name. It
> works if I only give the value for name - it gets the instance and then I can
> add the new values and save
>
> if the instance does not exist and I only give the value for name, I get a
> programming error that the not null constraints on slno and mode are violated.
>
> so it looks to me that get_or_create will only work reliably when there are no
> not null or unique constraints in the model. Or am I missing something?

I think you are missing the optional kwarg 'initial', which is a
dictionary of initial values to use when the instance does not exist.
So not

foo, created = Foo.objects.get_or_create(name=name,
                                         sino=sino,mode=mode)

but

foo, created = Foo.objects.get_or_create(name=name,
                                initial={'sino': sino, 'mode': mode, })

Cheers

Tom

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: problems with get_or_create

Daniel Roseman-2
On Mar 25, 9:32 am, Tom Evans <[hidden email]> wrote:

> On Thu, Mar 25, 2010 at 5:52 AM, Kenneth Gonsalves <[hidden email]> wrote:
> > hi,
>
> > I have a model like this:
>
> > name - unique
> > slno - not null
> > mode - not null
>
> > If the instance does not exist and I give all three values to get_or_create,
> > it works
>
> > if the instance exists, and I give values for slno and mode which are different
> > from the existing ones - I get a programming error as get_or_create tries to
> > create a new instance and falls foul of the unique constraint on name. It
> > works if I only give the value for name - it gets the instance and then I can
> > add the new values and save
>
> > if the instance does not exist and I only give the value for name, I get a
> > programming error that the not null constraints on slno and mode are violated.
>
> > so it looks to me that get_or_create will only work reliably when there are no
> > not null or unique constraints in the model. Or am I missing something?
>
> I think you are missing the optional kwarg 'initial', which is a
> dictionary of initial values to use when the instance does not exist.
> So not
>
> foo, created = Foo.objects.get_or_create(name=name,
>                                          sino=sino,mode=mode)
>
> but
>
> foo, created = Foo.objects.get_or_create(name=name,
>                                 initial={'sino': sino, 'mode': mode, })
>
> Cheers
>
> Tom

Actually the kwarg is 'defaults', not 'initial'.

See http://docs.djangoproject.com/en/1.1/ref/models/querysets/#get-or-create-kwargs
--
DR.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: problems with get_or_create

Tom Evans-3
On Thu, Mar 25, 2010 at 10:29 AM, Daniel Roseman <[hidden email]> wrote:
>
> Actually the kwarg is 'defaults', not 'initial'.
>
> See http://docs.djangoproject.com/en/1.1/ref/models/querysets/#get-or-create-kwargs
> --
> DR.
>

Indeed :/

coffee++

Cheers

Tom

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: problems with get_or_create

Malcolm Tredinnick
In reply to this post by Kenneth Gonsalves-2
On Thu, 2010-03-25 at 11:22 +0530, Kenneth Gonsalves wrote:

> hi,
>
> I have a model like this:
>
> name - unique
> slno - not null
> mode - not null
>
> If the instance does not exist and I give all three values to get_or_create,
> it works
>
> if the instance exists, and I give values for slno and mode which are different
> from the existing ones - I get a programming error as get_or_create tries to
> create a new instance and falls foul of the unique constraint on name. It
> works if I only give the value for name - it gets the instance and then I can
> add the new values and save

I don't understand what you are saying here. If the instance exists (as
in, the filter parameters passed to get would return a single item),
that instance is returned and the default parameters are never
considered. So the values you pass for slno and mode are irrelevant. If
the equivalent get() call would not return anything, then the instance
does not exist and it's clear that the default parameter must contain
correct data that can be used to create a new object -- including not
violating any unique constraints. Perhaps you could show the code call
you are making, rather than writing it out as a sentence.

In short: get_or_create() first runs the get(). If that returns
something, it is what is returned. If the get() fails with no object
being present, then the instance DOES NOT exist and a new one is
created, with all the same requirements as if you'd created a new object
from scratch (e.g. no unique constraint violations).

Malcolm


--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: problems with get_or_create

Tom Evans-3
On Thu, Mar 25, 2010 at 3:47 PM, Malcolm Tredinnick
<[hidden email]> wrote:

> On Thu, 2010-03-25 at 11:22 +0530, Kenneth Gonsalves wrote:
>> hi,
>>
>> I have a model like this:
>>
>> name - unique
>> slno - not null
>> mode - not null
>>
>> If the instance does not exist and I give all three values to get_or_create,
>> it works
>>
>> if the instance exists, and I give values for slno and mode which are different
>> from the existing ones - I get a programming error as get_or_create tries to
>> create a new instance and falls foul of the unique constraint on name. It
>> works if I only give the value for name - it gets the instance and then I can
>> add the new values and save
>
> I don't understand what you are saying here. If the instance exists (as
> in, the filter parameters passed to get would return a single item),
> that instance is returned and the default parameters are never
> considered. So the values you pass for slno and mode are irrelevant. If
> the equivalent get() call would not return anything, then the instance
> does not exist and it's clear that the default parameter must contain
> correct data that can be used to create a new object -- including not
> violating any unique constraints. Perhaps you could show the code call
> you are making, rather than writing it out as a sentence.
>
> In short: get_or_create() first runs the get(). If that returns
> something, it is what is returned. If the get() fails with no object
> being present, then the instance DOES NOT exist and a new one is
> created, with all the same requirements as if you'd created a new object
> from scratch (e.g. no unique constraint violations).
>
> Malcolm
>
>

He is describing this situation:

>>> User.objects.get(username='[hidden email]')
<User: [hidden email]>
>>> User.objects.get_or_create(username='[hidden email]')
(<User: [hidden email]>, False)
>>> User.objects.get(username='[hidden email]').first_name
u'Tom'
>>> User.objects.get(username='[hidden email]', first_name='Thomas')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "....site-packages/django/db/models/manager.py", line 119, in get
    return self.get_query_set().get(*args, **kwargs)
  File "....django/db/models/query.py", line 305, in get
    % self.model._meta.object_name)
DoesNotExist: User matching query does not exist.
>>> User.objects.get_or_create(username='[hidden email]', first_name='Thomas')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "....site-packages/django/db/models/manager.py", line 122, in
get_or_create
    return self.get_query_set().get_or_create(**kwargs)
  File "....site-packages/django/db/models/query.py", line 343, in get_or_create
    raise e
IntegrityError: (1062, "Duplicate entry '' for key 2")

IE, where Model.objects.get(**kwargs) does not return an object, but
Model.objects.create(**kwargs) would fail, which is fixed by knowing
about the 'defaults' kwarg to get_or_create:

>>> User.objects.get_or_create(username='[hidden email]',
... defaults={'first_name': 'Thomas', })
(<User: [hidden email]>, False)

Cheers

Tom

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: problems with get_or_create

Malcolm Tredinnick
On Thu, 2010-03-25 at 15:58 +0000, Tom Evans wrote:
[...]

>
> He is describing this situation:
>
> >>> User.objects.get(username='[hidden email]')
> <User: [hidden email]>
> >>> User.objects.get_or_create(username='[hidden email]')
> (<User: [hidden email]>, False)
> >>> User.objects.get(username='[hidden email]').first_name
> u'Tom'
> >>> User.objects.get(username='[hidden email]', first_name='Thomas')
> Traceback (most recent call last):
>   File "<console>", line 1, in <module>
>   File "....site-packages/django/db/models/manager.py", line 119, in get
>     return self.get_query_set().get(*args, **kwargs)
>   File "....django/db/models/query.py", line 305, in get
>     % self.model._meta.object_name)
> DoesNotExist: User matching query does not exist.
> >>> User.objects.get_or_create(username='[hidden email]', first_name='Thomas')
> Traceback (most recent call last):
>   File "<console>", line 1, in <module>
>   File "....site-packages/django/db/models/manager.py", line 122, in
> get_or_create
>     return self.get_query_set().get_or_create(**kwargs)
>   File "....site-packages/django/db/models/query.py", line 343, in get_or_create
>     raise e
> IntegrityError: (1062, "Duplicate entry '' for key 2")

No, he's not, because Kenneth said that the instance exists. In your
example, the instance doesn't exist and thus a new object is attempted
to be created. The restrictions at that point are the same as when
creating a new object from scratch, as noted in my reply. The caller of
get_or_create() will always know what the various constraints are on the
model and must allow for that in their code.

Regards,
Malcolm


--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: problems with get_or_create

Tom Evans-3
On Thu, Mar 25, 2010 at 4:23 PM, Malcolm Tredinnick
<[hidden email]> wrote:
> No, he's not, because Kenneth said that the instance exists. In your

Actually, he described both situations where the instance exists, but
with different values for the non-key attributes of the model, and
where the instance doesn't exist.
pedant++

> example, the instance doesn't exist and thus a new object is attempted
> to be created. The restrictions at that point are the same as when
> creating a new object from scratch, as noted in my reply. The caller of
> get_or_create() will always know what the various constraints are on the
> model and must allow for that in their code.
>
> Regards,
> Malcolm
>
>

Well, quite. His problem was that he wasn't allowing for the
constraints in his code because he was unaware of the 'defaults' kwarg
to get_or_create(). With that addition, his problems when an instance
with the same unique key exists, and when no instance exists are both
fixed.

Cheers

Tom

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: problems with get_or_create

Kenneth Gonsalves-2
In reply to this post by Malcolm Tredinnick
On Thursday 25 Mar 2010 9:17:12 pm Malcolm Tredinnick wrote:

> I don't understand what you are saying here. If the instance exists (as
> in, the filter parameters passed to get would return a single item),
> that instance is returned and the default parameters are never
> considered. So the values you pass for slno and mode are irrelevant. If
> the equivalent get() call would not return anything, then the instance
> does not exist and it's clear that the default parameter must contain
> correct data that can be used to create a new object -- including not
> violating any unique constraints. Perhaps you could show the code call
> you are making, rather than writing it out as a sentence.
>
> In short: get_or_create() first runs the get(). If that returns
> something, it is what is returned. If the get() fails with no object
> being present, then the instance DOES NOT exist and a new one is
> created, with all the same requirements as if you'd created a new object
> from scratch (e.g. no unique constraint violations).
>

sorry for not replying to this - for some reason all your recent mails have
gone to the spam folder! I only became aware of this mail when I saw a reply
to it. Here is what I mean - A simple model with two fields, 'name' is unique,
and 'num' is not null:

>>> Tm.objects.get_or_create(name='me',num=1)                                                                                
(<Tm: me>, True)                                                                                                              
>>> Tm.objects.get_or_create(name='me',num=2)                                                                                
Traceback (most recent call last):                                                                                          
  File "<console>", line 1, in <module>                                                                                      
  File "/usr/lib/python2.6/django/db/models/manager.py", line 135, in
get_or_create                                          
    return self.get_query_set().get_or_create(**kwargs)                                                                      
  File "/usr/lib/python2.6/django/db/models/query.py", line 375, in
get_or_create                                            
    transaction.savepoint_rollback(sid, using=self.db)                                                                      
  File "/usr/lib/python2.6/django/db/transaction.py", line 242, in
savepoint_rollback                                        
    connection._savepoint_rollback(sid)                                                                                      
  File "/usr/lib/python2.6/django/db/backends/__init__.py", line 61, in
_savepoint_rollback                                  
    self.cursor().execute(self.ops.savepoint_rollback_sql(sid))
  File "/usr/lib/python2.6/django/db/backends/__init__.py", line 75, in cursor
    cursor = self._cursor()
  File "/usr/lib/python2.6/django/db/backends/postgresql/base.py", line 149,
in _cursor
    cursor.execute("SET client_encoding to 'UNICODE'")
ProgrammingError: ERROR:  current transaction is aborted, commands ignored
until end of transaction block

SET client_encoding to 'UNICODE'
--
regards
Kenneth Gonsalves
Senior Associate
NRC-FOSS
http://certificate.nrcfoss.au-kbc.org.in

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: problems with get_or_create

Karen Tracey-2
On Fri, Mar 26, 2010 at 1:49 AM, Kenneth Gonsalves <[hidden email]> wrote:
Here is what I mean - A simple model with two fields, 'name' is unique,
and 'num' is not null:

>>> Tm.objects.get_or_create(name='me',num=1)
(<Tm: me>, True)
>>> Tm.objects.get_or_create(name='me',num=2)

If you want get_or_create to return the instance with name 'me', regardless of the value of num which that instance may have, then you need to include the num value in the defaults argument to get_or_create, NOT as a bare keyword argument. From http://docs.djangoproject.com/en/dev/ref/models/querysets/#get-or-create-kwargs:

"Any keyword arguments passed to get_or_create() -- except an optional one called defaults -- will be used in a get() call."

So with the arguments as you have passed them, get_or_create is looking for a model with name='me' and num=2, and since one does not exist, it is going to go down the path of attempting to create an instance with those values. Since you have specified at the database level that name must be unique, that is not going to work: attempting to create a 2nd row with an already-existing name value is going to raise an IntegrityError.

To accomplish what you want, you need to move num into the defaults argument, so that it will not be used on the get() lookup, but will be used if the get fails to find a match and an object needs to be created. That is:

Tm.objects.get_or_create(name='me', defaults={'num': 2})


Traceback (most recent call last):
[snip]
  File "/usr/lib/python2.6/django/db/backends/__init__.py", line 61, in
_savepoint_rollback
   self.cursor().execute(self.ops.savepoint_rollback_sql(sid))
 File "/usr/lib/python2.6/django/db/backends/__init__.py", line 75, in cursor
   cursor = self._cursor()
 File "/usr/lib/python2.6/django/db/backends/postgresql/base.py", line 149,
in _cursor
   cursor.execute("SET client_encoding to 'UNICODE'")
ProgrammingError: ERROR:  current transaction is aborted, commands ignored
until end of transaction block

SET client_encoding to 'UNICODE'

Here you are running into a bug in the postgresql backend.  Is there some reason you are using the old psycopg1 backend instead of the psycopg2 one? The latter is really a much better choice at this point. 

Anyway, what has happened here is the get_or_create code has gone down the path of trying to create the new object, which has failed because of the unique constraint on name. It is trying to rollback to the previously-created savepoint, but that is failing due to a bug that was fixed yesterday in r12848 (trunk) and r12849 (1.1.X branch).  So to avoid this you could either simply switch to the psycopg2 backend (which did not have the bug) or update to a latest trunk or 1.1.X Django code.

Note when you fix the call to get_or_create to properly specify which of the values must be used on the get() part of get_or_create(), you won't be going down this path as you are now, so you likely won't see this exception during basic testing even if you don't change backends or update to latest Django code. However, you could still go down this path legitimately -- if two threads "simultaneously" attempt to create an object with the same name, both may see that it does not exist and try to create it. Only one will succeed, and the other will go down this path. You'll need this path to work properly in order for the 2nd thread to be able to retrieve the object created by the other thread. So even if you notice that fixing the get_or_create call makes this other exception go away, you should still update the code you are using, if you want the call to behave correctly in all situations.

Karen

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Reply | Threaded
Open this post in threaded view
|

Re: problems with get_or_create

Kenneth Gonsalves-2
thanks for the detailed reply - I now understand the issue. And yes, I did
read the documentation, but did not absorb it properly. Use of pyscopg1 is due
to inertia. When I was first using django I had a lot of legacy code using
psycopg1 and it was too much of a bother to migrate. That code is no longer my
responsibility now, so I will probably shift. Am top posting as there is
nothing specific to reply to. Thanks again.

On Friday 26 Mar 2010 4:55:39 pm Karen Tracey wrote:

> On Fri, Mar 26, 2010 at 1:49 AM, Kenneth Gonsalves <[hidden email]>wrote:
> > Here is what I mean - A simple model with two fields, 'name' is unique,
> >
> > and 'num' is not null:
> > >>> Tm.objects.get_or_create(name='me',num=1)
> >
> > (<Tm: me>, True)
> >
> > >>> Tm.objects.get_or_create(name='me',num=2)
>
> If you want get_or_create to return the instance with name 'me', regardless
> of the value of num which that instance may have, then you need to include
> the num value in the defaults argument to get_or_create, NOT as a bare
> keyword argument. From
> http://docs.djangoproject.com/en/dev/ref/models/querysets/#get-or-create-kw
> args
>
>
> "Any keyword arguments passed to get_or_create() -- except an optional one
> called defaults -- will be used in a get() call."
>
> So with the arguments as you have passed them, get_or_create is looking for
> a model with name='me' and num=2, and since one does not exist, it is going
> to go down the path of attempting to create an instance with those values.
> Since you have specified at the database level that name must be unique,
> that is not going to work: attempting to create a 2nd row with an
> already-existing name value is going to raise an IntegrityError.
>
> To accomplish what you want, you need to move num into the defaults
> argument, so that it will not be used on the get() lookup, but will be used
> if the get fails to find a match and an object needs to be created. That
>  is:
>
> Tm.objects.get_or_create(name='me', defaults={'num': 2})
>
> Traceback (most recent call last):
> > [snip]
> >   File "/usr/lib/python2.6/django/db/backends/__init__.py", line 61, in
> > _savepoint_rollback
> >    self.cursor().execute(self.ops.savepoint_rollback_sql(sid))
> >  File "/usr/lib/python2.6/django/db/backends/__init__.py", line 75, in
> > cursor
> >    cursor = self._cursor()
> >  File "/usr/lib/python2.6/django/db/backends/postgresql/base.py", line
> > 149, in _cursor
> >    cursor.execute("SET client_encoding to 'UNICODE'")
> > ProgrammingError: ERROR:  current transaction is aborted, commands
> > ignored until end of transaction block
> >
> > SET client_encoding to 'UNICODE'
>
> Here you are running into a bug in the postgresql backend.  Is there some
> reason you are using the old psycopg1 backend instead of the psycopg2 one?
> The latter is really a much better choice at this point.
>
> Anyway, what has happened here is the get_or_create code has gone down the
> path of trying to create the new object, which has failed because of the
> unique constraint on name. It is trying to rollback to the
> previously-created savepoint, but that is failing due to a bug that was
> fixed yesterday in r12848 (trunk) and r12849 (1.1.X branch).  So to avoid
> this you could either simply switch to the psycopg2 backend (which did not
> have the bug) or update to a latest trunk or 1.1.X Django code.
>
> Note when you fix the call to get_or_create to properly specify which of
>  the values must be used on the get() part of get_or_create(), you won't be
>  going down this path as you are now, so you likely won't see this
>  exception during basic testing even if you don't change backends or update
>  to latest Django code. However, you could still go down this path
>  legitimately -- if two threads "simultaneously" attempt to create an
>  object with the same name, both may see that it does not exist and try to
>  create it. Only one will succeed, and the other will go down this path.
>  You'll need this path to work properly in order for the 2nd thread to be
>  able to retrieve the object created by the other thread. So even if you
>  notice that fixing the get_or_create call makes this other exception go
>  away, you should still update the code you are using, if you want the call
>  to behave correctly in all situations.
>
> Karen
>

--
regards
Kenneth Gonsalves
Senior Associate
NRC-FOSS
http://certificate.nrcfoss.au-kbc.org.in

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

Reply | Threaded
Open this post in threaded view
|

Re: problems with get_or_create

Karen Tracey-2
On Sat, Mar 27, 2010 at 2:54 AM, Kenneth Gonsalves <[hidden email]> wrote:
thanks for the detailed reply - I now understand the issue. And yes, I did
read the documentation, but did not absorb it properly.

A number of people have tripped over this sort of use of get_or_create; there is an open ticket (#12579) to improve the doc here.
 
Karen

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.