Dynamically adding a field to a model - via contribute_to_class

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Dynamically adding a field to a model - via contribute_to_class

gernot.cseh
Hi,

I don't know if crossposting with stackoverflow is allowed here but essentially my problem is explained in https://stackoverflow.com/questions/46162104/django-dynamic-model-fields-and-migrations.
What I want to create is a custom ModelField of type DecimalField, that dynamically adds another ModelField (CharField) to the source model. As far as I can tell, this is nicely explained in https://blog.elsdoerfer.name/2008/01/08/fuzzydates-or-one-django-model-field-multiple-database-columns/.

Let's assume I start with a fresh project and app and add this code to models.py:

from django.db import models
from django.db.models import signals


_currency_field_name
= lambda name: '{}_extension'.format(name)


class PriceField(models.DecimalField):

   
def contribute_to_class(self, cls, name):
       
# add the extra currency field (CharField) to the class
       
if not cls._meta.abstract:
            currency_field
= models.CharField(
                max_length
=3,
                editable
=False,
               
null=True,
                blank
=True
           
)
            cls
.add_to_class(_currency_field_name(name), currency_field)
       
# add the original price field (DecimalField) to the class
       
super().contribute_to_class(cls, name)

       
# TODO: set the descriptor
       
# setattr(cls, self.name, FooDescriptor(self))


class FooModel(models.Model):

    price
= PriceField('agrhhhhh', decimal_places=3, max_digits=10, blank=True, null=True)

If I then call ./manage.py makemigrations the following migration file for the app is created:

# Generated by Django 1.11.4 on 2017-09-11 18:02
from __future__ import unicode_literals

from django.db import migrations, models
import testing.models


class Migration(migrations.Migration):

    initial
= True

    dependencies
= [
   
]

    operations
= [
        migrations
.CreateModel(
            name
='FooModel',
            fields
=[
               
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
               
('price', testing.models.PriceField(blank=True, decimal_places=3, max_digits=10, null=True, verbose_name='agrhhhhh')),
               
('price_extension', models.CharField(blank=True, editable=False, max_length=3, null=True)),
           
],
       
),
   
]

All good, as far as I can tell. The problem comes up if I then call ./manage.py migrate which errors with the following exception:

./manage.py migrate testing
Operations to perform:
 
Apply all migrations: testing
Running migrations:
 
Applying testing.0001_initial...Traceback (most recent call last):
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/utils.py", line 63, in execute
   
return self.cursor.execute(sql)
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
   
return Database.Cursor.execute(self, query)
sqlite3
.OperationalError: duplicate column name: price_extension

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 
File "./manage.py", line 22, in <module>
    execute_from_command_line
(sys.argv)
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility
.execute()
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/core/management/__init__.py", line 355, in execute
   
self.fetch_command(subcommand).run_from_argv(self.argv)
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
   
self.execute(*args, **cmd_options)
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output
= self.handle(*args, **options)
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial
=fake_initial,
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state
= self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state
= self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state
= migration.apply(state, schema_editor)
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation
.database_forwards(self.app_label, schema_editor, old_state, project_state)
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/migrations/operations/models.py", line 97, in database_forwards
    schema_editor
.create_model(model)
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/base/schema.py", line 303, in create_model
   
self.execute(sql, params or None)
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/base/schema.py", line 120, in execute
    cursor
.execute(sql, params)
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/utils.py", line 80, in execute
   
return super(CursorDebugWrapper, self).execute(sql, params)
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/cachalot/monkey_patch.py", line 113, in inner
   
out = original(cursor, sql, *args, **kwargs)
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/utils.py", line 65, in execute
   
return self.cursor.execute(sql, params)
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__
    six
.reraise(dj_exc_type, dj_exc_value, traceback)
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
   
raise value.with_traceback(tb)
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/utils.py", line 63, in execute
   
return self.cursor.execute(sql)
 
File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
   
return Database.Cursor.execute(self, query)
django
.db.utils.OperationalError: duplicate column name: price_extension

As said, I have a fresh project with no exisiting DB and tables so far. Why is it complaining that there allready exista a column named "price_extension"?. The migration file only contains one field called "price_extension"?

A sample project can be cloned from https://github.com/hetsch/django_testing. 

Thank you a lot for your help!

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/9ca1469c-abcb-4b30-a383-f4654f1705b2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Dynamically adding a field to a model - via contribute_to_class

Michal Petrucha-3
On Tue, Sep 12, 2017 at 12:34:26AM -0700, [hidden email] wrote:

> Hi,
>
> I don't know if crossposting with stackoverflow is allowed here but
> essentially my problem is explained
> in https://stackoverflow.com/questions/46162104/django-dynamic-model-fields-and-migrations.
> What I want to create is a custom ModelField of type DecimalField, that
> dynamically adds another ModelField (CharField) to the source model. As far
> as I can tell, this is nicely explained
> in https://blog.elsdoerfer.name/2008/01/08/fuzzydates-or-one-django-model-field-multiple-database-columns/.
>
> Let's assume I start with a fresh project and app and add this code to
> models.py:
>
> from django.db import models
> from django.db.models import signals
>
>
> _currency_field_name = lambda name: '{}_extension'.format(name)
>
>
> class PriceField(models.DecimalField):
>
>     def contribute_to_class(self, cls, name):
>         # add the extra currency field (CharField) to the class
>         if not cls._meta.abstract:
>             currency_field = models.CharField(
>                 max_length=3,
>                 editable=False,
>                 null=True,
>                 blank=True
>             )
>             cls.add_to_class(_currency_field_name(name), currency_field)
>         # add the original price field (DecimalField) to the class
>         super().contribute_to_class(cls, name)
>
>         # TODO: set the descriptor
>         # setattr(cls, self.name, FooDescriptor(self))
>
>
> class FooModel(models.Model):
>
>     price = PriceField('agrhhhhh', decimal_places=3, max_digits=10, blank=True, null=True)
>
>
> If I then call *./manage.py makemigrations* the following migration file
> for the app is created:
>
> # Generated by Django 1.11.4 on 2017-09-11 18:02
> from __future__ import unicode_literals
>
> from django.db import migrations, models
> import testing.models
>
>
> class Migration(migrations.Migration):
>
>     initial = True
>
>     dependencies = [
>     ]
>
>     operations = [
>         migrations.CreateModel(
>             name='FooModel',
>             fields=[
>                 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
>                 ('price', testing.models.PriceField(blank=True, decimal_places=3, max_digits=10, null=True, verbose_name='agrhhhhh')),
>                 ('price_extension', models.CharField(blank=True, editable=False, max_length=3, null=True)),
>             ],
>         ),
>     ]
>
>
> All good, as far as I can tell. The problem comes up if I then call *./manage.py
> migrate* which errors with the following exception:
>
> ./manage.py migrate testing
> Operations to perform:
>   Apply all migrations: testing
> Running migrations:
>   Applying testing.0001_initial...Traceback (most recent call last):
>   File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/utils.py", line 63, in execute
>     return self.cursor.execute(sql)
>   File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
>     return Database.Cursor.execute(self, query)
> sqlite3.OperationalError: duplicate column name: price_extension
>
[...]
> As said, I have a fresh project with no exisiting DB and tables so far. Why
> is it complaining that there allready exista a column named
> "price_extension"?. The migration file only contains one field called
> "price_extension"?

I'm not quite certain about this, but I believe that when the
migration runner reconstructs a version of your FooModel, it iterates
over the list of fields, and adds each of them to the reconstructed
model class one by one. So what happens is that your custom PriceField
gets added, which in turn creates its price_extension field, and then
next thing, the migration runner adds the price_extension field one
more time. So you end up with two instances of the price_extension
field on the same model, and when eventually the create_model schema
operation is executed, it adds each of them as another column to the
table, which is obviously wrong.

As for what you could do to avoid this situation – I'd suggest that
you add an extra argument to PriceField which tells the field that it
shouldn't create the additional extension field. Then, you implement
the deconstruct method on the field, and include this additional flag.
That way, auto-detected migrations will include the extra argument to
the field instance, and there shouldn't be any duplication of the
extension field on the resulting models reconstructed by migrations.

Good luck,

Michal

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/20170913120505.GO8762%40koniiiik.org.
For more options, visit https://groups.google.com/d/optout.

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Dynamically adding a field to a model - via contribute_to_class

Melvyn Sopacua
If I may suggest a better approach:

- Make the code as ImageField does (see it's width_field and height_field)
- Then focus on writing the migration for this field

Unfortunately, I don't see a way for custom fields to provide hints to
the migrations framework. So a field that automatically creates
another field is hard to deal with.

Another different approach is to use a custom ForeignKey [1]. Then you
create only one field, but at the cost of a join.

[1] https://github.com/furious-luke/django-address/blob/master/address/models.py#L289

On Wed, Sep 13, 2017 at 2:05 PM, Michal Petrucha
<[hidden email]> wrote:

> On Tue, Sep 12, 2017 at 12:34:26AM -0700, [hidden email] wrote:
>> Hi,
>>
>> I don't know if crossposting with stackoverflow is allowed here but
>> essentially my problem is explained
>> in https://stackoverflow.com/questions/46162104/django-dynamic-model-fields-and-migrations.
>> What I want to create is a custom ModelField of type DecimalField, that
>> dynamically adds another ModelField (CharField) to the source model. As far
>> as I can tell, this is nicely explained
>> in https://blog.elsdoerfer.name/2008/01/08/fuzzydates-or-one-django-model-field-multiple-database-columns/.
>>
>> Let's assume I start with a fresh project and app and add this code to
>> models.py:
>>
>> from django.db import models
>> from django.db.models import signals
>>
>>
>> _currency_field_name = lambda name: '{}_extension'.format(name)
>>
>>
>> class PriceField(models.DecimalField):
>>
>>     def contribute_to_class(self, cls, name):
>>         # add the extra currency field (CharField) to the class
>>         if not cls._meta.abstract:
>>             currency_field = models.CharField(
>>                 max_length=3,
>>                 editable=False,
>>                 null=True,
>>                 blank=True
>>             )
>>             cls.add_to_class(_currency_field_name(name), currency_field)
>>         # add the original price field (DecimalField) to the class
>>         super().contribute_to_class(cls, name)
>>
>>         # TODO: set the descriptor
>>         # setattr(cls, self.name, FooDescriptor(self))
>>
>>
>> class FooModel(models.Model):
>>
>>     price = PriceField('agrhhhhh', decimal_places=3, max_digits=10, blank=True, null=True)
>>
>>
>> If I then call *./manage.py makemigrations* the following migration file
>> for the app is created:
>>
>> # Generated by Django 1.11.4 on 2017-09-11 18:02
>> from __future__ import unicode_literals
>>
>> from django.db import migrations, models
>> import testing.models
>>
>>
>> class Migration(migrations.Migration):
>>
>>     initial = True
>>
>>     dependencies = [
>>     ]
>>
>>     operations = [
>>         migrations.CreateModel(
>>             name='FooModel',
>>             fields=[
>>                 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
>>                 ('price', testing.models.PriceField(blank=True, decimal_places=3, max_digits=10, null=True, verbose_name='agrhhhhh')),
>>                 ('price_extension', models.CharField(blank=True, editable=False, max_length=3, null=True)),
>>             ],
>>         ),
>>     ]
>>
>>
>> All good, as far as I can tell. The problem comes up if I then call *./manage.py
>> migrate* which errors with the following exception:
>>
>> ./manage.py migrate testing
>> Operations to perform:
>>   Apply all migrations: testing
>> Running migrations:
>>   Applying testing.0001_initial...Traceback (most recent call last):
>>   File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/utils.py", line 63, in execute
>>     return self.cursor.execute(sql)
>>   File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
>>     return Database.Cursor.execute(self, query)
>> sqlite3.OperationalError: duplicate column name: price_extension
>>
> [...]
>> As said, I have a fresh project with no exisiting DB and tables so far. Why
>> is it complaining that there allready exista a column named
>> "price_extension"?. The migration file only contains one field called
>> "price_extension"?
>
> I'm not quite certain about this, but I believe that when the
> migration runner reconstructs a version of your FooModel, it iterates
> over the list of fields, and adds each of them to the reconstructed
> model class one by one. So what happens is that your custom PriceField
> gets added, which in turn creates its price_extension field, and then
> next thing, the migration runner adds the price_extension field one
> more time. So you end up with two instances of the price_extension
> field on the same model, and when eventually the create_model schema
> operation is executed, it adds each of them as another column to the
> table, which is obviously wrong.
>
> As for what you could do to avoid this situation – I'd suggest that
> you add an extra argument to PriceField which tells the field that it
> shouldn't create the additional extension field. Then, you implement
> the deconstruct method on the field, and include this additional flag.
> That way, auto-detected migrations will include the extra argument to
> the field instance, and there shouldn't be any duplication of the
> extension field on the resulting models reconstructed by migrations.
>
> Good luck,
>
> Michal
>
> --
> You received this message because you are subscribed to the Google Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
> To post to this group, send email to [hidden email].
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/20170913120505.GO8762%40koniiiik.org.
> For more options, visit https://groups.google.com/d/optout.



--
Melvyn Sopacua

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CA%2Bgw1GUsF%3DjNxMbKUgc3ntg3XPJ59et5cc4sZX36aMb8CP_kMw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.