[Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

Matthieu Rudelle
Hi there, 

I can't find any previous ticket proposing a solution to this problem so here are my findings: 

**Use case**:
When using continuous delivery several versions of the code can be running in parallel on se same DB. Say for instance that release 2.42 is in production, 2.43 is about to be rolled out and in this release one field (say ''MyModel.my_unused_field'') is not used anymore and was removed. Before rolling out 2.43 the DB is migrated and column ''my_unused_field'' of ''MyModel'' is removed. This makes 2.42 crash saying that one column is not found even though 2.42 does not use the field anywhere in the code.

**Temporary solution**:
Do not makemigrations until de 2.44 release, but it does not scale well with many contributors and CI tools (doing their awesome job of making sure migrations and models are in sync) will complain.

**Proposed solution**:
Have a ''disabled'' param on Field. When activated this field is not fetched from the DB but replaced by a hardcoded value. 
In our use case, ''disabled'' is added at the 2.42 release, then when 2.43 rolls out and migrates the DB no error is thrown.

**Refs**:
- django-users discussion: https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8
- the same problem discussed in this article, but with a slightly different solution: https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist (third item in the wishlist)
- Previous django-developpers discussion: https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ

What do you guys think?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/637c3542-c5bd-4493-8b12-44eb61f34a68%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

Ryan Hiebert-2
I'm not sure about the solution you mentioned, but the problem you mention is one that I definitely do deal with. At my work we have been happy with using a "safe" migrate command that only runs migrations that are marked as safe to run before the deployment happens, to address exactly this kind of scenario. You still need to make sure you don't mix stuff in the same migration that delete and add a column, for example, but it makes the process much easier, and we've been able to add the safemigrate command to our regular automated deployments. The full migrate then only runs when manually triggered, to deal with removing columns, etc.

You can check out our repository. We've been pretty happy with how it's working for us. https://github.com/aspiredu/django-safemigrate

On Mon, Jun 24, 2019 at 8:15 AM Matthieu Rudelle <[hidden email]> wrote:
Hi there, 

I can't find any previous ticket proposing a solution to this problem so here are my findings: 

**Use case**:
When using continuous delivery several versions of the code can be running in parallel on se same DB. Say for instance that release 2.42 is in production, 2.43 is about to be rolled out and in this release one field (say ''MyModel.my_unused_field'') is not used anymore and was removed. Before rolling out 2.43 the DB is migrated and column ''my_unused_field'' of ''MyModel'' is removed. This makes 2.42 crash saying that one column is not found even though 2.42 does not use the field anywhere in the code.

**Temporary solution**:
Do not makemigrations until de 2.44 release, but it does not scale well with many contributors and CI tools (doing their awesome job of making sure migrations and models are in sync) will complain.

**Proposed solution**:
Have a ''disabled'' param on Field. When activated this field is not fetched from the DB but replaced by a hardcoded value. 
In our use case, ''disabled'' is added at the 2.42 release, then when 2.43 rolls out and migrates the DB no error is thrown.

**Refs**:
- the same problem discussed in this article, but with a slightly different solution: https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist (third item in the wishlist)

What do you guys think?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/637c3542-c5bd-4493-8b12-44eb61f34a68%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CABpHFHSa4hXVrp7M2MWLcsPXSRnCnvhMQ%3D8D3q1Rg8AsAXU4Cw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

Dan Davis
I'd agree that it is a definite use case.  In the dev-ops world, it is evidently called a "blue torquoise green deployment".   It could be done as long as the code is not adding a table/field/etc.   

My discussion on django-users, https://groups.google.com/forum/#!topic/django-users/QCmy9reH8cI, also included this issue, but I was also concerned about:
* Removing a model
* Renaming a field
* etc.

The django-safemigrate repository/app seems to be easier to generalize, as such an option could be added to makemigrations.
Some questions:
  • How does the "safe" field of migrations work with other migrations related commands, such as squashmigrations?   It seems to me that only migrations that share the same value of "safe" could be squashed.
  • Can makemigrations figure out which migrations are "safe", such as:
    • It is always safe to add a column or table.
    • It is never safe to drop or rename a column or table.

On Mon, Jun 24, 2019 at 9:48 AM Ryan Hiebert <[hidden email]> wrote:
I'm not sure about the solution you mentioned, but the problem you mention is one that I definitely do deal with. At my work we have been happy with using a "safe" migrate command that only runs migrations that are marked as safe to run before the deployment happens, to address exactly this kind of scenario. You still need to make sure you don't mix stuff in the same migration that delete and add a column, for example, but it makes the process much easier, and we've been able to add the safemigrate command to our regular automated deployments. The full migrate then only runs when manually triggered, to deal with removing columns, etc.

You can check out our repository. We've been pretty happy with how it's working for us. https://github.com/aspiredu/django-safemigrate

On Mon, Jun 24, 2019 at 8:15 AM Matthieu Rudelle <[hidden email]> wrote:
Hi there, 

I can't find any previous ticket proposing a solution to this problem so here are my findings: 

**Use case**:
When using continuous delivery several versions of the code can be running in parallel on se same DB. Say for instance that release 2.42 is in production, 2.43 is about to be rolled out and in this release one field (say ''MyModel.my_unused_field'') is not used anymore and was removed. Before rolling out 2.43 the DB is migrated and column ''my_unused_field'' of ''MyModel'' is removed. This makes 2.42 crash saying that one column is not found even though 2.42 does not use the field anywhere in the code.

**Temporary solution**:
Do not makemigrations until de 2.44 release, but it does not scale well with many contributors and CI tools (doing their awesome job of making sure migrations and models are in sync) will complain.

**Proposed solution**:
Have a ''disabled'' param on Field. When activated this field is not fetched from the DB but replaced by a hardcoded value. 
In our use case, ''disabled'' is added at the 2.42 release, then when 2.43 rolls out and migrates the DB no error is thrown.

**Refs**:
- the same problem discussed in this article, but with a slightly different solution: https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist (third item in the wishlist)

What do you guys think?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/637c3542-c5bd-4493-8b12-44eb61f34a68%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CABpHFHSa4hXVrp7M2MWLcsPXSRnCnvhMQ%3D8D3q1Rg8AsAXU4Cw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFzonYbKso7RwYkMVG%2BcMrso5RZwcnH%3D8Z2bXuC-EK%2BVqS7U9Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

Ryan Hiebert-2


On Mon, Jun 24, 2019, 21:29 Dan Davis <[hidden email]> wrote:

Some questions:
  • How does the "safe" field of migrations work with other migrations related commands, such as squashmigrations?   It seems to me that only migrations that share the same value of "safe" could be squashed.
If does not affect squashing. I think this is roughly reasonable, at least for my uses, although I haven't had any real world use for squash myself. I recently had to remake the migrations in my repo for $WORK project, and it made the most sense to keep them as the default safe after deploy, to match what other apps would get. I think in most cases it won't matter when the first migration is safe for practical use.

  • Can makemigrations figure out which migrations are "safe", such as:
    • It is always safe to add a column or table.
    • It is never safe to drop or rename a column or table.
This is a feature that has sounded neat to me, approximately feasible, but hasn't yet been worth the time to implement. We've been happy adding the safety annotations manually.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CABpHFHSb2AHKb-XhZcAk6gZ3%2B%2Bt5nz7UkBUNyQ_FC%3DsYGgO_Yw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

Dan Davis
If does not affect squashing. I think this is roughly reasonable, at least for my uses, although I haven't had any real world use for squash myself. I recently had to remake the migrations in my repo for $WORK project, and it made the most sense to keep them as the default safe after deploy, to match what other apps would get. I think in most cases it won't matter when the first migration is safe for practical use.

I don't usually use squashmigrations either, being more likely to simply rewrite or make them initial.   However, if the functionality of your module were to be added to Django itself, or become more productized, then these are the issues to think about, right?

I certainly think the use case (support for something like blue/green deployments with shared database) is very important.
I'm not a super experienced Django contributor, so I'd love to hear what more experienced hands have to say about it.

On Mon, Jun 24, 2019 at 11:28 PM Ryan Hiebert <[hidden email]> wrote:


On Mon, Jun 24, 2019, 21:29 Dan Davis <[hidden email]> wrote:

Some questions:
  • How does the "safe" field of migrations work with other migrations related commands, such as squashmigrations?   It seems to me that only migrations that share the same value of "safe" could be squashed.
If does not affect squashing. I think this is roughly reasonable, at least for my uses, although I haven't had any real world use for squash myself. I recently had to remake the migrations in my repo for $WORK project, and it made the most sense to keep them as the default safe after deploy, to match what other apps would get. I think in most cases it won't matter when the first migration is safe for practical use.

  • Can makemigrations figure out which migrations are "safe", such as:
    • It is always safe to add a column or table.
    • It is never safe to drop or rename a column or table.
This is a feature that has sounded neat to me, approximately feasible, but hasn't yet been worth the time to implement. We've been happy adding the safety annotations manually.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CABpHFHSb2AHKb-XhZcAk6gZ3%2B%2Bt5nz7UkBUNyQ_FC%3DsYGgO_Yw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFzonYZpeYZGqGk3SMgYSHPk7%2BZ4vQfDNv-GaQJrjFZLivL0gQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

pavel.tyslacki
In reply to this post by Matthieu Rudelle
there are similar to `alter table drop column`  issue:  `alter table rename column`, `drop table`, `rename table`. (honestly `alter table drop column` and `drop table` a bit different wiith `alter table remane column` and `rename table`)

Look like you general flow for migration:
1. change code and create migrations
2. apply migrations
3. apply code for all your instances

the issue in this cases I can describe: we have code that we still used.

for `alter table drop column`, `drop table` next scenario works fine (as your temp solution too):

1. remove column/table usage from codebase
2. apply this code for all of your instances
3. apply migration with field/table removal

But if you mix migrations for `alter table add column` and `alter table drop column` - you cannot safely apply both migrations simultaneously, and your proposal sounds pretty reasonable there.

However if you add disabled option, then make migration, remove field, then make migration, then apply this changes to your environment - you will get same issue. So to automate deployment process and to get achievable result you want I think you need something more complex: deploying atomic changes, but for me it sounds more about deployment than django itself.




On Monday, 24 June 2019 16:15:04 UTC+3, Matthieu Rudelle wrote:
Hi there, 

I can't find any previous ticket proposing a solution to this problem so here are my findings: 

**Use case**:
When using continuous delivery several versions of the code can be running in parallel on se same DB. Say for instance that release 2.42 is in production, 2.43 is about to be rolled out and in this release one field (say ''MyModel.my_unused_field'') is not used anymore and was removed. Before rolling out 2.43 the DB is migrated and column ''my_unused_field'' of ''MyModel'' is removed. This makes 2.42 crash saying that one column is not found even though 2.42 does not use the field anywhere in the code.

**Temporary solution**:
Do not makemigrations until de 2.44 release, but it does not scale well with many contributors and CI tools (doing their awesome job of making sure migrations and models are in sync) will complain.

**Proposed solution**:
Have a ''disabled'' param on Field. When activated this field is not fetched from the DB but replaced by a hardcoded value. 
In our use case, ''disabled'' is added at the 2.42 release, then when 2.43 rolls out and migrates the DB no error is thrown.

**Refs**:
- django-users discussion: <a href="https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8&#39;;return true;">https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8
- the same problem discussed in this article, but with a slightly different solution: <a href="https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fpankrat.github.io%2F2015%2Fdjango-migrations-without-downtimes%2F%23django-wishlist\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE1j3a0VoWtbFP2hIcrdmQh39pKxg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fpankrat.github.io%2F2015%2Fdjango-migrations-without-downtimes%2F%23django-wishlist\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE1j3a0VoWtbFP2hIcrdmQh39pKxg&#39;;return true;">https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist (third item in the wishlist)
- Previous django-developpers discussion: <a href="https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ&#39;;return true;">https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ

What do you guys think?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/0422ed55-5c7f-4ed2-b096-7df02311b917%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

Matthieu Rudelle
How does the "safe" field of migrations work with other migrations related commands, such as squashmigrations? 

Squashmigrations typically targets and produces migrations that are old enough to be assumed safe.
 
You can check out our repository. We've been pretty happy with how it's working for us. https://github.com/aspiredu/django-safemigrate

safe-migrate is a neat solution. Although it lacks support for rollbacks: as soon as "migrate" is run the app is stuck in the current release

If migrations are seen as a rolling window, you want a window of X releases (2 in our case) to play well together (minus discrepancies due to updates to the application logic) so being able to postpone a column removal to a later release is important. The "disabled" feature (or anything similar to a flag written in the codebase) has this added benefit.
Squash migration would be used to squash up to but excluding the current sliding window. 

there are similar to `alter table drop column`  issue:  `alter table rename column`, `drop table`, `rename table`. (honestly `alter table drop column` and `drop table` a bit different wiith `alter table remane column` and `rename table`)
  • `alter table rename column` can be reduced to `alter table add column` and later `alter table drop column` plus some temporary update replication implemented in the app (idem for `rename table`) 
  • `drop table` follow the same logic with the SQL actions delayed to a later release
But if you mix migrations for `alter table add column` and `alter table drop column` - you cannot safely apply both migrations simultaneously, and your proposal sounds pretty reasonable there. 
 
However if you add disabled option, then make migration, remove field, then make migration, then apply this changes to your environment - you will get same issue. So to automate deployment process and to get achievable result you want I think you need something more complex: deploying atomic changes, but for me it sounds more about deployment than django itself.

As long as the columns are different these actions can happen in parallel. The flag will just delay the DB migration to a later release and the deployment needs this kind of flexibility from django to avoid direct access to the DB by the deployment process.

On Tuesday, June 25, 2019 at 7:06:38 AM UTC+2, Paveł Tyślacki wrote:
there are similar to `alter table drop column`  issue:  `alter table rename column`, `drop table`, `rename table`. (honestly `alter table drop column` and `drop table` a bit different wiith `alter table remane column` and `rename table`)

Look like you general flow for migration:
1. change code and create migrations
2. apply migrations
3. apply code for all your instances

the issue in this cases I can describe: we have code that we still used.

for `alter table drop column`, `drop table` next scenario works fine (as your temp solution too):

1. remove column/table usage from codebase
2. apply this code for all of your instances
3. apply migration with field/table removal

But if you mix migrations for `alter table add column` and `alter table drop column` - you cannot safely apply both migrations simultaneously, and your proposal sounds pretty reasonable there.

However if you add disabled option, then make migration, remove field, then make migration, then apply this changes to your environment - you will get same issue. So to automate deployment process and to get achievable result you want I think you need something more complex: deploying atomic changes, but for me it sounds more about deployment than django itself.




On Monday, 24 June 2019 16:15:04 UTC+3, Matthieu Rudelle wrote:
Hi there, 

I can't find any previous ticket proposing a solution to this problem so here are my findings: 

**Use case**:
When using continuous delivery several versions of the code can be running in parallel on se same DB. Say for instance that release 2.42 is in production, 2.43 is about to be rolled out and in this release one field (say ''MyModel.my_unused_field'') is not used anymore and was removed. Before rolling out 2.43 the DB is migrated and column ''my_unused_field'' of ''MyModel'' is removed. This makes 2.42 crash saying that one column is not found even though 2.42 does not use the field anywhere in the code.

**Temporary solution**:
Do not makemigrations until de 2.44 release, but it does not scale well with many contributors and CI tools (doing their awesome job of making sure migrations and models are in sync) will complain.

**Proposed solution**:
Have a ''disabled'' param on Field. When activated this field is not fetched from the DB but replaced by a hardcoded value. 
In our use case, ''disabled'' is added at the 2.42 release, then when 2.43 rolls out and migrates the DB no error is thrown.

**Refs**:
- django-users discussion: <a href="https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8&#39;;return true;">https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8
- the same problem discussed in this article, but with a slightly different solution: <a href="https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fpankrat.github.io%2F2015%2Fdjango-migrations-without-downtimes%2F%23django-wishlist\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE1j3a0VoWtbFP2hIcrdmQh39pKxg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fpankrat.github.io%2F2015%2Fdjango-migrations-without-downtimes%2F%23django-wishlist\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE1j3a0VoWtbFP2hIcrdmQh39pKxg&#39;;return true;">https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist (third item in the wishlist)
- Previous django-developpers discussion: <a href="https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ&#39;;return true;">https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ

What do you guys think?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/59bd29b1-5de3-4239-b534-fc5146995f72%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

Matthieu Rudelle
Does that sound like feature-request material? It seems to me django is the good place to add this flexibility as it has exclusive access to the underlying DB.

On Tue, Jun 25, 2019 at 9:08 AM Matthieu Rudelle <[hidden email]> wrote:
How does the "safe" field of migrations work with other migrations related commands, such as squashmigrations? 

Squashmigrations typically targets and produces migrations that are old enough to be assumed safe.
 
You can check out our repository. We've been pretty happy with how it's working for us. https://github.com/aspiredu/django-safemigrate

safe-migrate is a neat solution. Although it lacks support for rollbacks: as soon as "migrate" is run the app is stuck in the current release

If migrations are seen as a rolling window, you want a window of X releases (2 in our case) to play well together (minus discrepancies due to updates to the application logic) so being able to postpone a column removal to a later release is important. The "disabled" feature (or anything similar to a flag written in the codebase) has this added benefit.
Squash migration would be used to squash up to but excluding the current sliding window. 

there are similar to `alter table drop column`  issue:  `alter table rename column`, `drop table`, `rename table`. (honestly `alter table drop column` and `drop table` a bit different wiith `alter table remane column` and `rename table`)
  • `alter table rename column` can be reduced to `alter table add column` and later `alter table drop column` plus some temporary update replication implemented in the app (idem for `rename table`) 
  • `drop table` follow the same logic with the SQL actions delayed to a later release
But if you mix migrations for `alter table add column` and `alter table drop column` - you cannot safely apply both migrations simultaneously, and your proposal sounds pretty reasonable there. 
 
However if you add disabled option, then make migration, remove field, then make migration, then apply this changes to your environment - you will get same issue. So to automate deployment process and to get achievable result you want I think you need something more complex: deploying atomic changes, but for me it sounds more about deployment than django itself.

As long as the columns are different these actions can happen in parallel. The flag will just delay the DB migration to a later release and the deployment needs this kind of flexibility from django to avoid direct access to the DB by the deployment process.

On Tuesday, June 25, 2019 at 7:06:38 AM UTC+2, Paveł Tyślacki wrote:
there are similar to `alter table drop column`  issue:  `alter table rename column`, `drop table`, `rename table`. (honestly `alter table drop column` and `drop table` a bit different wiith `alter table remane column` and `rename table`)

Look like you general flow for migration:
1. change code and create migrations
2. apply migrations
3. apply code for all your instances

the issue in this cases I can describe: we have code that we still used.

for `alter table drop column`, `drop table` next scenario works fine (as your temp solution too):

1. remove column/table usage from codebase
2. apply this code for all of your instances
3. apply migration with field/table removal

But if you mix migrations for `alter table add column` and `alter table drop column` - you cannot safely apply both migrations simultaneously, and your proposal sounds pretty reasonable there.

However if you add disabled option, then make migration, remove field, then make migration, then apply this changes to your environment - you will get same issue. So to automate deployment process and to get achievable result you want I think you need something more complex: deploying atomic changes, but for me it sounds more about deployment than django itself.




On Monday, 24 June 2019 16:15:04 UTC+3, Matthieu Rudelle wrote:
Hi there, 

I can't find any previous ticket proposing a solution to this problem so here are my findings: 

**Use case**:
When using continuous delivery several versions of the code can be running in parallel on se same DB. Say for instance that release 2.42 is in production, 2.43 is about to be rolled out and in this release one field (say ''MyModel.my_unused_field'') is not used anymore and was removed. Before rolling out 2.43 the DB is migrated and column ''my_unused_field'' of ''MyModel'' is removed. This makes 2.42 crash saying that one column is not found even though 2.42 does not use the field anywhere in the code.

**Temporary solution**:
Do not makemigrations until de 2.44 release, but it does not scale well with many contributors and CI tools (doing their awesome job of making sure migrations and models are in sync) will complain.

**Proposed solution**:
Have a ''disabled'' param on Field. When activated this field is not fetched from the DB but replaced by a hardcoded value. 
In our use case, ''disabled'' is added at the 2.42 release, then when 2.43 rolls out and migrates the DB no error is thrown.

**Refs**:
- the same problem discussed in this article, but with a slightly different solution: https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist (third item in the wishlist)

What do you guys think?

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/Gr9x2OlWYN4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/59bd29b1-5de3-4239-b534-fc5146995f72%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANP1NSz45RBwO2piYRCw42ACDdsvtSo2HNF91uUWxagQZXAC0w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

Pakal
In reply to this post by Matthieu Rudelle
Just for reference, here is a package I crossed recently and which has an approach for that : https://github.com/3YOURMIND/django-deprecate-fields

On Monday, June 24, 2019 at 3:15:04 PM UTC+2, Matthieu Rudelle wrote:
Hi there, 

I can't find any previous ticket proposing a solution to this problem so here are my findings: 

**Use case**:
When using continuous delivery several versions of the code can be running in parallel on se same DB. Say for instance that release 2.42 is in production, 2.43 is about to be rolled out and in this release one field (say ''MyModel.my_unused_field'') is not used anymore and was removed. Before rolling out 2.43 the DB is migrated and column ''my_unused_field'' of ''MyModel'' is removed. This makes 2.42 crash saying that one column is not found even though 2.42 does not use the field anywhere in the code.

**Temporary solution**:
Do not makemigrations until de 2.44 release, but it does not scale well with many contributors and CI tools (doing their awesome job of making sure migrations and models are in sync) will complain.

**Proposed solution**:
Have a ''disabled'' param on Field. When activated this field is not fetched from the DB but replaced by a hardcoded value. 
In our use case, ''disabled'' is added at the 2.42 release, then when 2.43 rolls out and migrates the DB no error is thrown.

**Refs**:
- django-users discussion: <a href="https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8&#39;;return true;">https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8
- the same problem discussed in this article, but with a slightly different solution: <a href="https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fpankrat.github.io%2F2015%2Fdjango-migrations-without-downtimes%2F%23django-wishlist\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE1j3a0VoWtbFP2hIcrdmQh39pKxg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fpankrat.github.io%2F2015%2Fdjango-migrations-without-downtimes%2F%23django-wishlist\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE1j3a0VoWtbFP2hIcrdmQh39pKxg&#39;;return true;">https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist (third item in the wishlist)
- Previous django-developpers discussion: <a href="https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ&#39;;return true;">https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ

What do you guys think?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/d05e7d82-b7cd-4ced-a23c-d3c9dfa3b826%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

Matthieu Rudelle
Nice find, It's an interesting way to solve it. We might give it a try.

On Thursday, June 27, 2019 at 11:12:16 PM UTC+2, Pkl wrote:
Just for reference, here is a package I crossed recently and which has an approach for that : <a href="https://github.com/3YOURMIND/django-deprecate-fields" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2F3YOURMIND%2Fdjango-deprecate-fields\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHmJzIy9HSjnCHLf6_9Zt5WoKhOew&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2F3YOURMIND%2Fdjango-deprecate-fields\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHmJzIy9HSjnCHLf6_9Zt5WoKhOew&#39;;return true;">https://github.com/3YOURMIND/django-deprecate-fields

On Monday, June 24, 2019 at 3:15:04 PM UTC+2, Matthieu Rudelle wrote:
Hi there, 

I can't find any previous ticket proposing a solution to this problem so here are my findings: 

**Use case**:
When using continuous delivery several versions of the code can be running in parallel on se same DB. Say for instance that release 2.42 is in production, 2.43 is about to be rolled out and in this release one field (say ''MyModel.my_unused_field'') is not used anymore and was removed. Before rolling out 2.43 the DB is migrated and column ''my_unused_field'' of ''MyModel'' is removed. This makes 2.42 crash saying that one column is not found even though 2.42 does not use the field anywhere in the code.

**Temporary solution**:
Do not makemigrations until de 2.44 release, but it does not scale well with many contributors and CI tools (doing their awesome job of making sure migrations and models are in sync) will complain.

**Proposed solution**:
Have a ''disabled'' param on Field. When activated this field is not fetched from the DB but replaced by a hardcoded value. 
In our use case, ''disabled'' is added at the 2.42 release, then when 2.43 rolls out and migrates the DB no error is thrown.

**Refs**:
- django-users discussion: <a href="https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8&#39;;return true;">https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8
- the same problem discussed in this article, but with a slightly different solution: <a href="https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fpankrat.github.io%2F2015%2Fdjango-migrations-without-downtimes%2F%23django-wishlist\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE1j3a0VoWtbFP2hIcrdmQh39pKxg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fpankrat.github.io%2F2015%2Fdjango-migrations-without-downtimes%2F%23django-wishlist\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE1j3a0VoWtbFP2hIcrdmQh39pKxg&#39;;return true;">https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist (third item in the wishlist)
- Previous django-developpers discussion: <a href="https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ&#39;;return true;">https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ

What do you guys think?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/4ee30e7d-f9e5-48ae-8df7-bf415fd34c71%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

Pakal
This society also has linters and such, so that DB migrations are safer regarding deployment rollouts - https://medium.com/3yourmind/keeping-django-database-migrations-backward-compatible-727820260dbb


On Saturday, June 29, 2019 at 10:36:11 AM UTC+2, Matthieu Rudelle wrote:
Nice find, It's an interesting way to solve it. We might give it a try.

On Thursday, June 27, 2019 at 11:12:16 PM UTC+2, Pkl wrote:
Just for reference, here is a package I crossed recently and which has an approach for that : <a href="https://github.com/3YOURMIND/django-deprecate-fields" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2F3YOURMIND%2Fdjango-deprecate-fields\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHmJzIy9HSjnCHLf6_9Zt5WoKhOew&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2F3YOURMIND%2Fdjango-deprecate-fields\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHmJzIy9HSjnCHLf6_9Zt5WoKhOew&#39;;return true;">https://github.com/3YOURMIND/django-deprecate-fields

On Monday, June 24, 2019 at 3:15:04 PM UTC+2, Matthieu Rudelle wrote:
Hi there, 

I can't find any previous ticket proposing a solution to this problem so here are my findings: 

**Use case**:
When using continuous delivery several versions of the code can be running in parallel on se same DB. Say for instance that release 2.42 is in production, 2.43 is about to be rolled out and in this release one field (say ''MyModel.my_unused_field'') is not used anymore and was removed. Before rolling out 2.43 the DB is migrated and column ''my_unused_field'' of ''MyModel'' is removed. This makes 2.42 crash saying that one column is not found even though 2.42 does not use the field anywhere in the code.

**Temporary solution**:
Do not makemigrations until de 2.44 release, but it does not scale well with many contributors and CI tools (doing their awesome job of making sure migrations and models are in sync) will complain.

**Proposed solution**:
Have a ''disabled'' param on Field. When activated this field is not fetched from the DB but replaced by a hardcoded value. 
In our use case, ''disabled'' is added at the 2.42 release, then when 2.43 rolls out and migrates the DB no error is thrown.

**Refs**:
- django-users discussion: <a href="https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8&#39;;return true;">https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8
- the same problem discussed in this article, but with a slightly different solution: <a href="https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fpankrat.github.io%2F2015%2Fdjango-migrations-without-downtimes%2F%23django-wishlist\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE1j3a0VoWtbFP2hIcrdmQh39pKxg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fpankrat.github.io%2F2015%2Fdjango-migrations-without-downtimes%2F%23django-wishlist\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE1j3a0VoWtbFP2hIcrdmQh39pKxg&#39;;return true;">https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist (third item in the wishlist)
- Previous django-developpers discussion: <a href="https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ&#39;;return true;">https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ

What do you guys think?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/012ce55e-a01d-49dc-bb95-993987725a5d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.