Quantcast

Automatic reverse migrations

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Automatic reverse migrations

paul-6
In a modern Continuous Delivery environment, it's expected that there is an automated process for deploying code, and therefore performing database migrations. This is all straightforward.

I haven't been able to find any good solutions for automatic rollback though. The main problem that I see is that there is (AFAIK) no easy way to definitively know which migrations to unapply to roll back to the previous verison. If you try to rollback from the new N+1 version, you have the migrations, but no recording of the previous version's state. If you rollback from the previous N version, you don't have the new migration files to do the DB rollback.

What I'd really like is a way of recording a 'db migration checkpoint' which could be generated per-release (or whenever else you care to checkpoint your migration state), and would say something like `v1: {app1:0002,0003, app2: 0004}, v2: {app1:0004, app2: 0005, 0006}`, thus letting me roll back all of the migrations in the v2 deploy with a single command.

Does anyone have suggestions or references here? I may try rolling the above solution if there is no prior art, but I want to avoid reinventing the wheel here, as it seems that this issue must have been hit by many other users before me.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/6cafecca-2c71-4a6f-b93b-fe3e64847bc2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Automatic reverse migrations

Andrew Godwin-3
Hi Paul,

I have tried this in the past, but it's basically an unsolvable problem in the generic case. Databases do not take well to snapshotting or changing schema, and some operations are naturally irreversible. If you find a way that works well, I suggest you write it up so others can learn from it!

Andrew

On Fri, Apr 14, 2017 at 10:33 AM, <[hidden email]> wrote:
In a modern Continuous Delivery environment, it's expected that there is an automated process for deploying code, and therefore performing database migrations. This is all straightforward.

I haven't been able to find any good solutions for automatic rollback though. The main problem that I see is that there is (AFAIK) no easy way to definitively know which migrations to unapply to roll back to the previous verison. If you try to rollback from the new N+1 version, you have the migrations, but no recording of the previous version's state. If you rollback from the previous N version, you don't have the new migration files to do the DB rollback.

What I'd really like is a way of recording a 'db migration checkpoint' which could be generated per-release (or whenever else you care to checkpoint your migration state), and would say something like `v1: {app1:0002,0003, app2: 0004}, v2: {app1:0004, app2: 0005, 0006}`, thus letting me roll back all of the migrations in the v2 deploy with a single command.

Does anyone have suggestions or references here? I may try rolling the above solution if there is no prior art, but I want to avoid reinventing the wheel here, as it seems that this issue must have been hit by many other users before me.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/6cafecca-2c71-4a6f-b93b-fe3e64847bc2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAFwN1uoaEQ_8nYt6YH5pqY%3Dcz4F0zt3NPVoHe0oGmu0o0x9L3w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Automatic reverse migrations

Camilo Torres
Hi,
May both of you try to implement "blue-green deployments", and switch only if all tests are OK.
You may also be interested in running a test deployment in a testing environment prior to production.
Don't know of a solution integrated into django.

On Friday, April 14, 2017 at 2:26:51 PM UTC-4, Andrew Godwin wrote:
Hi Paul,

I have tried this in the past, but it's basically an unsolvable problem in the generic case. Databases do not take well to snapshotting or changing schema, and some operations are naturally irreversible. If you find a way that works well, I suggest you write it up so others can learn from it!

Andrew

On Fri, Apr 14, 2017 at 10:33 AM, <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="MLRuuDTAAQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">pa...@...> wrote:
In a modern Continuous Delivery environment, it's expected that there is an automated process for deploying code, and therefore performing database migrations. This is all straightforward.

I haven't been able to find any good solutions for automatic rollback though. The main problem that I see is that there is (AFAIK) no easy way to definitively know which migrations to unapply to roll back to the previous verison. If you try to rollback from the new N+1 version, you have the migrations, but no recording of the previous version's state. If you rollback from the previous N version, you don't have the new migration files to do the DB rollback.

What I'd really like is a way of recording a 'db migration checkpoint' which could be generated per-release (or whenever else you care to checkpoint your migration state), and would say something like `v1: {app1:0002,0003, app2: 0004}, v2: {app1:0004, app2: 0005, 0006}`, thus letting me roll back all of the migrations in the v2 deploy with a single command.

Does anyone have suggestions or references here? I may try rolling the above solution if there is no prior art, but I want to avoid reinventing the wheel here, as it seems that this issue must have been hit by many other users before me.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="MLRuuDTAAQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-users...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="MLRuuDTAAQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django...@....
Visit this group at <a href="https://groups.google.com/group/django-users" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/django-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-users&#39;;return true;">https://groups.google.com/group/django-users.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-users/6cafecca-2c71-4a6f-b93b-fe3e64847bc2%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-users/6cafecca-2c71-4a6f-b93b-fe3e64847bc2%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-users/6cafecca-2c71-4a6f-b93b-fe3e64847bc2%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/django-users/6cafecca-2c71-4a6f-b93b-fe3e64847bc2%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/362cbdc0-9dee-4c1b-a68d-b671eed49336%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Automatic reverse migrations

paul-6
Thanks Andrew, will see what I can rustle up.

Camilo -- blue/green deploys don't really help with this problem. As each application deployment still talks to the same database, once you have run the migration you can't just abandon your deployment, you need to unwind the db migration first.

On Saturday, April 15, 2017 at 6:01:03 PM UTC-7, Camilo Torres wrote:
Hi,
May both of you try to implement "blue-green deployments", and switch only if all tests are OK.
You may also be interested in running a test deployment in a testing environment prior to production.
Don't know of a solution integrated into django.

On Friday, April 14, 2017 at 2:26:51 PM UTC-4, Andrew Godwin wrote:
Hi Paul,

I have tried this in the past, but it's basically an unsolvable problem in the generic case. Databases do not take well to snapshotting or changing schema, and some operations are naturally irreversible. If you find a way that works well, I suggest you write it up so others can learn from it!

Andrew

On Fri, Apr 14, 2017 at 10:33 AM, <[hidden email]> wrote:
In a modern Continuous Delivery environment, it's expected that there is an automated process for deploying code, and therefore performing database migrations. This is all straightforward.

I haven't been able to find any good solutions for automatic rollback though. The main problem that I see is that there is (AFAIK) no easy way to definitively know which migrations to unapply to roll back to the previous verison. If you try to rollback from the new N+1 version, you have the migrations, but no recording of the previous version's state. If you rollback from the previous N version, you don't have the new migration files to do the DB rollback.

What I'd really like is a way of recording a 'db migration checkpoint' which could be generated per-release (or whenever else you care to checkpoint your migration state), and would say something like `v1: {app1:0002,0003, app2: 0004}, v2: {app1:0004, app2: 0005, 0006}`, thus letting me roll back all of the migrations in the v2 deploy with a single command.

Does anyone have suggestions or references here? I may try rolling the above solution if there is no prior art, but I want to avoid reinventing the wheel here, as it seems that this issue must have been hit by many other users before me.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/django-users" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/django-users&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-users&#39;;return true;">https://groups.google.com/group/django-users.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-users/6cafecca-2c71-4a6f-b93b-fe3e64847bc2%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-users/6cafecca-2c71-4a6f-b93b-fe3e64847bc2%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-users/6cafecca-2c71-4a6f-b93b-fe3e64847bc2%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/django-users/6cafecca-2c71-4a6f-b93b-fe3e64847bc2%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/8d8f46a7-8819-4d18-908d-9681d8b6a39e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Automatic reverse migrations

Mike Dewhirst-3
On 18/04/2017 9:58 AM, [hidden email] wrote:
> Thanks Andrew, will see what I can rustle up.
>
> Camilo -- blue/green deploys don't really help with this problem. As
> each application deployment still talks to the same database, once you
> have run the migration you can't just abandon your deployment, you
> need to unwind the db migration first.

I agree it is an unsolvable problem to automate it. Or rather, not worth
the development cost of making an automatic "unwinder".

For the number of times it is likely to be needed in general it would be
less costly to do another new migration towards the previous state.

>
> On Saturday, April 15, 2017 at 6:01:03 PM UTC-7, Camilo Torres wrote:
>
>     Hi,
>     May both of you try to implement "blue-green deployments", and
>     switch only if all tests are OK.
>     You may also be interested in running a test deployment in a
>     testing environment prior to production.
>     Don't know of a solution integrated into django.
>
>     On Friday, April 14, 2017 at 2:26:51 PM UTC-4, Andrew Godwin wrote:
>
>         Hi Paul,
>
>         I have tried this in the past, but it's basically an
>         unsolvable problem in the generic case. Databases do not take
>         well to snapshotting or changing schema, and some operations
>         are naturally irreversible. If you find a way that works well,
>         I suggest you write it up so others can learn from it!
>
>         Andrew
>
>         On Fri, Apr 14, 2017 at 10:33 AM, <[hidden email]> wrote:
>
>             In a modern Continuous Delivery environment, it's expected
>             that there is an automated process for deploying code, and
>             therefore performing database migrations. This is all
>             straightforward.
>
>             I haven't been able to find any good solutions for
>             automatic rollback though. The main problem that I see is
>             that there is (AFAIK) no easy way to definitively know
>             which migrations to unapply to roll back to the previous
>             verison. If you try to rollback from the new N+1 version,
>             you have the migrations, but no recording of the previous
>             version's state. If you rollback from the previous N
>             version, you don't have the new migration files to do the
>             DB rollback.
>
>             What I'd really like is a way of recording a 'db migration
>             checkpoint' which could be generated per-release (or
>             whenever else you care to checkpoint your migration
>             state), and would say something like `v1: {app1:0002,0003,
>             app2: 0004}, v2: {app1:0004, app2: 0005, 0006}`, thus
>             letting me roll back all of the migrations in the v2
>             deploy with a single command.
>
>             Does anyone have suggestions or references here? I may try
>             rolling the above solution if there is no prior art, but I
>             want to avoid reinventing the wheel here, as it seems that
>             this issue must have been hit by many other users before me.
>             --
>             You received this message because you are subscribed to
>             the Google Groups "Django users" group.
>             To unsubscribe from this group and stop receiving emails
>             from it, send an email to [hidden email].
>             To post to this group, send email to
>             [hidden email].
>             Visit this group at
>             https://groups.google.com/group/django-users
>             <https://groups.google.com/group/django-users>.
>             To view this discussion on the web visit
>             https://groups.google.com/d/msgid/django-users/6cafecca-2c71-4a6f-b93b-fe3e64847bc2%40googlegroups.com
>             <https://groups.google.com/d/msgid/django-users/6cafecca-2c71-4a6f-b93b-fe3e64847bc2%40googlegroups.com?utm_medium=email&utm_source=footer>.
>             For more options, visit https://groups.google.com/d/optout
>             <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [hidden email]
> <mailto:[hidden email]>.
> To post to this group, send email to [hidden email]
> <mailto:[hidden email]>.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/8d8f46a7-8819-4d18-908d-9681d8b6a39e%40googlegroups.com 
> <https://groups.google.com/d/msgid/django-users/8d8f46a7-8819-4d18-908d-9681d8b6a39e%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/21a5c50b-2ef2-e729-64a4-71f96b8f213b%40dewhirst.com.au.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Automatic reverse migrations

paul-6
I think it's possible to do better than the current status quo with a bit more metadata, and without changing the current flow for users that don't want/need more -- essentially all I'm looking for is a way to record which migrations were applied in a given run of `./manage.py migrate`, so that I can revert the same set in a downgrade. This would be equally useful for reversing manually applied migrations as for doing an automated rollback.

The API I'm thinking of is something like this (zero thought invested in naming):

```
# ./manage.py migratewithtag 1.2.3
> running migration app1.0002
> running migration app2.0004
> writing tag before and after state to django_migrationtags table {app1:{before: 0001, after:0002},...}
# ./manage.py rollbacktag 1.2.3
> running migration app1.0001
> running migration app2.0003
```

This is essentially just a layer of metadata above the current migrations table.

I've yet to start playing with an implementation, but of course suggestions welcome for potential problems with this approach.

On Mon, Apr 17, 2017 at 5:21 PM, Mike Dewhirst <[hidden email]> wrote:
On 18/04/2017 9:58 AM, [hidden email] wrote:
Thanks Andrew, will see what I can rustle up.

Camilo -- blue/green deploys don't really help with this problem. As each application deployment still talks to the same database, once you have run the migration you can't just abandon your deployment, you need to unwind the db migration first.

I agree it is an unsolvable problem to automate it. Or rather, not worth the development cost of making an automatic "unwinder".

For the number of times it is likely to be needed in general it would be less costly to do another new migration towards the previous state.


On Saturday, April 15, 2017 at 6:01:03 PM UTC-7, Camilo Torres wrote:

    Hi,
    May both of you try to implement "blue-green deployments", and
    switch only if all tests are OK.
    You may also be interested in running a test deployment in a
    testing environment prior to production.
    Don't know of a solution integrated into django.

    On Friday, April 14, 2017 at 2:26:51 PM UTC-4, Andrew Godwin wrote:

        Hi Paul,

        I have tried this in the past, but it's basically an
        unsolvable problem in the generic case. Databases do not take
        well to snapshotting or changing schema, and some operations
        are naturally irreversible. If you find a way that works well,
        I suggest you write it up so others can learn from it!

        Andrew

        On Fri, Apr 14, 2017 at 10:33 AM, <[hidden email]> wrote:

            In a modern Continuous Delivery environment, it's expected
            that there is an automated process for deploying code, and
            therefore performing database migrations. This is all
            straightforward.

            I haven't been able to find any good solutions for
            automatic rollback though. The main problem that I see is
            that there is (AFAIK) no easy way to definitively know
            which migrations to unapply to roll back to the previous
            verison. If you try to rollback from the new N+1 version,
            you have the migrations, but no recording of the previous
            version's state. If you rollback from the previous N
            version, you don't have the new migration files to do the
            DB rollback.

            What I'd really like is a way of recording a 'db migration
            checkpoint' which could be generated per-release (or
            whenever else you care to checkpoint your migration
            state), and would say something like `v1: {app1:0002,0003,
            app2: 0004}, v2: {app1:0004, app2: 0005, 0006}`, thus
            letting me roll back all of the migrations in the v2
            deploy with a single command.

            Does anyone have suggestions or references here? I may try
            rolling the above solution if there is no prior art, but I
            want to avoid reinventing the wheel here, as it seems that
            this issue must have been hit by many other users before me.
            --             You received this message because you are subscribed to
            the Google Groups "Django users" group.
            To unsubscribe from this group and stop receiving emails
            from it, send an email to [hidden email].
            To post to this group, send email to
            [hidden email].
            Visit this group at
            https://groups.google.com/group/django-users
            <https://groups.google.com/group/django-users>.
            To view this discussion on the web visit
            https://groups.google.com/d/msgid/django-users/6cafecca-2c71-4a6f-b93b-fe3e64847bc2%40googlegroups.com
            <https://groups.google.com/d/msgid/django-users/6cafecca-2c71-4a6f-b93b-fe3e64847bc2%40googlegroups.com?utm_medium=email&utm_source=footer>.
            For more options, visit https://groups.google.com/d/optout
            <https://groups.google.com/d/optout>.


--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email] <mailto:[hidden email]>.
To post to this group, send email to [hidden email] <mailto:[hidden email]>.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/8d8f46a7-8819-4d18-908d-9681d8b6a39e%40googlegroups.com <https://groups.google.com/d/msgid/django-users/8d8f46a7-8819-4d18-908d-9681d8b6a39e%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/39Un2Jm--aU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/21a5c50b-2ef2-e729-64a4-71f96b8f213b%40dewhirst.com.au.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAA%2BchP-xgskxxgEK1wj_C1xr%2BQmmh7zamKXSvV0WDp_pksWfCQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Loading...