Thoughts on diff-based migrations with django?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Thoughts on diff-based migrations with django?

djrobstep
Hi all,

Some thoughts on schema migrations that may interest you. By way of background, I'm the author of migra (https://github.com/djrobstep/migra), a schema comparison/diff tool for Postgres.

I initially wrote this tool because I wanted to be able to generate migration scripts automatically, without needing historical migration files.

It means that you can do things like sync your database to your models mostly automatically, and also explicitly test for a matching schema. Instead of a chain of migration files you only ever need to keep track of one, containing any pending changes).

I've used this approach with success on a number of projects now, and it seems to work pretty well. I also talked about this approach to migrations at the most recent PostgresOpen (https://www.youtube.com/watch?v=xr498W8oMRo).

The downside of this approach is that it's slightly more operationally complex (you need have a temporary copy of your production schema available for comparison purposes, typically a --schema-only dump file) and also that it's postgres-only at this point. It's also conceptually quite different to the typical Django/Rails style of migrations which represents a barrier to entry.

In spite of this, I think doing migrations this way has some compelling advantages, that are relevant to users of Django.

I'm interested in your opinions on this topic as Django developers. What do you think? Is it something worth offering as an option for managing Django migrations? If your feelings are positive, I'm happy to take a look at extending the code accordingly.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on diff-based migrations with django?

Curtis Maloney-2
Hi,

I must say your idea intrigues me... and I'd certainly like to discuss
it further.

The first thought that comes to mind is how would such a system handle
data migrations?

Being able to leap from one schema state to another is great, buy
typically in the life-cycle of a reusable app it's required to also
migrate old data into new schema.

This also happens within project-specific apps.

--
Curtis


On 02/20/2018 05:05 PM, djrobstep wrote:

> Hi all,
>
> Some thoughts on schema migrations that may interest you. By way of
> background, I'm the author of migra
> (https://github.com/djrobstep/migra), a schema comparison/diff tool for
> Postgres.
>
> I initially wrote this tool because I wanted to be able to generate
> migration scripts automatically, without needing historical migration files.
>
> It means that you can do things like sync your database to your models
> mostly automatically, and also explicitly test for a matching schema.
> Instead of a chain of migration files you only ever need to keep track
> of one, containing any pending changes).
>
> I've used this approach with success on a number of projects now, and it
> seems to work pretty well. I also talked about this approach to
> migrations at the most recent PostgresOpen
> (https://www.youtube.com/watch?v=xr498W8oMRo).
>
> The downside of this approach is that it's slightly more operationally
> complex (you need have a temporary copy of your production schema
> available for comparison purposes, typically a --schema-only dump file)
> and also that it's postgres-only at this point. It's also conceptually
> quite different to the typical Django/Rails style of migrations which
> represents a barrier to entry.
>
> In spite of this, I think doing migrations this way has some compelling
> advantages, that are relevant to users of Django.
>
> I'm interested in your opinions on this topic as Django developers. What
> do you think? Is it something worth offering as an option for managing
> Django migrations? If your feelings are positive, I'm happy to take a
> look at extending the code accordingly.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [hidden email]
> <mailto:[hidden email]>.
> To post to this group, send email to [hidden email]
> <mailto:[hidden email]>.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com 
> <https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/4ef38de9-780c-e2f9-9ce2-5d9621d6bf64%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on diff-based migrations with django?

djrobstep
Yes, this approach certainly doesn't magically solve data migration issues, however the data migrations question has been a common one.

The nature of data migrations is that they are very one-off and context dependent and so still something you'd need to script up manually. You might use the diff tool to generate the migration script required but then edit it to add any queries needed for carrying out the data migration.

And doing things this way, there's nothing stopping you continuing to chain multiple migration scripts together to support multiple versions in the context of a reusable app.

On Tuesday, 20 February 2018 23:27:31 UTC+11, Curtis Maloney wrote:
Hi,

I must say your idea intrigues me... and I'd certainly like to discuss
it further.

The first thought that comes to mind is how would such a system handle
data migrations?

Being able to leap from one schema state to another is great, buy
typically in the life-cycle of a reusable app it's required to also
migrate old data into new schema.

This also happens within project-specific apps.

--
Curtis


On 02/20/2018 05:05 PM, djrobstep wrote:

> Hi all,
>
> Some thoughts on schema migrations that may interest you. By way of
> background, I'm the author of migra
> (<a href="https://github.com/djrobstep/migra" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjrobstep%2Fmigra\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF7uJiFZ_Yq8IJQceciZW69Xq-kIg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjrobstep%2Fmigra\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF7uJiFZ_Yq8IJQceciZW69Xq-kIg&#39;;return true;">https://github.com/djrobstep/migra), a schema comparison/diff tool for
> Postgres.
>
> I initially wrote this tool because I wanted to be able to generate
> migration scripts automatically, without needing historical migration files.
>
> It means that you can do things like sync your database to your models
> mostly automatically, and also explicitly test for a matching schema.
> Instead of a chain of migration files you only ever need to keep track
> of one, containing any pending changes).
>
> I've used this approach with success on a number of projects now, and it
> seems to work pretty well. I also talked about this approach to
> migrations at the most recent PostgresOpen
> (<a href="https://www.youtube.com/watch?v=xr498W8oMRo" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.youtube.com/watch?v\x3dxr498W8oMRo&#39;;return true;" onclick="this.href=&#39;https://www.youtube.com/watch?v\x3dxr498W8oMRo&#39;;return true;">https://www.youtube.com/watch?v=xr498W8oMRo).
>
> The downside of this approach is that it's slightly more operationally
> complex (you need have a temporary copy of your production schema
> available for comparison purposes, typically a --schema-only dump file)
> and also that it's postgres-only at this point. It's also conceptually
> quite different to the typical Django/Rails style of migrations which
> represents a barrier to entry.
>
> In spite of this, I think doing migrations this way has some compelling
> advantages, that are relevant to users of Django.
>
> I'm interested in your opinions on this topic as Django developers. What
> do you think? Is it something worth offering as an option for managing
> Django migrations? If your feelings are positive, I'm happy to take a
> look at extending the code accordingly.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="oGt7HnnZBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-develop...@googlegroups.com
> <mailto:<a href="javascript:" target="_blank" gdf-obfuscated-mailto="oGt7HnnZBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-developers+unsubscribe@...>.
> To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="oGt7HnnZBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-d...@googlegroups.com
> <mailto:<a href="javascript:" target="_blank" gdf-obfuscated-mailto="oGt7HnnZBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-d...@googlegroups.com>.
> Visit this group at <a href="https://groups.google.com/group/django-developers" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> <a href="https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com&#39;;return true;">https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com
> <<a href="https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/62c00b16-a854-4c49-a5c9-7e83dc0c9a2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on diff-based migrations with django?

Andrew Godwin-3
In reply to this post by djrobstep
Hi,

This was an approach we deliberately avoided during development of the current migration system - it works poorly with diverging feature branches and requires that your developers have access to production schema at all times (and additionally that you don't have divergent production/staging/QA deploys, like many environments do).

I think it would be a fine addition as an optional third-party tool, but it's not something that really makes sense to include in Django at this point I think.

Andrew

On Mon, Feb 19, 2018 at 10:05 PM, djrobstep <[hidden email]> wrote:
Hi all,

Some thoughts on schema migrations that may interest you. By way of background, I'm the author of migra (https://github.com/djrobstep/migra), a schema comparison/diff tool for Postgres.

I initially wrote this tool because I wanted to be able to generate migration scripts automatically, without needing historical migration files.

It means that you can do things like sync your database to your models mostly automatically, and also explicitly test for a matching schema. Instead of a chain of migration files you only ever need to keep track of one, containing any pending changes).

I've used this approach with success on a number of projects now, and it seems to work pretty well. I also talked about this approach to migrations at the most recent PostgresOpen (https://www.youtube.com/watch?v=xr498W8oMRo).

The downside of this approach is that it's slightly more operationally complex (you need have a temporary copy of your production schema available for comparison purposes, typically a --schema-only dump file) and also that it's postgres-only at this point. It's also conceptually quite different to the typical Django/Rails style of migrations which represents a barrier to entry.

In spite of this, I think doing migrations this way has some compelling advantages, that are relevant to users of Django.

I'm interested in your opinions on this topic as Django developers. What do you think? Is it something worth offering as an option for managing Django migrations? If your feelings are positive, I'm happy to take a look at extending the code accordingly.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1urzK%3DwWT0SfS32sg_YhcdJnG_2h9Cbp8u2Uouzs6LaWFg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on diff-based migrations with django?

djrobstep
Yes, I don't think it would make sense to include in django directly. Because the diffing approach works at the database directly, below the layer of the models, the two parts don't really need to be integrated. All that is really required for django to support this as an option is to provide a documented way to emit the from-scratch creation SQL for the currently defined models.

As far as I can tell there used to be an admin command available for this in version 1.x but it was removed? Is there already another supported way to do this?

Incidentally, I've used this approach on some larger projects with diverging branches and it worked pretty well. You do have to set some things up differently tho - keep meaning to write a blog post or something about this. The need to have access to a production schema is an additional hurdle, but well worth it for the additional testability and robustness, in my experience.

On Wednesday, 21 February 2018 18:44:30 UTC+11, Andrew Godwin wrote:
Hi,

This was an approach we deliberately avoided during development of the current migration system - it works poorly with diverging feature branches and requires that your developers have access to production schema at all times (and additionally that you don't have divergent production/staging/QA deploys, like many environments do).

I think it would be a fine addition as an optional third-party tool, but it's not something that really makes sense to include in Django at this point I think.

Andrew

On Mon, Feb 19, 2018 at 10:05 PM, djrobstep <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="2Y9a332iAgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">robert...@...> wrote:
Hi all,

Some thoughts on schema migrations that may interest you. By way of background, I'm the author of migra (<a href="https://github.com/djrobstep/migra" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjrobstep%2Fmigra\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF7uJiFZ_Yq8IJQceciZW69Xq-kIg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjrobstep%2Fmigra\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF7uJiFZ_Yq8IJQceciZW69Xq-kIg&#39;;return true;">https://github.com/djrobstep/migra), a schema comparison/diff tool for Postgres.

I initially wrote this tool because I wanted to be able to generate migration scripts automatically, without needing historical migration files.

It means that you can do things like sync your database to your models mostly automatically, and also explicitly test for a matching schema. Instead of a chain of migration files you only ever need to keep track of one, containing any pending changes).

I've used this approach with success on a number of projects now, and it seems to work pretty well. I also talked about this approach to migrations at the most recent PostgresOpen (<a href="https://www.youtube.com/watch?v=xr498W8oMRo" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.youtube.com/watch?v\x3dxr498W8oMRo&#39;;return true;" onclick="this.href=&#39;https://www.youtube.com/watch?v\x3dxr498W8oMRo&#39;;return true;">https://www.youtube.com/watch?v=xr498W8oMRo).

The downside of this approach is that it's slightly more operationally complex (you need have a temporary copy of your production schema available for comparison purposes, typically a --schema-only dump file) and also that it's postgres-only at this point. It's also conceptually quite different to the typical Django/Rails style of migrations which represents a barrier to entry.

In spite of this, I think doing migrations this way has some compelling advantages, that are relevant to users of Django.

I'm interested in your opinions on this topic as Django developers. What do you think? Is it something worth offering as an option for managing Django migrations? If your feelings are positive, I'm happy to take a look at extending the code accordingly.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="2Y9a332iAgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-develop...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="2Y9a332iAgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-d...@googlegroups.com.
Visit this group at <a href="https://groups.google.com/group/django-developers" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/f17d9d96-db18-4501-ad10-f174bb1dc468%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on diff-based migrations with django?

djrobstep
Bump! Is there a documented way to generate the from-scratch creation SQL for the currently defined models? If not, is it possible to add something that spits this out? Happy to take a look at implementing this.


On Sunday, 25 February 2018 13:48:27 UTC+11, djrobstep wrote:
Yes, I don't think it would make sense to include in django directly. Because the diffing approach works at the database directly, below the layer of the models, the two parts don't really need to be integrated. All that is really required for django to support this as an option is to provide a documented way to emit the from-scratch creation SQL for the currently defined models.

As far as I can tell there used to be an admin command available for this in version 1.x but it was removed? Is there already another supported way to do this?

Incidentally, I've used this approach on some larger projects with diverging branches and it worked pretty well. You do have to set some things up differently tho - keep meaning to write a blog post or something about this. The need to have access to a production schema is an additional hurdle, but well worth it for the additional testability and robustness, in my experience.

On Wednesday, 21 February 2018 18:44:30 UTC+11, Andrew Godwin wrote:
Hi,

This was an approach we deliberately avoided during development of the current migration system - it works poorly with diverging feature branches and requires that your developers have access to production schema at all times (and additionally that you don't have divergent production/staging/QA deploys, like many environments do).

I think it would be a fine addition as an optional third-party tool, but it's not something that really makes sense to include in Django at this point I think.

Andrew

On Mon, Feb 19, 2018 at 10:05 PM, djrobstep <[hidden email]> wrote:
Hi all,

Some thoughts on schema migrations that may interest you. By way of background, I'm the author of migra (<a href="https://github.com/djrobstep/migra" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjrobstep%2Fmigra\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF7uJiFZ_Yq8IJQceciZW69Xq-kIg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjrobstep%2Fmigra\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF7uJiFZ_Yq8IJQceciZW69Xq-kIg&#39;;return true;">https://github.com/djrobstep/migra), a schema comparison/diff tool for Postgres.

I initially wrote this tool because I wanted to be able to generate migration scripts automatically, without needing historical migration files.

It means that you can do things like sync your database to your models mostly automatically, and also explicitly test for a matching schema. Instead of a chain of migration files you only ever need to keep track of one, containing any pending changes).

I've used this approach with success on a number of projects now, and it seems to work pretty well. I also talked about this approach to migrations at the most recent PostgresOpen (<a href="https://www.youtube.com/watch?v=xr498W8oMRo" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.youtube.com/watch?v\x3dxr498W8oMRo&#39;;return true;" onclick="this.href=&#39;https://www.youtube.com/watch?v\x3dxr498W8oMRo&#39;;return true;">https://www.youtube.com/watch?v=xr498W8oMRo).

The downside of this approach is that it's slightly more operationally complex (you need have a temporary copy of your production schema available for comparison purposes, typically a --schema-only dump file) and also that it's postgres-only at this point. It's also conceptually quite different to the typical Django/Rails style of migrations which represents a barrier to entry.

In spite of this, I think doing migrations this way has some compelling advantages, that are relevant to users of Django.

I'm interested in your opinions on this topic as Django developers. What do you think? Is it something worth offering as an option for managing Django migrations? If your feelings are positive, I'm happy to take a look at extending the code accordingly.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/django-developers" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/896fc851-06b5-4db0-bb70-6d13b36af8da%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on diff-based migrations with django?

Florian Apolloner
Just for one migration; but there is the sqlmigrate management command which you can use as building base.

Cheers,
Florian

On Wednesday, May 2, 2018 at 7:07:27 AM UTC+2, djrobstep wrote:
Bump! Is there a documented way to generate the from-scratch creation SQL for the currently defined models? If not, is it possible to add something that spits this out? Happy to take a look at implementing this.


On Sunday, 25 February 2018 13:48:27 UTC+11, djrobstep wrote:
Yes, I don't think it would make sense to include in django directly. Because the diffing approach works at the database directly, below the layer of the models, the two parts don't really need to be integrated. All that is really required for django to support this as an option is to provide a documented way to emit the from-scratch creation SQL for the currently defined models.

As far as I can tell there used to be an admin command available for this in version 1.x but it was removed? Is there already another supported way to do this?

Incidentally, I've used this approach on some larger projects with diverging branches and it worked pretty well. You do have to set some things up differently tho - keep meaning to write a blog post or something about this. The need to have access to a production schema is an additional hurdle, but well worth it for the additional testability and robustness, in my experience.

On Wednesday, 21 February 2018 18:44:30 UTC+11, Andrew Godwin wrote:
Hi,

This was an approach we deliberately avoided during development of the current migration system - it works poorly with diverging feature branches and requires that your developers have access to production schema at all times (and additionally that you don't have divergent production/staging/QA deploys, like many environments do).

I think it would be a fine addition as an optional third-party tool, but it's not something that really makes sense to include in Django at this point I think.

Andrew

On Mon, Feb 19, 2018 at 10:05 PM, djrobstep <[hidden email]> wrote:
Hi all,

Some thoughts on schema migrations that may interest you. By way of background, I'm the author of migra (<a href="https://github.com/djrobstep/migra" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjrobstep%2Fmigra\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF7uJiFZ_Yq8IJQceciZW69Xq-kIg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjrobstep%2Fmigra\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF7uJiFZ_Yq8IJQceciZW69Xq-kIg&#39;;return true;">https://github.com/djrobstep/migra), a schema comparison/diff tool for Postgres.

I initially wrote this tool because I wanted to be able to generate migration scripts automatically, without needing historical migration files.

It means that you can do things like sync your database to your models mostly automatically, and also explicitly test for a matching schema. Instead of a chain of migration files you only ever need to keep track of one, containing any pending changes).

I've used this approach with success on a number of projects now, and it seems to work pretty well. I also talked about this approach to migrations at the most recent PostgresOpen (<a href="https://www.youtube.com/watch?v=xr498W8oMRo" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.youtube.com/watch?v\x3dxr498W8oMRo&#39;;return true;" onclick="this.href=&#39;https://www.youtube.com/watch?v\x3dxr498W8oMRo&#39;;return true;">https://www.youtube.com/watch?v=xr498W8oMRo).

The downside of this approach is that it's slightly more operationally complex (you need have a temporary copy of your production schema available for comparison purposes, typically a --schema-only dump file) and also that it's postgres-only at this point. It's also conceptually quite different to the typical Django/Rails style of migrations which represents a barrier to entry.

In spite of this, I think doing migrations this way has some compelling advantages, that are relevant to users of Django.

I'm interested in your opinions on this topic as Django developers. What do you think? Is it something worth offering as an option for managing Django migrations? If your feelings are positive, I'm happy to take a look at extending the code accordingly.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/django-developers" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/d0b7ce82-d2fb-4722-a6bb-74d2c670e283%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.