Default to BigAutoField

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Default to BigAutoField

Kenneth Reitz-4

Dear Django Dev,

 

At Heroku, we have the privilege of seeing an extremely broad range of customers utilizing tools like Django to build their applications and companies. One of the things that we’ve seen customers hit, time and time again when using tools like Django, is integer overflows for primary keys. Their application starts behaving unpredictably once they reach the overflow, not even knowing such a constraint exists, and they often think the problem is with their database provider, rather than with their schema. Once they realize what is wrong, it’s a relatively trivial fix, but a migration can take several hours to complete, which results in unacceptable amounts of downtime.

 

Because of this, Heroku, as a company, would like to encourage bigints as a sane reasonable default for primary keys for application models. If the Django project agrees with this idea, that would mean that Django would provide BigAutoField as the default for ‘id’ instead of AutoField.

 

Rails made this change recently, and has seen success in doing so.

 

I’m happy to provide the code to do this, but I wanted to discuss it here before doing so, to hear what the general consensus was to the proposal of such a change.

 

 

Pros:

  • Users of Django, following the docs, won’t accidentally hit the int overflow barrier.

  • Encourages best-practices from the beginning.

  • Bigints don’t take up much more storage than ints when using Postgres.

  • In-line with other frameworks moving forward on this issue, like Rails.

 

Cons:

  • Backwards compatibility would need to be considered.

 

Why not UUID?

 

I agree! I love using UUID for my primary keys, and I think a patch to Django which provides an AutoUUIDField would be wonderful. However, there are a few major drawbacks to making this the new default:

 

  1. It’s confusing to new users, would make onboarding process more difficult.

  2. UUID is difficult to implement in MySQL.

  3. UUID has larger storage requirements.

  4. Incrementing IDs are actually useful.

 


So, my proposal is to simply lift the int barrier to a bigint barrier for new Django applications, and I think it will save a lot of developers a lot of pain in the long run.

 

Many thanks,

 

Kenneth Reitz

Heroku Python


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

Jacob Kaplan-Moss-2
I think this would be a good improvement, and I'd like to see it. I've been bitten by integers overflowing at least twice I can remember in my career, which is two times too many.

However, a major thing we'd have to work out is the upgrade path Consider a simple model:

    class Person(Model):
        name = CharField()

In Django 1.11, this actually generates a model with an integer `id` field. But in we change it, in Django vNext, that `id` field would "turn into" a bigint magically without the underlying table changes. That'd be confusing: you'd expect the model to be "fixed" by pugrading to vNext, but it wouldn't be. I think the migrations engine would detect this as a migration (?), so perhaps that's the path forward, but it could still be super-confusing. We've never shipped a release of Django that required a migration to _all_ your models.

Have you thought about what the upgrade path should look like, Kenneth?

Jacob

On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz <[hidden email]> wrote:

Dear Django Dev,

 

At Heroku, we have the privilege of seeing an extremely broad range of customers utilizing tools like Django to build their applications and companies. One of the things that we’ve seen customers hit, time and time again when using tools like Django, is integer overflows for primary keys. Their application starts behaving unpredictably once they reach the overflow, not even knowing such a constraint exists, and they often think the problem is with their database provider, rather than with their schema. Once they realize what is wrong, it’s a relatively trivial fix, but a migration can take several hours to complete, which results in unacceptable amounts of downtime.

 

Because of this, Heroku, as a company, would like to encourage bigints as a sane reasonable default for primary keys for application models. If the Django project agrees with this idea, that would mean that Django would provide BigAutoField as the default for ‘id’ instead of AutoField.

 

Rails made this change recently, and has seen success in doing so.

 

I’m happy to provide the code to do this, but I wanted to discuss it here before doing so, to hear what the general consensus was to the proposal of such a change.

 

 

Pros:

  • Users of Django, following the docs, won’t accidentally hit the int overflow barrier.

  • Encourages best-practices from the beginning.

  • Bigints don’t take up much more storage than ints when using Postgres.

  • In-line with other frameworks moving forward on this issue, like Rails.

 

Cons:

  • Backwards compatibility would need to be considered.

 

Why not UUID?

 

I agree! I love using UUID for my primary keys, and I think a patch to Django which provides an AutoUUIDField would be wonderful. However, there are a few major drawbacks to making this the new default:

 

  1. It’s confusing to new users, would make onboarding process more difficult.

  2. UUID is difficult to implement in MySQL.

  3. UUID has larger storage requirements.

  4. Incrementing IDs are actually useful.

 


So, my proposal is to simply lift the int barrier to a bigint barrier for new Django applications, and I think it will save a lot of developers a lot of pain in the long run.

 

Many thanks,

 

Kenneth Reitz

Heroku Python


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAK8PqJELsQdpKEcg-a_iEKKwoYkoGmc8pikFM-hgG4g0CZE8XQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

Tom Forbes
How would this work with generic foreign keys as well? Those migrations couldn't be automatic.

On 9 Jun 2017 20:43, "Jacob Kaplan-Moss" <[hidden email]> wrote:
I think this would be a good improvement, and I'd like to see it. I've been bitten by integers overflowing at least twice I can remember in my career, which is two times too many.

However, a major thing we'd have to work out is the upgrade path Consider a simple model:

    class Person(Model):
        name = CharField()

In Django 1.11, this actually generates a model with an integer `id` field. But in we change it, in Django vNext, that `id` field would "turn into" a bigint magically without the underlying table changes. That'd be confusing: you'd expect the model to be "fixed" by pugrading to vNext, but it wouldn't be. I think the migrations engine would detect this as a migration (?), so perhaps that's the path forward, but it could still be super-confusing. We've never shipped a release of Django that required a migration to _all_ your models.

Have you thought about what the upgrade path should look like, Kenneth?

Jacob

On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz <[hidden email]> wrote:

Dear Django Dev,

 

At Heroku, we have the privilege of seeing an extremely broad range of customers utilizing tools like Django to build their applications and companies. One of the things that we’ve seen customers hit, time and time again when using tools like Django, is integer overflows for primary keys. Their application starts behaving unpredictably once they reach the overflow, not even knowing such a constraint exists, and they often think the problem is with their database provider, rather than with their schema. Once they realize what is wrong, it’s a relatively trivial fix, but a migration can take several hours to complete, which results in unacceptable amounts of downtime.

 

Because of this, Heroku, as a company, would like to encourage bigints as a sane reasonable default for primary keys for application models. If the Django project agrees with this idea, that would mean that Django would provide BigAutoField as the default for ‘id’ instead of AutoField.

 

Rails made this change recently, and has seen success in doing so.

 

I’m happy to provide the code to do this, but I wanted to discuss it here before doing so, to hear what the general consensus was to the proposal of such a change.

 

 

Pros:

  • Users of Django, following the docs, won’t accidentally hit the int overflow barrier.

  • Encourages best-practices from the beginning.

  • Bigints don’t take up much more storage than ints when using Postgres.

  • In-line with other frameworks moving forward on this issue, like Rails.

 

Cons:

  • Backwards compatibility would need to be considered.

 

Why not UUID?

 

I agree! I love using UUID for my primary keys, and I think a patch to Django which provides an AutoUUIDField would be wonderful. However, there are a few major drawbacks to making this the new default:

 

  1. It’s confusing to new users, would make onboarding process more difficult.

  2. UUID is difficult to implement in MySQL.

  3. UUID has larger storage requirements.

  4. Incrementing IDs are actually useful.

 


So, my proposal is to simply lift the int barrier to a bigint barrier for new Django applications, and I think it will save a lot of developers a lot of pain in the long run.

 

Many thanks,

 

Kenneth Reitz

Heroku Python


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAK8PqJELsQdpKEcg-a_iEKKwoYkoGmc8pikFM-hgG4g0CZE8XQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJNmxUNXRpHNdiFiwqbPQXe%3D8kcKY6ca7HADC4g%3DGtW-Vw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

Kenneth Reitz-4
My initial thought was to just have this apply to *new* Django applications, if that's possible.

Running this migration could take quite some time on some larger apps, and would require a non-trivial amount of time to execute. 

--
Kenneth Reitz

On Jun 9, 2017, at 3:53 PM, Tom Forbes <[hidden email]> wrote:

How would this work with generic foreign keys as well? Those migrations couldn't be automatic.

On 9 Jun 2017 20:43, "Jacob Kaplan-Moss" <[hidden email]> wrote:
I think this would be a good improvement, and I'd like to see it. I've been bitten by integers overflowing at least twice I can remember in my career, which is two times too many.

However, a major thing we'd have to work out is the upgrade path Consider a simple model:

    class Person(Model):
        name = CharField()

In Django 1.11, this actually generates a model with an integer `id` field. But in we change it, in Django vNext, that `id` field would "turn into" a bigint magically without the underlying table changes. That'd be confusing: you'd expect the model to be "fixed" by pugrading to vNext, but it wouldn't be. I think the migrations engine would detect this as a migration (?), so perhaps that's the path forward, but it could still be super-confusing. We've never shipped a release of Django that required a migration to _all_ your models.

Have you thought about what the upgrade path should look like, Kenneth?

Jacob

On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz <[hidden email]> wrote:
Dear Django Dev,

 

At Heroku, we have the privilege of seeing an extremely broad range of customers utilizing tools like Django to build their applications and companies. One of the things that we’ve seen customers hit, time and time again when using tools like Django, is integer overflows for primary keys. Their application starts behaving unpredictably once they reach the overflow, not even knowing such a constraint exists, and they often think the problem is with their database provider, rather than with their schema. Once they realize what is wrong, it’s a relatively trivial fix, but a migration can take several hours to complete, which results in unacceptable amounts of downtime.

 

Because of this, Heroku, as a company, would like to encourage bigints as a sane reasonable default for primary keys for application models. If the Django project agrees with this idea, that would mean that Django would provide BigAutoField as the default for ‘id’ instead of AutoField.

 

Rails made this change recently, and has seen success in doing so.

 

I’m happy to provide the code to do this, but I wanted to discuss it here before doing so, to hear what the general consensus was to the proposal of such a change.

 

 

Pros:
  • Users of Django, following the docs, won’t accidentally hit the int overflow barrier.
  • Encourages best-practices from the beginning.
  • Bigints don’t take up much more storage than ints when using Postgres.
  • In-line with other frameworks moving forward on this issue, like Rails.

 

Cons:
  • Backwards compatibility would need to be considered.

 

Why not UUID?

 

I agree! I love using UUID for my primary keys, and I think a patch to Django which provides an AutoUUIDField would be wonderful. However, there are a few major drawbacks to making this the new default:

 

  1. It’s confusing to new users, would make onboarding process more difficult.
  2. UUID is difficult to implement in MySQL.
  3. UUID has larger storage requirements.
  4. Incrementing IDs are actually useful.

 


So, my proposal is to simply lift the int barrier to a bigint barrier for new Django applications, and I think it will save a lot of developers a lot of pain in the long run.

 

Many thanks,

 

Kenneth Reitz
Heroku Python


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAK8PqJELsQdpKEcg-a_iEKKwoYkoGmc8pikFM-hgG4g0CZE8XQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/imBJwRrtJkk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJNmxUNXRpHNdiFiwqbPQXe%3D8kcKY6ca7HADC4g%3DGtW-Vw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CB8E79C1-A61F-404A-A3FF-9F356CDAE7AF%40heroku.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

Kenneth Reitz-4
However, it should also be noted that those same larger applications are the ones that are likely to run into this problem eventually, so perhaps forcing the migration is the best path moving forward. 

Interested in hearing thoughts about this.

--
Kenneth Reitz

On Jun 9, 2017, at 3:54 PM, Kenneth Reitz <[hidden email]> wrote:

My initial thought was to just have this apply to *new* Django applications, if that's possible.

Running this migration could take quite some time on some larger apps, and would require a non-trivial amount of time to execute. 

--
Kenneth Reitz

On Jun 9, 2017, at 3:53 PM, Tom Forbes <[hidden email]> wrote:

How would this work with generic foreign keys as well? Those migrations couldn't be automatic.

On 9 Jun 2017 20:43, "Jacob Kaplan-Moss" <[hidden email]> wrote:
I think this would be a good improvement, and I'd like to see it. I've been bitten by integers overflowing at least twice I can remember in my career, which is two times too many.

However, a major thing we'd have to work out is the upgrade path Consider a simple model:

    class Person(Model):
        name = CharField()

In Django 1.11, this actually generates a model with an integer `id` field. But in we change it, in Django vNext, that `id` field would "turn into" a bigint magically without the underlying table changes. That'd be confusing: you'd expect the model to be "fixed" by pugrading to vNext, but it wouldn't be. I think the migrations engine would detect this as a migration (?), so perhaps that's the path forward, but it could still be super-confusing. We've never shipped a release of Django that required a migration to _all_ your models.

Have you thought about what the upgrade path should look like, Kenneth?

Jacob

On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz <[hidden email]> wrote:
Dear Django Dev,

 

At Heroku, we have the privilege of seeing an extremely broad range of customers utilizing tools like Django to build their applications and companies. One of the things that we’ve seen customers hit, time and time again when using tools like Django, is integer overflows for primary keys. Their application starts behaving unpredictably once they reach the overflow, not even knowing such a constraint exists, and they often think the problem is with their database provider, rather than with their schema. Once they realize what is wrong, it’s a relatively trivial fix, but a migration can take several hours to complete, which results in unacceptable amounts of downtime.

 

Because of this, Heroku, as a company, would like to encourage bigints as a sane reasonable default for primary keys for application models. If the Django project agrees with this idea, that would mean that Django would provide BigAutoField as the default for ‘id’ instead of AutoField.

 

Rails made this change recently, and has seen success in doing so.

 

I’m happy to provide the code to do this, but I wanted to discuss it here before doing so, to hear what the general consensus was to the proposal of such a change.

 

 

Pros:
  • Users of Django, following the docs, won’t accidentally hit the int overflow barrier.
  • Encourages best-practices from the beginning.
  • Bigints don’t take up much more storage than ints when using Postgres.
  • In-line with other frameworks moving forward on this issue, like Rails.

 

Cons:
  • Backwards compatibility would need to be considered.

 

Why not UUID?

 

I agree! I love using UUID for my primary keys, and I think a patch to Django which provides an AutoUUIDField would be wonderful. However, there are a few major drawbacks to making this the new default:

 

  1. It’s confusing to new users, would make onboarding process more difficult.
  2. UUID is difficult to implement in MySQL.
  3. UUID has larger storage requirements.
  4. Incrementing IDs are actually useful.

 


So, my proposal is to simply lift the int barrier to a bigint barrier for new Django applications, and I think it will save a lot of developers a lot of pain in the long run.

 

Many thanks,

 

Kenneth Reitz
Heroku Python


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAK8PqJELsQdpKEcg-a_iEKKwoYkoGmc8pikFM-hgG4g0CZE8XQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/imBJwRrtJkk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJNmxUNXRpHNdiFiwqbPQXe%3D8kcKY6ca7HADC4g%3DGtW-Vw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/FC19F608-C8EF-4AF6-9B00-2E78D6099E81%40heroku.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

Curtis Maloney-2
In reply to this post by Jacob Kaplan-Moss-2
I know people hate settings, but what about one for auto id field type?

It would let you handle backwards compatibility, uuid, and bigint...

--
C


On 10 June 2017 5:42:42 AM AEST, Jacob Kaplan-Moss <[hidden email]> wrote:
I think this would be a good improvement, and I'd like to see it. I've been bitten by integers overflowing at least twice I can remember in my career, which is two times too many.

However, a major thing we'd have to work out is the upgrade path Consider a simple model:

    class Person(Model):
        name = CharField()

In Django 1.11, this actually generates a model with an integer `id` field. But in we change it, in Django vNext, that `id` field would "turn into" a bigint magically without the underlying table changes. That'd be confusing: you'd expect the model to be "fixed" by pugrading to vNext, but it wouldn't be. I think the migrations engine would detect this as a migration (?), so perhaps that's the path forward, but it could still be super-confusing. We've never shipped a release of Django that required a migration to _all_ your models.

Have you thought about what the upgrade path should look like, Kenneth?

Jacob

On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz <[hidden email]> wrote:

Dear Django Dev,

 

At Heroku, we have the privilege of seeing an extremely broad range of customers utilizing tools like Django to build their applications and companies. One of the things that we’ve seen customers hit, time and time again when using tools like Django, is integer overflows for primary keys. Their application starts behaving unpredictably once they reach the overflow, not even knowing such a constraint exists, and they often think the problem is with their database provider, rather than with their schema. Once they realize what is wrong, it’s a relatively trivial fix, but a migration can take several hours to complete, which results in unacceptable amounts of downtime.

 

Because of this, Heroku, as a company, would like to encourage bigints as a sane reasonable default for primary keys for application models. If the Django project agrees with this idea, that would mean that Django would provide BigAutoField as the default for ‘id’ instead of AutoField.

 

Rails made this change recently, and has seen success in doing so.

 

I’m happy to provide the code to do this, but I wanted to discuss it here before doing so, to hear what the general consensus was to the proposal of such a change.

 

 

Pros:

  • Users of Django, following the docs, won’t accidentally hit the int overflow barrier.

  • Encourages best-practices from the beginning.

  • Bigints don’t take up much more storage than ints when using Postgres.

  • In-line with other frameworks moving forward on this issue, like Rails.

 

Cons:

  • Backwards compatibility would need to be considered.

 

Why not UUID?

 

I agree! I love using UUID for my primary keys, and I think a patch to Django which provides an AutoUUIDField would be wonderful. However, there are a few major drawbacks to making this the new default:

 

  1. It’s confusing to new users, would make onboarding process more difficult.

  2. UUID is difficult to implement in MySQL.

  3. UUID has larger storage requirements.

  4. Incrementing IDs are actually useful.

 


So, my proposal is to simply lift the int barrier to a bigint barrier for new Django applications, and I think it will save a lot of developers a lot of pain in the long run.

 

Many thanks,

 

Kenneth Reitz

Heroku Python


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9E7797E5-3F14-4E1D-AEB9-950BB45F87E6%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

Tim Graham-2
I'm not sure how this could work with migrations. In a sense, it would involve making the auto-generated primary key "swappable", including foreign keys that point to it. This sounds like a headache.

I haven't thought of a possible solution since Kenneth floated this idea in #django-dev yesterday.

On Friday, June 9, 2017 at 7:11:05 PM UTC-4, Curtis Maloney wrote:
I know people hate settings, but what about one for auto id field type?

It would let you handle backwards compatibility, uuid, and bigint...

--
C


On 10 June 2017 5:42:42 AM AEST, Jacob Kaplan-Moss <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="-DtWC3pRBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">ja...@...> wrote:
I think this would be a good improvement, and I'd like to see it. I've been bitten by integers overflowing at least twice I can remember in my career, which is two times too many.

However, a major thing we'd have to work out is the upgrade path Consider a simple model:

    class Person(Model):
        name = CharField()

In Django 1.11, this actually generates a model with an integer `id` field. But in we change it, in Django vNext, that `id` field would "turn into" a bigint magically without the underlying table changes. That'd be confusing: you'd expect the model to be "fixed" by pugrading to vNext, but it wouldn't be. I think the migrations engine would detect this as a migration (?), so perhaps that's the path forward, but it could still be super-confusing. We've never shipped a release of Django that required a migration to _all_ your models.

Have you thought about what the upgrade path should look like, Kenneth?

Jacob

On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="-DtWC3pRBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">ken...@...> wrote:

Dear Django Dev,

 

At Heroku, we have the privilege of seeing an extremely broad range of customers utilizing tools like Django to build their applications and companies. One of the things that we’ve seen customers hit, time and time again when using tools like Django, is integer overflows for primary keys. Their application starts behaving unpredictably once they reach the overflow, not even knowing such a constraint exists, and they often think the problem is with their database provider, rather than with their schema. Once they realize what is wrong, it’s a relatively trivial fix, but a migration can take several hours to complete, which results in unacceptable amounts of downtime.

 

Because of this, Heroku, as a company, would like to encourage bigints as a sane reasonable default for primary keys for application models. If the Django project agrees with this idea, that would mean that Django would provide BigAutoField as the default for ‘id’ instead of AutoField.

 

Rails <a href="http://www.mccartie.com/2016/12/05/rails-5.1.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.mccartie.com%2F2016%2F12%2F05%2Frails-5.1.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEP1Czxx4n_TafzcoUQqPoo-UUpNA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.mccartie.com%2F2016%2F12%2F05%2Frails-5.1.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEP1Czxx4n_TafzcoUQqPoo-UUpNA&#39;;return true;">made this change recently, and has seen success in doing so.

 

I’m happy to provide the code to do this, but I wanted to discuss it here before doing so, to hear what the general consensus was to the proposal of such a change.

 

 

Pros:

  • Users of Django, following the docs, won’t accidentally hit the int overflow barrier.

  • Encourages best-practices from the beginning.

  • Bigints don’t take up much more storage than ints when using Postgres.

  • In-line with other frameworks moving forward on this issue, like <a href="http://www.mccartie.com/2016/12/05/rails-5.1.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.mccartie.com%2F2016%2F12%2F05%2Frails-5.1.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEP1Czxx4n_TafzcoUQqPoo-UUpNA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.mccartie.com%2F2016%2F12%2F05%2Frails-5.1.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEP1Czxx4n_TafzcoUQqPoo-UUpNA&#39;;return true;">Rails.

 

Cons:

  • Backwards compatibility would need to be considered.

 

Why not UUID?

 

I agree! I love using UUID for my primary keys, and I think a patch to Django which provides an AutoUUIDField would be wonderful. However, there are a few major drawbacks to making this the new default:

 

  1. It’s confusing to new users, would make onboarding process more difficult.

  2. UUID is difficult to implement in MySQL.

  3. UUID has larger storage requirements.

  4. Incrementing IDs are actually useful.

 


So, my proposal is to simply lift the int barrier to a bigint barrier for new Django applications, and I think it will save a lot of developers a lot of pain in the long run.

 

Many thanks,

 

Kenneth Reitz

Heroku Python


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="-DtWC3pRBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-develop...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="-DtWC3pRBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-d...@googlegroups.com.
Visit this group at <a href="https://groups.google.com/group/django-developers" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.


--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3753d789-264b-4771-9949-06283ee5c6f2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

Curtis Maloney-2
I don't see it helping once you hit the problem, but like custom user its a recommendable setting for pre planning

--
C


On 10 June 2017 11:37:04 AM AEST, Tim Graham <[hidden email]> wrote:
I'm not sure how this could work with migrations. In a sense, it would involve making the auto-generated primary key "swappable", including foreign keys that point to it. This sounds like a headache.

I haven't thought of a possible solution since Kenneth floated this idea in #django-dev yesterday.

On Friday, June 9, 2017 at 7:11:05 PM UTC-4, Curtis Maloney wrote:
I know people hate settings, but what about one for auto id field type?

It would let you handle backwards compatibility, uuid, and bigint...

--
C


On 10 June 2017 5:42:42 AM AEST, Jacob Kaplan-Moss <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="-DtWC3pRBwAJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">ja...@...> wrote:
I think this would be a good improvement, and I'd like to see it. I've been bitten by integers overflowing at least twice I can remember in my career, which is two times too many.

However, a major thing we'd have to work out is the upgrade path Consider a simple model:

    class Person(Model):
        name = CharField()

In Django 1.11, this actually generates a model with an integer `id` field. But in we change it, in Django vNext, that `id` field would "turn into" a bigint magically without the underlying table changes. That'd be confusing: you'd expect the model to be "fixed" by pugrading to vNext, but it wouldn't be. I think the migrations engine would detect this as a migration (?), so perhaps that's the path forward, but it could still be super-confusing. We've never shipped a release of Django that required a migration to _all_ your models.

Have you thought about what the upgrade path should look like, Kenneth?

Jacob

On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="-DtWC3pRBwAJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">ken...@...> wrote:

Dear Django Dev,

 

At Heroku, we have the privilege of seeing an extremely broad range of customers utilizing tools like Django to build their applications and companies. One of the things that we’ve seen customers hit, time and time again when using tools like Django, is integer overflows for primary keys. Their application starts behaving unpredictably once they reach the overflow, not even knowing such a constraint exists, and they often think the problem is with their database provider, rather than with their schema. Once they realize what is wrong, it’s a relatively trivial fix, but a migration can take several hours to complete, which results in unacceptable amounts of downtime.

 

Because of this, Heroku, as a company, would like to encourage bigints as a sane reasonable default for primary keys for application models. If the Django project agrees with this idea, that would mean that Django would provide BigAutoField as the default for ‘id’ instead of AutoField.

 

Rails <a href="http://www.mccartie.com/2016/12/05/rails-5.1.html" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.mccartie.com%2F2016%2F12%2F05%2Frails-5.1.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEP1Czxx4n_TafzcoUQqPoo-UUpNA';return true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.mccartie.com%2F2016%2F12%2F05%2Frails-5.1.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEP1Czxx4n_TafzcoUQqPoo-UUpNA';return true;">made this change recently, and has seen success in doing so.

 

I’m happy to provide the code to do this, but I wanted to discuss it here before doing so, to hear what the general consensus was to the proposal of such a change.

 

 

Pros:

  • Users of Django, following the docs, won’t accidentally hit the int overflow barrier.

  • Encourages best-practices from the beginning.

  • Bigints don’t take up much more storage than ints when using Postgres.

  • In-line with other frameworks moving forward on this issue, like <a href="http://www.mccartie.com/2016/12/05/rails-5.1.html" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.mccartie.com%2F2016%2F12%2F05%2Frails-5.1.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEP1Czxx4n_TafzcoUQqPoo-UUpNA';return true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.mccartie.com%2F2016%2F12%2F05%2Frails-5.1.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEP1Czxx4n_TafzcoUQqPoo-UUpNA';return true;">Rails.

 

Cons:

  • Backwards compatibility would need to be considered.

 

Why not UUID?

 

I agree! I love using UUID for my primary keys, and I think a patch to Django which provides an AutoUUIDField would be wonderful. However, there are a few major drawbacks to making this the new default:

 

  1. It’s confusing to new users, would make onboarding process more difficult.

  2. UUID is difficult to implement in MySQL.

  3. UUID has larger storage requirements.

  4. Incrementing IDs are actually useful.

 


So, my proposal is to simply lift the int barrier to a bigint barrier for new Django applications, and I think it will save a lot of developers a lot of pain in the long run.

 

Many thanks,

 

Kenneth Reitz

Heroku Python


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="-DtWC3pRBwAJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">django-develop...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="-DtWC3pRBwAJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">django-d...@googlegroups.com.
Visit this group at <a href="https://groups.google.com/group/django-developers" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/group/django-developers';return true;" onclick="this.href='https://groups.google.com/group/django-developers';return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter';return true;" onclick="this.href='https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter';return true;">https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.


--
Sent from my Android device with K-9 Mail. Please excuse my brevity.


--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/17F1F6AE-807B-4BB7-908C-DD40A7B7A6E3%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

Collin Anderson-2
In reply to this post by Tim Graham-2
I might be wrong, but if the default changes, won't the migrations detect it and migrate it just fine, including foreign keys?

All of my migrations have this:
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),

So everyone would either need to manually specify the AutoField to keep the old behavior, or run makemigrations to auto-generate migrations to BigAutoField. This seems similar to increasing the max_length of EmailField, user.username, and user.last_name, though would affect a lot more models in this case.

(I'm not sure what I think about the setting idea.)

(While we're at it, maybe we could make it a (new) PositiveBigAutoField to help out the mysql folks and close the oldest open ticket:



On Fri, Jun 9, 2017 at 9:37 PM, Tim Graham <[hidden email]> wrote:
I'm not sure how this could work with migrations. In a sense, it would involve making the auto-generated primary key "swappable", including foreign keys that point to it. This sounds like a headache.

I haven't thought of a possible solution since Kenneth floated this idea in #django-dev yesterday.

On Friday, June 9, 2017 at 7:11:05 PM UTC-4, Curtis Maloney wrote:
I know people hate settings, but what about one for auto id field type?

It would let you handle backwards compatibility, uuid, and bigint...

--
C


On 10 June 2017 5:42:42 AM AEST, Jacob Kaplan-Moss <[hidden email]> wrote:
I think this would be a good improvement, and I'd like to see it. I've been bitten by integers overflowing at least twice I can remember in my career, which is two times too many.

However, a major thing we'd have to work out is the upgrade path Consider a simple model:

    class Person(Model):
        name = CharField()

In Django 1.11, this actually generates a model with an integer `id` field. But in we change it, in Django vNext, that `id` field would "turn into" a bigint magically without the underlying table changes. That'd be confusing: you'd expect the model to be "fixed" by pugrading to vNext, but it wouldn't be. I think the migrations engine would detect this as a migration (?), so perhaps that's the path forward, but it could still be super-confusing. We've never shipped a release of Django that required a migration to _all_ your models.

Have you thought about what the upgrade path should look like, Kenneth?

Jacob

On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz <[hidden email]> wrote:

Dear Django Dev,

 

At Heroku, we have the privilege of seeing an extremely broad range of customers utilizing tools like Django to build their applications and companies. One of the things that we’ve seen customers hit, time and time again when using tools like Django, is integer overflows for primary keys. Their application starts behaving unpredictably once they reach the overflow, not even knowing such a constraint exists, and they often think the problem is with their database provider, rather than with their schema. Once they realize what is wrong, it’s a relatively trivial fix, but a migration can take several hours to complete, which results in unacceptable amounts of downtime.

 

Because of this, Heroku, as a company, would like to encourage bigints as a sane reasonable default for primary keys for application models. If the Django project agrees with this idea, that would mean that Django would provide BigAutoField as the default for ‘id’ instead of AutoField.

 

Rails made this change recently, and has seen success in doing so.

 

I’m happy to provide the code to do this, but I wanted to discuss it here before doing so, to hear what the general consensus was to the proposal of such a change.

 

 

Pros:

  • Users of Django, following the docs, won’t accidentally hit the int overflow barrier.

  • Encourages best-practices from the beginning.

  • Bigints don’t take up much more storage than ints when using Postgres.

  • In-line with other frameworks moving forward on this issue, like Rails.

 

Cons:

  • Backwards compatibility would need to be considered.

 

Why not UUID?

 

I agree! I love using UUID for my primary keys, and I think a patch to Django which provides an AutoUUIDField would be wonderful. However, there are a few major drawbacks to making this the new default:

 

  1. It’s confusing to new users, would make onboarding process more difficult.

  2. UUID is difficult to implement in MySQL.

  3. UUID has larger storage requirements.

  4. Incrementing IDs are actually useful.

 


So, my proposal is to simply lift the int barrier to a bigint barrier for new Django applications, and I think it will save a lot of developers a lot of pain in the long run.

 

Many thanks,

 

Kenneth Reitz

Heroku Python


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3753d789-264b-4771-9949-06283ee5c6f2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFO84S6BL3XD2KZjHa99q0tuELkGgbZHqnv-2qBpDjvog6jH8A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

Andrew Godwin-3
As long as you changed the actual field that ended up under the "id" column, then yes, migrations should detect it and apply all the changes during an upgrade, including to foreign keys.

Generic foreign keys are another problem, however, not to mention the prospect of how many migrations and ALTER TABLES this would result in on a large project. We could include in the upgrade notes how to stave it off (explicitly specify an ID field), but I'm sort of on the fence as to if that's good enough.

Andrew

On Sat, Jun 10, 2017 at 9:53 AM, Collin Anderson <[hidden email]> wrote:
I might be wrong, but if the default changes, won't the migrations detect it and migrate it just fine, including foreign keys?

All of my migrations have this:
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),

So everyone would either need to manually specify the AutoField to keep the old behavior, or run makemigrations to auto-generate migrations to BigAutoField. This seems similar to increasing the max_length of EmailField, user.username, and user.last_name, though would affect a lot more models in this case.

(I'm not sure what I think about the setting idea.)

(While we're at it, maybe we could make it a (new) PositiveBigAutoField to help out the mysql folks and close the oldest open ticket:



On Fri, Jun 9, 2017 at 9:37 PM, Tim Graham <[hidden email]> wrote:
I'm not sure how this could work with migrations. In a sense, it would involve making the auto-generated primary key "swappable", including foreign keys that point to it. This sounds like a headache.

I haven't thought of a possible solution since Kenneth floated this idea in #django-dev yesterday.

On Friday, June 9, 2017 at 7:11:05 PM UTC-4, Curtis Maloney wrote:
I know people hate settings, but what about one for auto id field type?

It would let you handle backwards compatibility, uuid, and bigint...

--
C


On 10 June 2017 5:42:42 AM AEST, Jacob Kaplan-Moss <[hidden email]> wrote:
I think this would be a good improvement, and I'd like to see it. I've been bitten by integers overflowing at least twice I can remember in my career, which is two times too many.

However, a major thing we'd have to work out is the upgrade path Consider a simple model:

    class Person(Model):
        name = CharField()

In Django 1.11, this actually generates a model with an integer `id` field. But in we change it, in Django vNext, that `id` field would "turn into" a bigint magically without the underlying table changes. That'd be confusing: you'd expect the model to be "fixed" by pugrading to vNext, but it wouldn't be. I think the migrations engine would detect this as a migration (?), so perhaps that's the path forward, but it could still be super-confusing. We've never shipped a release of Django that required a migration to _all_ your models.

Have you thought about what the upgrade path should look like, Kenneth?

Jacob

On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz <[hidden email]> wrote:

Dear Django Dev,

 

At Heroku, we have the privilege of seeing an extremely broad range of customers utilizing tools like Django to build their applications and companies. One of the things that we’ve seen customers hit, time and time again when using tools like Django, is integer overflows for primary keys. Their application starts behaving unpredictably once they reach the overflow, not even knowing such a constraint exists, and they often think the problem is with their database provider, rather than with their schema. Once they realize what is wrong, it’s a relatively trivial fix, but a migration can take several hours to complete, which results in unacceptable amounts of downtime.

 

Because of this, Heroku, as a company, would like to encourage bigints as a sane reasonable default for primary keys for application models. If the Django project agrees with this idea, that would mean that Django would provide BigAutoField as the default for ‘id’ instead of AutoField.

 

Rails made this change recently, and has seen success in doing so.

 

I’m happy to provide the code to do this, but I wanted to discuss it here before doing so, to hear what the general consensus was to the proposal of such a change.

 

 

Pros:

  • Users of Django, following the docs, won’t accidentally hit the int overflow barrier.

  • Encourages best-practices from the beginning.

  • Bigints don’t take up much more storage than ints when using Postgres.

  • In-line with other frameworks moving forward on this issue, like Rails.

 

Cons:

  • Backwards compatibility would need to be considered.

 

Why not UUID?

 

I agree! I love using UUID for my primary keys, and I think a patch to Django which provides an AutoUUIDField would be wonderful. However, there are a few major drawbacks to making this the new default:

 

  1. It’s confusing to new users, would make onboarding process more difficult.

  2. UUID is difficult to implement in MySQL.

  3. UUID has larger storage requirements.

  4. Incrementing IDs are actually useful.

 


So, my proposal is to simply lift the int barrier to a bigint barrier for new Django applications, and I think it will save a lot of developers a lot of pain in the long run.

 

Many thanks,

 

Kenneth Reitz

Heroku Python


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3753d789-264b-4771-9949-06283ee5c6f2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFO84S6BL3XD2KZjHa99q0tuELkGgbZHqnv-2qBpDjvog6jH8A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1upLQfsf7y1ajEp2tARooELASCmDoxkErdU0SL9w6cQmgQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

Claude Paroz
In reply to this post by Kenneth Reitz-4
I think we should first discuss if it makes sense to set a BigAutoField by default for all tables. I'm not convinced of that yet. I looked at my projects with this perspective and found for example a case where I have many small lookup tables (containing between 2-20 entries) for which I know I would never use BigAutoField if I'd design the schema on paper.

For me, it's a bit like the `on_delete` parameter for foreign keys. A no "one size fits all" situation.
For example, a quick analysis of contrib models (sorry for bad formatting):

Model                                 BigAutoField appropriate
=======================
admin.LogEntry                 yes
auth.User                            no
auth.Group                          no
auth.Permission                 no
contenttype.ContentType no
flatpage.FlatPage              no
redirect.Redirect                no
sessions.Session              yes
sites.Site                             no

Shouldn't we treat that issue by better documentation instead?
Another idea is to leverage the system check framework (--deploy part) to warn when the max id is over 50% of available range.
We are perfectionists, aren't we :-)

Claude

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/0fba4c2c-cd56-4c7d-82b9-7be0a7a3d233%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

Curtis Maloney-2
Right, hence my point of having a global setting to say "the default
auto-field is ..."

This would:
a) let people continue to use AutoField
b) let people opt for BigAutoField
c) let 3rd party projects be agnostic
d) let people use UUIDField(default=uuid.uuid4)
e) possibly make it feasible to change mid-stream from AutoField to
BigAutoField...

--
C

On 10/06/17 19:33, Claude Paroz wrote:

> I think we should first discuss if it makes sense to set a BigAutoField
> by default for all tables. I'm not convinced of that yet. I looked at my
> projects with this perspective and found for example a case where I have
> many small lookup tables (containing between 2-20 entries) for which I
> know I would never use BigAutoField if I'd design the schema on paper.
>
> For me, it's a bit like the `on_delete` parameter for foreign keys. A no
> "one size fits all" situation.
> For example, a quick analysis of contrib models (sorry for bad formatting):
>
> Model                                 BigAutoField appropriate
> =======================
> admin.LogEntry                 yes
> auth.User                            no
> auth.Group                          no
> auth.Permission                 no
> contenttype.ContentType no
> flatpage.FlatPage              no
> redirect.Redirect                no
> sessions.Session              yes
> sites.Site                             no
>
> Shouldn't we treat that issue by better documentation instead?
> Another idea is to leverage the system check framework (--deploy part)
> to warn when the max id is over 50% of available range.
> We are perfectionists, aren't we :-)
>
> Claude
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [hidden email]
> <mailto:[hidden email]>.
> To post to this group, send email to [hidden email]
> <mailto:[hidden email]>.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/0fba4c2c-cd56-4c7d-82b9-7be0a7a3d233%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/0fba4c2c-cd56-4c7d-82b9-7be0a7a3d233%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/43c2a63f-87de-fda1-87c3-412574c13804%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

Curtis Maloney-2
f) let MySQL users opt for PositiveBigAutoField if they want...


On 10/06/17 19:40, Curtis Maloney wrote:

> Right, hence my point of having a global setting to say "the default
> auto-field is ..."
>
> This would:
> a) let people continue to use AutoField
> b) let people opt for BigAutoField
> c) let 3rd party projects be agnostic
> d) let people use UUIDField(default=uuid.uuid4)
> e) possibly make it feasible to change mid-stream from AutoField to
> BigAutoField...
>
> --
> C
>
> On 10/06/17 19:33, Claude Paroz wrote:
>> I think we should first discuss if it makes sense to set a BigAutoField
>> by default for all tables. I'm not convinced of that yet. I looked at my
>> projects with this perspective and found for example a case where I have
>> many small lookup tables (containing between 2-20 entries) for which I
>> know I would never use BigAutoField if I'd design the schema on paper.
>>
>> For me, it's a bit like the `on_delete` parameter for foreign keys. A no
>> "one size fits all" situation.
>> For example, a quick analysis of contrib models (sorry for bad
>> formatting):
>>
>> Model                                 BigAutoField appropriate
>> =======================
>> admin.LogEntry                 yes
>> auth.User                            no
>> auth.Group                          no
>> auth.Permission                 no
>> contenttype.ContentType no
>> flatpage.FlatPage              no
>> redirect.Redirect                no
>> sessions.Session              yes
>> sites.Site                             no
>>
>> Shouldn't we treat that issue by better documentation instead?
>> Another idea is to leverage the system check framework (--deploy part)
>> to warn when the max id is over 50% of available range.
>> We are perfectionists, aren't we :-)
>>
>> Claude
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to [hidden email]
>> <mailto:[hidden email]>.
>> To post to this group, send email to [hidden email]
>> <mailto:[hidden email]>.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/0fba4c2c-cd56-4c7d-82b9-7be0a7a3d233%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/django-developers/0fba4c2c-cd56-4c7d-82b9-7be0a7a3d233%40googlegroups.com?utm_medium=email&utm_source=footer>.
>>
>> For more options, visit https://groups.google.com/d/optout.
>

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/ec59bb2e-bce9-924f-1509-60709a07c5dc%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

Tom Forbes
I really like the idea of a global configurable setting. One problem with a setting is that it's not always changeable, which settings kind of imply (IMO). Going from int->bigint is always possible, but going backwards may not be, nor might going from int->guid.

I attempted an int->guid migration on one of our systems and ran into some migrations and postgres-specific issues. Seems like it might be incredibly difficult to get a simple toggle setting to work with it, and there are lots of pitfalls.

A warning to the system check framework could be added to check if the primary key type is changed and potentially incompatible generic foreign key columns exist. And in the docs rather than use a plain IntegerField, just use a field that changes based on the setting.

On 10 Jun 2017 10:41, "Curtis Maloney" <[hidden email]> wrote:
f) let MySQL users opt for PositiveBigAutoField if they want...



On 10/06/17 19:40, Curtis Maloney wrote:
Right, hence my point of having a global setting to say "the default
auto-field is ..."

This would:
a) let people continue to use AutoField
b) let people opt for BigAutoField
c) let 3rd party projects be agnostic
d) let people use UUIDField(default=uuid.uuid4)
e) possibly make it feasible to change mid-stream from AutoField to
BigAutoField...

--
C

On 10/06/17 19:33, Claude Paroz wrote:
I think we should first discuss if it makes sense to set a BigAutoField
by default for all tables. I'm not convinced of that yet. I looked at my
projects with this perspective and found for example a case where I have
many small lookup tables (containing between 2-20 entries) for which I
know I would never use BigAutoField if I'd design the schema on paper.

For me, it's a bit like the `on_delete` parameter for foreign keys. A no
"one size fits all" situation.
For example, a quick analysis of contrib models (sorry for bad
formatting):

Model                                 BigAutoField appropriate
=======================
admin.LogEntry                 yes
auth.User                            no
auth.Group                          no
auth.Permission                 no
contenttype.ContentType no
flatpage.FlatPage              no
redirect.Redirect                no
sessions.Session              yes
sites.Site                             no

Shouldn't we treat that issue by better documentation instead?
Another idea is to leverage the system check framework (--deploy part)
to warn when the max id is over 50% of available range.
We are perfectionists, aren't we :-)

Claude

--
You received this message because you are subscribed to the Google
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [hidden email]
<mailto:[hidden email]>.
To post to this group, send email to [hidden email]
<mailto:[hidden email]>.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/0fba4c2c-cd56-4c7d-82b9-7be0a7a3d233%40googlegroups.com

<https://groups.google.com/d/msgid/django-developers/0fba4c2c-cd56-4c7d-82b9-7be0a7a3d233%40googlegroups.com?utm_medium=email&utm_source=footer>.

For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/ec59bb2e-bce9-924f-1509-60709a07c5dc%40tinbrain.net.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJMntP93kaJ%2BbPuHzAr2rKqZbbSbu1TRc%2Bm%2Bp-%2BiO%2BojvQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

Claude Paroz
In reply to this post by Curtis Maloney-2
Le samedi 10 juin 2017 11:40:42 UTC+2, Curtis Maloney a écrit :
Right, hence my point of having a global setting to say "the default
auto-field is ..."

This would:
a) ...

I see, but this conforms to the pattern "use the same id field type for all models of my project". I'm not sure we should encourage that.

Another idea would be to offer variants of models.Model which models could inherit from, like models.BigIntModel or models.UUIDModel.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3904d3f9-2ed8-432a-83c3-3bbac1dacc55%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

Curtis Maloney-2


On 10/06/17 22:21, Claude Paroz wrote:
> Le samedi 10 juin 2017 11:40:42 UTC+2, Curtis Maloney a écrit :
>
>     Right, hence my point of having a global setting to say "the default
>     auto-field is ..."

> I see, but this conforms to the pattern "use the same id field type for
> all models of my project". I'm not sure we should encourage that.

Yeah... naming would be key with "DEFAULT_AUTO_FIELD_TYPE" or whatever...

> Another idea would be to offer variants of models.Model which models
> could inherit from, like models.BigIntModel or models.UUIDModel.

Ah, well... now you're talking.  But then, you can do this already as an
abstract base with TYPE as id...

class BigIntModel(models.Model):
     id = BigAutoField(primary_key=True)
     class Meta:
         abstract = True


--
C

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/a7da67fb-349a-c77c-8324-32dc16bd0ada%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

Claude Paroz
Le samedi 10 juin 2017 14:25:49 UTC+2, Curtis Maloney a écrit :
On 10/06/17 22:21, Claude Paroz wrote:
> Another idea would be to offer variants of models.Model which models
> could inherit from, like models.BigIntModel or models.UUIDModel.

Ah, well... now you're talking.  But then, you can do this already as an
abstract base with TYPE as id...

class BigIntModel(models.Model):
     id = BigAutoField(primary_key=True)
     class Meta:
         abstract = True

Absolutely! It would only be a convenience thing. The more I think about this, the more I think that it might be addressed by better documentation (and maybe some convenience shortcuts).

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/d6cbd561-190e-40b5-8419-aac3e42f66b4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

emorley
In reply to this post by Claude Paroz
On Saturday, 10 June 2017 10:33:35 UTC+1, Claude Paroz wrote:
Another idea is to leverage the system check framework (--deploy part) to warn when the max id is over 50% of available range.

I was about to suggest the same. Seems like something worth doing regardless of whether we change the default (since it would help catch issues in existing installations who haven't yet migrated).

Ed

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/fe0c7429-e9e9-406f-8701-8ebc550641e3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

Ash Christopher-2
In reply to this post by Kenneth Reitz-4
I really like this idea. We created a custom BigAutoField for our systems with sharded databases, and fast growing data.

I understand the desire to fix it for new projects moving forward, but this smells a little like what happened with custom User models - it was introduced for new projects, but there was no clear upgrade/migration path for existing projects. 


On Friday, 9 June 2017 15:54:43 UTC-4, Kenneth Reitz wrote:
My initial thought was to just have this apply to *new* Django applications, if that's possible.

Running this migration could take quite some time on some larger apps, and would require a non-trivial amount of time to execute. 

--
Kenneth Reitz

On Jun 9, 2017, at 3:53 PM, Tom Forbes <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="bm4_0-xhBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">t...@...> wrote:

How would this work with generic foreign keys as well? Those migrations couldn't be automatic.

On 9 Jun 2017 20:43, "Jacob Kaplan-Moss" <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="bm4_0-xhBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">ja...@...> wrote:
I think this would be a good improvement, and I'd like to see it. I've been bitten by integers overflowing at least twice I can remember in my career, which is two times too many.

However, a major thing we'd have to work out is the upgrade path Consider a simple model:

    class Person(Model):
        name = CharField()

In Django 1.11, this actually generates a model with an integer `id` field. But in we change it, in Django vNext, that `id` field would "turn into" a bigint magically without the underlying table changes. That'd be confusing: you'd expect the model to be "fixed" by pugrading to vNext, but it wouldn't be. I think the migrations engine would detect this as a migration (?), so perhaps that's the path forward, but it could still be super-confusing. We've never shipped a release of Django that required a migration to _all_ your models.

Have you thought about what the upgrade path should look like, Kenneth?

Jacob

On Fri, Jun 9, 2017 at 11:24 AM, Kenneth Reitz <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="bm4_0-xhBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">ken...@...> wrote:
Dear Django Dev,

 

At Heroku, we have the privilege of seeing an extremely broad range of customers utilizing tools like Django to build their applications and companies. One of the things that we’ve seen customers hit, time and time again when using tools like Django, is integer overflows for primary keys. Their application starts behaving unpredictably once they reach the overflow, not even knowing such a constraint exists, and they often think the problem is with their database provider, rather than with their schema. Once they realize what is wrong, it’s a relatively trivial fix, but a migration can take several hours to complete, which results in unacceptable amounts of downtime.

 

Because of this, Heroku, as a company, would like to encourage bigints as a sane reasonable default for primary keys for application models. If the Django project agrees with this idea, that would mean that Django would provide BigAutoField as the default for ‘id’ instead of AutoField.

 

Rails <a href="http://www.mccartie.com/2016/12/05/rails-5.1.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.mccartie.com%2F2016%2F12%2F05%2Frails-5.1.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEP1Czxx4n_TafzcoUQqPoo-UUpNA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.mccartie.com%2F2016%2F12%2F05%2Frails-5.1.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEP1Czxx4n_TafzcoUQqPoo-UUpNA&#39;;return true;">made this change recently, and has seen success in doing so.

 

I’m happy to provide the code to do this, but I wanted to discuss it here before doing so, to hear what the general consensus was to the proposal of such a change.

 

 

Pros:
  • Users of Django, following the docs, won’t accidentally hit the int overflow barrier.
  • Encourages best-practices from the beginning.
  • Bigints don’t take up much more storage than ints when using Postgres.
  • In-line with other frameworks moving forward on this issue, like <a href="http://www.mccartie.com/2016/12/05/rails-5.1.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.mccartie.com%2F2016%2F12%2F05%2Frails-5.1.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEP1Czxx4n_TafzcoUQqPoo-UUpNA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.mccartie.com%2F2016%2F12%2F05%2Frails-5.1.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEP1Czxx4n_TafzcoUQqPoo-UUpNA&#39;;return true;">Rails.

 

Cons:
  • Backwards compatibility would need to be considered.

 

Why not UUID?

 

I agree! I love using UUID for my primary keys, and I think a patch to Django which provides an AutoUUIDField would be wonderful. However, there are a few major drawbacks to making this the new default:

 

  1. It’s confusing to new users, would make onboarding process more difficult.
  2. UUID is difficult to implement in MySQL.
  3. UUID has larger storage requirements.
  4. Incrementing IDs are actually useful.

 


So, my proposal is to simply lift the int barrier to a bigint barrier for new Django applications, and I think it will save a lot of developers a lot of pain in the long run.

 

Many thanks,

 

Kenneth Reitz
Heroku Python


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="bm4_0-xhBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-develop...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="bm4_0-xhBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-d...@googlegroups.com.
Visit this group at <a href="https://groups.google.com/group/django-developers" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/django-developers/6fe3401c-4404-4bd8-9d22-58df95cd1348%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="bm4_0-xhBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-develop...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="bm4_0-xhBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-d...@googlegroups.com.
Visit this group at <a href="https://groups.google.com/group/django-developers" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/CAK8PqJELsQdpKEcg-a_iEKKwoYkoGmc8pikFM-hgG4g0CZE8XQ%40mail.gmail.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/CAK8PqJELsQdpKEcg-a_iEKKwoYkoGmc8pikFM-hgG4g0CZE8XQ%40mail.gmail.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/CAK8PqJELsQdpKEcg-a_iEKKwoYkoGmc8pikFM-hgG4g0CZE8XQ%40mail.gmail.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/django-developers/CAK8PqJELsQdpKEcg-a_iEKKwoYkoGmc8pikFM-hgG4g0CZE8XQ%40mail.gmail.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit <a href="https://groups.google.com/d/topic/django-developers/imBJwRrtJkk/unsubscribe" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/topic/django-developers/imBJwRrtJkk/unsubscribe&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/topic/django-developers/imBJwRrtJkk/unsubscribe&#39;;return true;">https://groups.google.com/d/topic/django-developers/imBJwRrtJkk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="bm4_0-xhBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-develop...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="bm4_0-xhBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-d...@googlegroups.com.
Visit this group at <a href="https://groups.google.com/group/django-developers" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/CAFNZOJNmxUNXRpHNdiFiwqbPQXe%3D8kcKY6ca7HADC4g%3DGtW-Vw%40mail.gmail.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/CAFNZOJNmxUNXRpHNdiFiwqbPQXe%3D8kcKY6ca7HADC4g%3DGtW-Vw%40mail.gmail.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/CAFNZOJNmxUNXRpHNdiFiwqbPQXe%3D8kcKY6ca7HADC4g%3DGtW-Vw%40mail.gmail.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/django-developers/CAFNZOJNmxUNXRpHNdiFiwqbPQXe%3D8kcKY6ca7HADC4g%3DGtW-Vw%40mail.gmail.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/895bb01e-9f0d-4fa2-8aac-c7b172b61e97%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Default to BigAutoField

Melvyn Sopacua
In reply to this post by Kenneth Reitz-4

On Friday 09 June 2017 15:59:50 Kenneth Reitz wrote:

> However, it should also be noted that those same larger applications

> are the ones that are likely to run into this problem eventually, so

> perhaps forcing the migration is the best path moving forward.

 

 

Existing models are the problem. Then again the database knows the truth. So with a little inspection during apps.get_models we might be able to do the right thing and even allow migrating in steps.

 

Apps is also the place to mark an app as migrated.

 

In fact - couldn't an AppConfig grow a method "get_autoid_type()" and inject the right one?

 

You asked fr thoughts, so there's my 2c stream.

--

Melvyn Sopacua

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/8118805.cf4lglgPtt%40devstation.
For more options, visit https://groups.google.com/d/optout.
12