[Django] #29015: Make Django aware of PostgreSQL database name length limits

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Django] #29015: Make Django aware of PostgreSQL database name length limits

Django
#29015: Make Django aware of PostgreSQL database name length limits
-----------------------------------------+------------------------
               Reporter:  Tadej Janež    |          Owner:  nobody
                   Type:  Bug            |         Status:  new
              Component:  Uncategorized  |        Version:  2.0
               Severity:  Normal         |       Keywords:
           Triage Stage:  Unreviewed     |      Has patch:  0
    Needs documentation:  0              |    Needs tests:  0
Patch needs improvement:  0              |  Easy pickings:  0
                  UI/UX:  0              |
-----------------------------------------+------------------------
 PostgreSQL has a maximum identifier lenght limit that is set to 63 bytes
 by default (see https://www.postgresql.org/docs/current/static/runtime-
 config-preset.html for more details).

 If a user chooses a database name longer than this, PostgreSQL will
 truncate the identifier to 63 bytes and operate normally.
 Django apparently doesn't notice this and behaves as if the database name
 is not truncated.

 However, in some cases, Django needs to be aware of this. For example,
 when running tests in parallel. Django will fail due to a pre-existing
 database with the same name and not finding the template database to
 clone.

 Here is an example using https://github.com/mdamien/django-tutorial.

 First setup up a virtual environment and clone the project:

 {{{
 mkvirtualenv --python=python3 django-db-name-too-long-bug
 pip install Django psycopg2
 git clone https://github.com/mdamien/django-tutorial
 cd django-tutorial
 }}}

 Edit `mysite/settings.py` and change database configuration to:

 {{{
 DATABASES = {
     'default': {
         'ENGINE': 'django.db.backends.postgresql',
         'NAME': 'foo',
      }
 }
 }}}

 Confirm that the tests run without an error:

 {{{
 $ ./manage.py test -v 2
 Creating test database for alias 'default' ('test_foo')...
 Operations to perform:
   Synchronize unmigrated apps: messages, staticfiles
   Apply all migrations: admin, auth, contenttypes, polls, sessions
 Synchronizing apps without migrations:
   Creating tables...
     Running deferred SQL...
 Running migrations:
   Applying contenttypes.0001_initial... OK
   Applying auth.0001_initial... OK
   Applying admin.0001_initial... OK
   Applying admin.0002_logentry_remove_auto_add... OK
   Applying contenttypes.0002_remove_content_type_name... OK
   Applying auth.0002_alter_permission_name_max_length... OK
   Applying auth.0003_alter_user_email_max_length... OK
   Applying auth.0004_alter_user_username_opts... OK
   Applying auth.0005_alter_user_last_login_null... OK
   Applying auth.0006_require_contenttypes_0002... OK
   Applying auth.0007_alter_validators_add_error_messages... OK
   Applying auth.0008_alter_user_username_max_length... OK
   Applying auth.0009_alter_user_last_name_max_length... OK
   Applying polls.0001_initial... OK
   Applying sessions.0001_initial... OK
 System check identified no issues (0 silenced).
 test_detail_view_with_a_future_question
 (polls.tests.QuestionIndexDetailTests) ... ok
 test_detail_view_with_a_past_question
 (polls.tests.QuestionIndexDetailTests) ... ok
 test_was_published_recently_with_future_question
 (polls.tests.QuestionMethodTests) ... ok
 test_was_published_recently_with_old_question
 (polls.tests.QuestionMethodTests) ... ok
 test_was_published_recently_with_recent_question
 (polls.tests.QuestionMethodTests) ... ok
 test_index_view_with_a_future_question (polls.tests.QuestionViewTests) ...
 ok
 test_index_view_with_a_past_question (polls.tests.QuestionViewTests) ...
 ok
 test_index_view_with_future_question_and_past_question
 (polls.tests.QuestionViewTests) ... ok
 test_index_view_with_no_questions (polls.tests.QuestionViewTests) ... ok
 test_index_view_with_two_past_questions (polls.tests.QuestionViewTests)
 ... ok

 ----------------------------------------------------------------------
 Ran 10 tests in 0.054s

 OK
 Destroying test database for alias 'default' ('test_foo')...
 }}}

 Confirm that the tests when run in parallel run without an error:

 {{{
 $ ./manage.py test -v 2 --parallel
 Creating test database for alias 'default' ('test_foo')...
 Operations to perform:
   Synchronize unmigrated apps: messages, staticfiles
   Apply all migrations: admin, auth, contenttypes, polls, sessions
 Synchronizing apps without migrations:
   Creating tables...
     Running deferred SQL...
 Running migrations:
   Applying contenttypes.0001_initial... OK
   Applying auth.0001_initial... OK
   Applying admin.0001_initial... OK
   Applying admin.0002_logentry_remove_auto_add... OK
   Applying contenttypes.0002_remove_content_type_name... OK
   Applying auth.0002_alter_permission_name_max_length... OK
   Applying auth.0003_alter_user_email_max_length... OK
   Applying auth.0004_alter_user_username_opts... OK
   Applying auth.0005_alter_user_last_login_null... OK
   Applying auth.0006_require_contenttypes_0002... OK
   Applying auth.0007_alter_validators_add_error_messages... OK
   Applying auth.0008_alter_user_username_max_length... OK
   Applying auth.0009_alter_user_last_name_max_length... OK
   Applying polls.0001_initial... OK
   Applying sessions.0001_initial... OK
 Cloning test database for alias 'default' ('test_foo')...
 Cloning test database for alias 'default' ('test_foo')...
 Cloning test database for alias 'default' ('test_foo')...
 System check identified no issues (0 silenced).
 test_was_published_recently_with_future_question
 (polls.tests.QuestionMethodTests) ... ok
 test_was_published_recently_with_old_question
 (polls.tests.QuestionMethodTests) ... ok
 test_was_published_recently_with_recent_question
 (polls.tests.QuestionMethodTests) ... ok
 test_detail_view_with_a_future_question
 (polls.tests.QuestionIndexDetailTests) ... ok
 test_detail_view_with_a_past_question
 (polls.tests.QuestionIndexDetailTests) ... ok
 test_index_view_with_a_future_question (polls.tests.QuestionViewTests) ...
 ok
 test_index_view_with_a_past_question (polls.tests.QuestionViewTests) ...
 ok
 test_index_view_with_future_question_and_past_question
 (polls.tests.QuestionViewTests) ... ok
 test_index_view_with_no_questions (polls.tests.QuestionViewTests) ... ok
 test_index_view_with_two_past_questions (polls.tests.QuestionViewTests)
 ... ok

 ----------------------------------------------------------------------
 Ran 10 tests in 0.119s

 OK
 Destroying test database for alias 'default' ('test_foo_1')...
 Destroying test database for alias 'default' ('test_foo_2')...
 Destroying test database for alias 'default' ('test_foo_3')...
 Destroying test database for alias 'default' ('test_foo')...
 }}}

 Now edit `mysite/settings.py` again and change database name to something
 very long. For example, set the following:

 {{{
 DATABASES = {
     'default': {
         'ENGINE': 'django.db.backends.postgresql',
         'NAME': 'something-very-very-
 veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyyyyy'
                 '-long-that-will-get-truncated',
     }
 }
 }}}

 Running the tests without `--parallel` works fine, even though Django
 mistakenly thinks it created a test database with such a long name:

 {{{
 $ ./manage.py test -v 2
 Creating test database for alias 'default' ('test_something-very-very-
 veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyyyyy-long-that-will-get-
 truncated')...
 Operations to perform:
   Synchronize unmigrated apps: messages, staticfiles
   Apply all migrations: admin, auth, contenttypes, polls, sessions
 Synchronizing apps without migrations:
   Creating tables...
     Running deferred SQL...
 Running migrations:
   Applying contenttypes.0001_initial... OK
   Applying auth.0001_initial... OK
   Applying admin.0001_initial... OK
   Applying admin.0002_logentry_remove_auto_add... OK
   Applying contenttypes.0002_remove_content_type_name... OK
   Applying auth.0002_alter_permission_name_max_length... OK
   Applying auth.0003_alter_user_email_max_length... OK
   Applying auth.0004_alter_user_username_opts... OK
   Applying auth.0005_alter_user_last_login_null... OK
   Applying auth.0006_require_contenttypes_0002... OK
   Applying auth.0007_alter_validators_add_error_messages... OK
   Applying auth.0008_alter_user_username_max_length... OK
   Applying auth.0009_alter_user_last_name_max_length... OK
   Applying polls.0001_initial... OK
   Applying sessions.0001_initial... OK
 System check identified no issues (0 silenced).
 test_detail_view_with_a_future_question
 (polls.tests.QuestionIndexDetailTests) ... ok
 test_detail_view_with_a_past_question
 (polls.tests.QuestionIndexDetailTests) ... ok
 test_was_published_recently_with_future_question
 (polls.tests.QuestionMethodTests) ... ok
 test_was_published_recently_with_old_question
 (polls.tests.QuestionMethodTests) ... ok
 test_was_published_recently_with_recent_question
 (polls.tests.QuestionMethodTests) ... ok
 test_index_view_with_a_future_question (polls.tests.QuestionViewTests) ...
 ok
 test_index_view_with_a_past_question (polls.tests.QuestionViewTests) ...
 ok
 test_index_view_with_future_question_and_past_question
 (polls.tests.QuestionViewTests) ... ok
 test_index_view_with_no_questions (polls.tests.QuestionViewTests) ... ok
 test_index_view_with_two_past_questions (polls.tests.QuestionViewTests)
 ... ok

 ----------------------------------------------------------------------
 Ran 10 tests in 0.051s

 OK
 Destroying test database for alias 'default' ('test_something-very-very-
 veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyyyyy-long-that-will-get-
 truncated')...
 }}}

 However, running the tests in parallel exposes Django's lack of
 understanding of identifier truncation in PostgreSQL:
 {{{
 $ ./manage.py test -v 2 --parallel
 Creating test database for alias 'default' ('test_something-very-very-
 veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyyyyy-long-that-will-get-
 truncated')...
 Operations to perform:
   Synchronize unmigrated apps: messages, staticfiles
   Apply all migrations: admin, auth, contenttypes, polls, sessions
 Synchronizing apps without migrations:
   Creating tables...
     Running deferred SQL...
 Running migrations:
   Applying contenttypes.0001_initial... OK
   Applying auth.0001_initial... OK
   Applying admin.0001_initial... OK
   Applying admin.0002_logentry_remove_auto_add... OK
   Applying contenttypes.0002_remove_content_type_name... OK
   Applying auth.0002_alter_permission_name_max_length... OK
   Applying auth.0003_alter_user_email_max_length... OK
   Applying auth.0004_alter_user_username_opts... OK
   Applying auth.0005_alter_user_last_login_null... OK
   Applying auth.0006_require_contenttypes_0002... OK
   Applying auth.0007_alter_validators_add_error_messages... OK
   Applying auth.0008_alter_user_username_max_length... OK
   Applying auth.0009_alter_user_last_name_max_length... OK
   Applying polls.0001_initial... OK
   Applying sessions.0001_initial... OK
 Cloning test database for alias 'default' ('test_something-very-very-
 veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyyyyy-long-that-will-get-
 truncated')...
 Destroying old test database for alias 'default' ('test_something-very-
 very-veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyyyyy-long-that-will-get-
 truncated_1')...
 Got an error creating the test database: template database
 "test_something-very-very-veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyy" does not
 exist
 }}}

 When Django created the main test database, `"test_something-very-very-
 veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyyyyy-long-that-will-get-truncated"`,
 in reality, PostgreSQL created a database named `"test_something-very-
 very-veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyy"`.

 And when Django tried to create a clone of this test database, it
 executed:

 {{{
 CREATE DATABASE "test_something-very-very-
 veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyyyyy-long-that-will-get-truncated_1"
 WITH TEMPLATE "test_something-very-very-
 veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyyyyy-long-that-will-get-truncated"
 }}}

 Of course, this doesn't work since PostgreSQL truncated the identifiers
 and a database named `"test_something-very-very-
 veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyy"` already exists.

 PostgreSQL also returns the second error since the template database
 `"test_something-very-very-veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyy"` does
 not exist.

 I want to note that at first sight it might appear that users won't
 encounter this in real life at all, but this is not true.
 If, for example, a user sets up the CI system to use a unique database
 name that consists of:

 {{{
 ${PROJECT_OWNER}-${PROJECT_NAME}-${BRANCH_NAME}-${BUILD_NUMBER}
 }}}

 this can quickly get larger than 63 bytes (i.e. characters) and the tests
 will fail with the above uninformative error.

--
Ticket URL: <https://code.djangoproject.com/ticket/29015>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/049.d04d8b247bd34b9a0a2b282c8973cec8%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #29015: Add an error when the PostgreSQL database name length limit is exceeded (was: Make Django aware of PostgreSQL database name length limits)

Django
#29015: Add an error when the PostgreSQL database name length limit is exceeded
-------------------------------------+-------------------------------------
     Reporter:  Tadej Janež          |                    Owner:  nobody
         Type:                       |                   Status:  new
  Cleanup/optimization               |
    Component:  Database layer       |                  Version:  2.0
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Tim Graham):

 * type:  Bug => Cleanup/optimization
 * stage:  Unreviewed => Accepted
 * component:  Uncategorized => Database layer (models, ORM)


--
Ticket URL: <https://code.djangoproject.com/ticket/29015#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/064.db4738f326ba05d8a7c085b94edb826b%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #29015: Add an error when the PostgreSQL database name length limit is exceeded

Django
In reply to this post by Django
#29015: Add an error when the PostgreSQL database name length limit is exceeded
-------------------------------------+-------------------------------------
     Reporter:  Tadej Janež          |                    Owner:  Priyansh
         Type:                       |  Saxena
  Cleanup/optimization               |                   Status:  assigned
    Component:  Database layer       |                  Version:  2.0
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Priyansh Saxena):

 * owner:  nobody => Priyansh Saxena
 * status:  new => assigned


--
Ticket URL: <https://code.djangoproject.com/ticket/29015#comment:2>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/064.8693d50e2de35b836353a2363fea9469%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #29015: Add an error when the PostgreSQL database name length limit is exceeded

Django
In reply to this post by Django
#29015: Add an error when the PostgreSQL database name length limit is exceeded
-------------------------------------+-------------------------------------
     Reporter:  Tadej Janež          |                    Owner:  Priyansh
         Type:                       |  Saxena
  Cleanup/optimization               |                   Status:  assigned
    Component:  Database layer       |                  Version:  2.0
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Adam (Chainz) Johnson):

 * cc: Adam (Chainz) Johnson (added)


--
Ticket URL: <https://code.djangoproject.com/ticket/29015#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/064.aeedcd23150ef81ab8b14fdfab8e873d%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #29015: Add an error when the PostgreSQL database name length limit is exceeded

Django
In reply to this post by Django
#29015: Add an error when the PostgreSQL database name length limit is exceeded
-------------------------------------+-------------------------------------
     Reporter:  Tadej Janež          |                    Owner:  Priyansh
         Type:                       |  Saxena
  Cleanup/optimization               |                   Status:  assigned
    Component:  Database layer       |                  Version:  2.0
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  1                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Priyansh Saxena):

 * has_patch:  0 => 1


Comment:

 [https://github.com/django/django/pull/9600]

--
Ticket URL: <https://code.djangoproject.com/ticket/29015#comment:4>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/064.349173add3273478120fa799c03f15db%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.