[Django] #30887: UnicodeEncodeError problem while saving binary to database

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[Django] #30887: UnicodeEncodeError problem while saving binary to database

Django
#30887: UnicodeEncodeError problem while saving binary to database
-------------------------------------+-------------------------------------
               Reporter:  Louis      |          Owner:  nobody
                   Type:  Bug        |         Status:  new
              Component:  Database   |        Version:  2.0
  layer (models, ORM)                |
               Severity:  Normal     |       Keywords:
           Triage Stage:             |      Has patch:  0
  Unreviewed                         |
    Needs documentation:  0          |    Needs tests:  0
Patch needs improvement:  0          |  Easy pickings:  0
                  UI/UX:  0          |
-------------------------------------+-------------------------------------
 {{{
 from django.db import models

 class JobContext(models.Model):
     data = models.BinaryField()

     class Meta:
         verbose_name = 'JobContext'

 JobContext.objects.create(data=b'\x80\x03X\x04\x00\x00\x00xxxxq\x00.')
 }}}

 ----
 raise exception below:
 Traceback (most recent call last):
   File "/usr/local/lib/python3.7/logging/__init__.py", line 1028, in emit
     stream.write(msg + self.terminator)
 UnicodeEncodeError: 'utf-8' codec can't encode character '\udc80' in
 position 173: surrogates not allowed

 due to File "/usr/local/lib/python3.7/site-
 packages/django/db/backends/utils.py", line 111, in execute

 {{{
 logger.debug(
                 '(%.3f) %s; args=%s', duration, sql, params,
                 extra={'duration': duration, 'sql': sql, 'params': params}
             )
 }}}

--
Ticket URL: <https://code.djangoproject.com/ticket/30887>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/056.f8f95ac60eaea9aa2c81a081eb50076d%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30887: UnicodeEncodeError problem while saving binary to database

Django
#30887: UnicodeEncodeError problem while saving binary to database
-------------------------------------+-------------------------------------
     Reporter:  Louis                |                    Owner:  nobody
         Type:  Bug                  |                   Status:  closed
    Component:  Database layer       |                  Version:  2.0
  (models, ORM)                      |               Resolution:
     Severity:  Normal               |  worksforme
     Keywords:                       |             Triage Stage:
                                     |  Unreviewed
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Carlton Gibson):

 * status:  new => closed
 * resolution:   => worksforme


Comment:

 Hmmm. This seems to be working for me on `master`.

 Creating a model as described works as expected.

 {{{
 In [1]: from app.models import JobContext

 In [2]:
 JobContext.objects.create(data=b'\x80\x03X\x04\x00\x00\x00xxxxq\x00.')
 2019-10-16 17:06:11,151 - DEBUG - django.db.backends - (0.002) INSERT INTO
 "app_jobcontext" ("data") VALUES (X'800358040000007878787871002E');
 args=[<memory at 0x103da5888>] [utils.py:130]
 2019-10-16 17:06:11,151 - DEBUG - django.db.backends - (0.002) INSERT INTO
 "app_jobcontext" ("data") VALUES (X'800358040000007878787871002E');
 args=[<memory at 0x103da5888>] [utils.py:130]
 Out[2]: <JobContext: JobContext object (1)>
 }}}

 In particular the error isn't thrown at the `logger.debug()` call.

 What are your Python/Django versions? (I see you tagged 2.0)

 Are you able to provide a sample project that reproduces?

 Thanks.

--
Ticket URL: <https://code.djangoproject.com/ticket/30887#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/071.b170517ea4777c2edad0e43d65beee88%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30887: UnicodeEncodeError problem while saving binary to database

Django
In reply to this post by Django
#30887: UnicodeEncodeError problem while saving binary to database
-------------------------------------+-------------------------------------
     Reporter:  Louis                |                    Owner:  nobody
         Type:  Bug                  |                   Status:  closed
    Component:  Database layer       |                  Version:  2.0
  (models, ORM)                      |               Resolution:
     Severity:  Normal               |  worksforme
     Keywords:                       |             Triage Stage:
                                     |  Unreviewed
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Simon Charette):

 It would also be useful to know which database backend you were using
 because `params` might be massaged to different types based on your
 database backend.

--
Ticket URL: <https://code.djangoproject.com/ticket/30887#comment:2>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/071.5caa970d96ab6674ccf6acdafda653b4%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30887: UnicodeEncodeError problem while saving binary to database

Django
In reply to this post by Django
#30887: UnicodeEncodeError problem while saving binary to database
-------------------------------------+-------------------------------------
     Reporter:  Louis                |                    Owner:  nobody
         Type:  Bug                  |                   Status:  closed
    Component:  Database layer       |                  Version:  2.0
  (models, ORM)                      |               Resolution:
     Severity:  Normal               |  worksforme
     Keywords:                       |             Triage Stage:
                                     |  Unreviewed
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Louis):

 Replying to [comment:1 Carlton Gibson]:
 > Hmmm. This seems to be working for me on `master`.
 >
 > Creating a model as described works as expected.
 >
 > {{{
 > In [1]: from app.models import JobContext
 >
 > In [2]:
 JobContext.objects.create(data=b'\x80\x03X\x04\x00\x00\x00xxxxq\x00.')
 > 2019-10-16 17:06:11,151 - DEBUG - django.db.backends - (0.002) INSERT
 INTO "app_jobcontext" ("data") VALUES (X'800358040000007878787871002E');
 args=[<memory at 0x103da5888>] [utils.py:130]
 > 2019-10-16 17:06:11,151 - DEBUG - django.db.backends - (0.002) INSERT
 INTO "app_jobcontext" ("data") VALUES (X'800358040000007878787871002E');
 args=[<memory at 0x103da5888>] [utils.py:130]
 > Out[2]: <JobContext: JobContext object (1)>
 > }}}
 >
 > In particular the error isn't thrown at the `logger.debug()` call.
 >
 >> What are your Python/Django versions? (I see you tagged 2.0)
 > Python 3.7.4 with Django 2.1.3
 >> Are you able to provide a sample project that reproduces?
 > To be updated here...
 > Thanks.

--
Ticket URL: <https://code.djangoproject.com/ticket/30887#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/071.7bb13c0f1e81e72f83b4cd8c9b569996%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30887: UnicodeEncodeError problem while saving binary to database

Django
In reply to this post by Django
#30887: UnicodeEncodeError problem while saving binary to database
-------------------------------------+-------------------------------------
     Reporter:  Louis                |                    Owner:  nobody
         Type:  Bug                  |                   Status:  closed
    Component:  Database layer       |                  Version:  2.1
  (models, ORM)                      |               Resolution:
     Severity:  Normal               |  worksforme
     Keywords:                       |             Triage Stage:
                                     |  Unreviewed
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Louis):

 * version:  2.0 => 2.1


--
Ticket URL: <https://code.djangoproject.com/ticket/30887#comment:4>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/071.2d68664fcd8ee2558f7f3cc9a6bad4a1%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30887: UnicodeEncodeError problem while saving binary to database

Django
In reply to this post by Django
#30887: UnicodeEncodeError problem while saving binary to database
-------------------------------------+-------------------------------------
     Reporter:  Louis                |                    Owner:  nobody
         Type:  Bug                  |                   Status:  new
    Component:  Database layer       |                  Version:  2.1
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:
                                     |  Unreviewed
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Carlton Gibson):

 * status:  closed => new
 * resolution:  worksforme =>


Comment:

 OK, thanks for the follow-up. Let me reopen to investigate.

--
Ticket URL: <https://code.djangoproject.com/ticket/30887#comment:5>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/071.8950ade8184c746ce516b4bd44f4d3bf%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30887: UnicodeEncodeError logging error with saving binary with PyMySQL driver. (was: UnicodeEncodeError problem while saving binary to database)

Django
In reply to this post by Django
#30887: UnicodeEncodeError logging error with saving binary with PyMySQL driver.
-------------------------------------+-------------------------------------
     Reporter:  Louis                |                    Owner:  nobody
         Type:  Bug                  |                   Status:  closed
    Component:  Database layer       |                  Version:  2.1
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:  invalid
     Keywords:                       |             Triage Stage:
                                     |  Unreviewed
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Carlton Gibson):

 * status:  new => closed
 * resolution:   => invalid


Comment:

 Thanks for the docker image. Very helpful!

 I think this is a bug with PyMySQL. I suggest moving to `mysqlclient`,
 [https://docs.djangoproject.com/en/2.2/ref/databases/#mysql-db-api-drivers
 which is the recommended driver now].

 I'm able to reproduce the bug with your image but, changing the driver
 (and removing the couple of lines from `djangotest/__init__.py`) resolves
 the issue:

 {{{
 root@11f9f5ba1dfb:/usr/src/app# pip freeze
 Django==2.1.3
 PyMySQL==0.9.3
 pytz==2019.3
 root@11f9f5ba1dfb:/usr/src/app# pip uninstall PyMySQL
 Uninstalling PyMySQL-0.9.3:
   Would remove:
     /usr/local/lib/python3.7/site-packages/PyMySQL-0.9.3.dist-info/*
     /usr/local/lib/python3.7/site-packages/pymysql/*
 Proceed (y/n)? y
   Successfully uninstalled PyMySQL-0.9.3
 root@11f9f5ba1dfb:/usr/src/app# pip install mysqlclient
 Collecting mysqlclient
   Downloading
 https://files.pythonhosted.org/packages/4d/38/c5f8bac9c50f3042c8f05615f84206f77f03db79781db841898fde1bb284/mysqlclient-1.4.4.tar.gz
 (86kB)
      |████████████████████████████████| 92kB 824kB/s
 Building wheels for collected packages: mysqlclient
   Building wheel for mysqlclient (setup.py) ... done
   Stored in directory:
 /root/.cache/pip/wheels/a0/04/57/031b9b01df38999df7dc7f4ee998a98ecdbd2d781f73e3ffbf
 Successfully built mysqlclient
 Installing collected packages: mysqlclient
 Successfully installed mysqlclient-1.4.4
 WARNING: You are using pip version 19.1.1, however version 19.3 is
 available.
 You should consider upgrading via the 'pip install --upgrade pip' command.
 root@11f9f5ba1dfb:/usr/src/app# python3 manage.py makemigrations
 Migrations for 'demo':
   demo/migrations/0001_initial.py
     - Create model JobContext
 root@11f9f5ba1dfb:/usr/src/app# python3 manage.py migrate
 Operations to perform:
   Apply all migrations: admin, auth, contenttypes, demo, sessions
 Running migrations:
   No migrations to apply.
 root@11f9f5ba1dfb:/usr/src/app# python3 manage.py shell
 Python 3.7.3 (default, Jun 11 2019, 01:05:09)
 [GCC 6.3.0 20170516] on linux
 Type "help", "copyright", "credits" or "license" for more information.
 (InteractiveConsole)
 >>> from demo.models import JobContext
 >>> JobContext.objects.create(data=b'\x80\x03X\x04\x00\x00\x00xxxxq\x00.')
 <JobContext: JobContext object (2)>
 >>>
 now exiting InteractiveConsole...
 root@11f9f5ba1dfb:/usr/src/app# pip freeze
 Django==2.1.3
 mysqlclient==1.4.4
 pytz==2019.3
 }}}

 I'm going to close on that basis.

--
Ticket URL: <https://code.djangoproject.com/ticket/30887#comment:6>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/071.886468993adb3eaf8eb27417b2edacc2%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30887: UnicodeEncodeError logging error with saving binary with PyMySQL driver.

Django
In reply to this post by Django
#30887: UnicodeEncodeError logging error with saving binary with PyMySQL driver.
-------------------------------------+-------------------------------------
     Reporter:  Louis                |                    Owner:  nobody
         Type:  Bug                  |                   Status:  closed
    Component:  Database layer       |                  Version:  2.1
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:  invalid
     Keywords:                       |             Triage Stage:
                                     |  Unreviewed
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Louis):

 Replying to [comment:6 Carlton Gibson]:
 > Thanks for the docker image. Very helpful!
 >
 > I think this is a bug with PyMySQL. I suggest moving to `mysqlclient`,
 [https://docs.djangoproject.com/en/2.2/ref/databases/#mysql-db-api-drivers
 which is the recommended driver now].
 >
 > I'm able to reproduce the bug with your image but, changing the driver
 (and removing the couple of lines from `djangotest/__init__.py`) resolves
 the issue:
 >
 > {{{
 > root@11f9f5ba1dfb:/usr/src/app# pip freeze
 > Django==2.1.3
 > PyMySQL==0.9.3
 > pytz==2019.3
 > root@11f9f5ba1dfb:/usr/src/app# pip uninstall PyMySQL
 > Uninstalling PyMySQL-0.9.3:
 >   Would remove:
 >     /usr/local/lib/python3.7/site-packages/PyMySQL-0.9.3.dist-info/*
 >     /usr/local/lib/python3.7/site-packages/pymysql/*
 > Proceed (y/n)? y
 >   Successfully uninstalled PyMySQL-0.9.3
 > root@11f9f5ba1dfb:/usr/src/app# pip install mysqlclient
 > Collecting mysqlclient
 >   Downloading
 https://files.pythonhosted.org/packages/4d/38/c5f8bac9c50f3042c8f05615f84206f77f03db79781db841898fde1bb284/mysqlclient-1.4.4.tar.gz
 (86kB)
 >      |████████████████████████████████| 92kB 824kB/s
 > Building wheels for collected packages: mysqlclient
 >   Building wheel for mysqlclient (setup.py) ... done
 >   Stored in directory:
 /root/.cache/pip/wheels/a0/04/57/031b9b01df38999df7dc7f4ee998a98ecdbd2d781f73e3ffbf
 > Successfully built mysqlclient
 > Installing collected packages: mysqlclient
 > Successfully installed mysqlclient-1.4.4
 > WARNING: You are using pip version 19.1.1, however version 19.3 is
 available.
 > You should consider upgrading via the 'pip install --upgrade pip'
 command.
 > root@11f9f5ba1dfb:/usr/src/app# python3 manage.py makemigrations
 > Migrations for 'demo':
 >   demo/migrations/0001_initial.py
 >     - Create model JobContext
 > root@11f9f5ba1dfb:/usr/src/app# python3 manage.py migrate
 > Operations to perform:
 >   Apply all migrations: admin, auth, contenttypes, demo, sessions
 > Running migrations:
 >   No migrations to apply.
 > root@11f9f5ba1dfb:/usr/src/app# python3 manage.py shell
 > Python 3.7.3 (default, Jun 11 2019, 01:05:09)
 > [GCC 6.3.0 20170516] on linux
 > Type "help", "copyright", "credits" or "license" for more information.
 > (InteractiveConsole)
 > >>> from demo.models import JobContext
 > >>>
 JobContext.objects.create(data=b'\x80\x03X\x04\x00\x00\x00xxxxq\x00.')
 > <JobContext: JobContext object (2)>
 > >>>
 > now exiting InteractiveConsole...
 > root@11f9f5ba1dfb:/usr/src/app# pip freeze
 > Django==2.1.3
 > mysqlclient==1.4.4
 > pytz==2019.3
 > }}}
 >
 > I'm going to close on that basis.

 OK, it turned out to be pymysql compatible problem, thanks for the help,
 will follow your suggestion then.

--
Ticket URL: <https://code.djangoproject.com/ticket/30887#comment:7>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/071.a87ea4c21079797e4667bcf0cbcbc1b5%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30887: UnicodeEncodeError logging error with saving binary with PyMySQL driver.

Django
In reply to this post by Django
#30887: UnicodeEncodeError logging error with saving binary with PyMySQL driver.
-------------------------------------+-------------------------------------
     Reporter:  Louis                |                    Owner:  nobody
         Type:  Bug                  |                   Status:  closed
    Component:  Database layer       |                  Version:  2.1
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:  duplicate
     Keywords:                       |             Triage Stage:
                                     |  Unreviewed
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Simon Charette):

 * resolution:  invalid => duplicate


Comment:

 For the record this is tracked in #30380 and was fixed in
 a41b09266dcdd01036d59d76fe926fe0386aaade which is part of the upcoming
 Django 3.0 release.

--
Ticket URL: <https://code.djangoproject.com/ticket/30887#comment:8>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/071.3116a0afd92449e68ffea5207d81fe10%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30887: UnicodeEncodeError logging error with saving binary with PyMySQL driver.

Django
In reply to this post by Django
#30887: UnicodeEncodeError logging error with saving binary with PyMySQL driver.
-------------------------------------+-------------------------------------
     Reporter:  Louis                |                    Owner:  nobody
         Type:  Bug                  |                   Status:  closed
    Component:  Database layer       |                  Version:  2.1
  (models, ORM)                      |
     Severity:  Normal               |               Resolution:  duplicate
     Keywords:                       |             Triage Stage:
                                     |  Unreviewed
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Louis):

 Replying to [comment:8 Simon Charette]:
 >> For the record this is tracked in #30380 and was fixed in
 a41b09266dcdd01036d59d76fe926fe0386aaade which is part of the upcoming
 Django 3.0 release.
 > good to know, thanks.

--
Ticket URL: <https://code.djangoproject.com/ticket/30887#comment:9>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/071.398ec1db78646a40c0c3cb922a2e36ab%40djangoproject.com.