Django 3.1.2 JSONField handling error with an Postgres Foreign Data Wrapper-based model

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Django 3.1.2 JSONField handling error with an Postgres Foreign Data Wrapper-based model

Shaheed Haque-2

Hi,

I have a Django model working fine under Django 3.0 (i.e. "Django<3.1") which looks like this:

===================
    class Job(models.Model):
        id = models.CharField(max_length=36, primary_key=True)
        queue = models.CharField(max_length=40)
        args = JSONField()
        kwargs = JSONField()
        type = models.CharField(max_length=80)
        ...

        class Meta:
            managed = False  # <------   The table is implemented as a Postgres FDW wrapper.
            db_table = 'jobs'
===================

I am testing the update to Django 3.1.2 and hit an error in executing this line:

    jobs = list(models.Job.objects.filter(queue='celery', state='scheduled'))

The error is as follows from pytest (i.e. stack trace with local variables too):

==================
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:287: in __iter__
   self._fetch_all()
       self       = <QuerySet []>
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:1308: in _fetch_all
   self._result_cache = list(self._iterable_class(self))
       self       = <QuerySet []>
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:70: in __iter__
   for row in compiler.results_iter(results):
       annotation_col_map = {}
       compiler   = <django.db.models.sql.compiler.SQLCompiler object at 0x7f8685e49160>
       db         = 'fdw'
       init_list  = ['id', 'queue', 'args', 'kwargs', 'type', 'state', ...]
       klass_info = {'model': <class 'paiyroll.models.batch.Job'>, 'select_fields': [0, 1, 2, 3, 4, 5, ...]}
       known_related_objects = []
       model_cls  = <class 'paiyroll.models.batch.Job'>
       model_fields_end = 9
       model_fields_start = 0
       queryset   = <QuerySet []>
       related_populators = []
       results    = [[('8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery', ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {}, 'paiyroll.tasks.function_run', 'scheduled', ...)]]
       select     = [(Col(jobs, paiyroll.Job.id), ('"jobs"."id"', []), None), (Col(jobs, paiyroll.Job.queue), ('"jobs"."queue"', []), None..., paiyroll.Job.type), ('"jobs"."type"', []), None), (Col(jobs, paiyroll.Job.state), ('"jobs"."state"', []), None), ...]
       select_fields = [0, 1, 2, 3, 4, 5, ...]
       self       = <django.db.models.query.ModelIterable object at 0x7f86836f3040>
/usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py:1100: in apply_converters
   value = converter(value, expression, connection)
       connection = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f869a321670>
       converter  = <bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>
       converters = [(2, ([<bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>], Col(jobs, pa...NField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: details>>], Col(jobs, paiyroll.Job.details)))]
       convs      = [<bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>]
       expression = Col(jobs, paiyroll.Job.args)
       pos        = 2
       row        = ['8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery', ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {}, 'paiyroll.tasks.function_run', 'scheduled', ...]
       rows       = <itertools.chain object at 0x7f8683ae7520>
       self       = <django.db.models.sql.compiler.SQLCompiler object at 0x7f8685e49160>
       value      = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]
/usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74: in from_db_value
   return json.loads(value, cls=self.decoder)
       connection = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f869a321670>
       expression = Col(jobs, paiyroll.Job.args)
       self       = <django.contrib.postgres.fields.jsonb.JSONField: args>
       value      = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

s = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], cls = None
object_hook = None, parse_float = None, parse_int = None, parse_constant = None
object_pairs_hook = None, kw = {}

   def loads(s, *, cls=None, object_hook=None, parse_float=None,
           parse_int=None, parse_constant=None, object_pairs_hook=None, **kw):
       """Deserialize ``s`` (a ``str``, ``bytes`` or ``bytearray`` instance
       containing a JSON document) to a Python object.

       ``object_hook`` is an optional function that will be called with the
       result of any object literal decode (a ``dict``). The return value of
       ``object_hook`` will be used instead of the ``dict``. This feature
       can be used to implement custom decoders (e.g. JSON-RPC class hinting).

       ``object_pairs_hook`` is an optional function that will be called with the
       result of any object literal decoded with an ordered list of pairs.  The
       return value of ``object_pairs_hook`` will be used instead of the ``dict``.
       This feature can be used to implement custom decoders.  If ``object_hook``
       is also defined, the ``object_pairs_hook`` takes priority.

       ``parse_float``, if specified, will be called with the string
       of every JSON float to be decoded. By default this is equivalent to
       float(num_str). This can be used to use another datatype or parser
       for JSON floats (e.g. decimal.Decimal).

       ``parse_int``, if specified, will be called with the string
       of every JSON int to be decoded. By default this is equivalent to
       int(num_str). This can be used to use another datatype or parser
       for JSON integers (e.g. float).

       ``parse_constant``, if specified, will be called with one of the
       following strings: -Infinity, Infinity, NaN.
       This can be used to raise an exception if invalid JSON numbers
       are encountered.

       To use a custom ``JSONDecoder`` subclass, specify it with the ``cls``
       kwarg; otherwise ``JSONDecoder`` is used.

       The ``encoding`` argument is ignored and deprecated since Python 3.1.
       """
       if isinstance(s, str):
           if s.startswith('\ufeff'):
               raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)",
                                     s, 0)
       else:
           if not isinstance(s, (bytes, bytearray)):
>               raise TypeError(f'the JSON object must be str, bytes or bytearray, '
                               f'not {s.__class__.__name__}')
E               TypeError: the JSON object must be str, bytes or bytearray, not list

cls        = None
kw         = {}
object_hook = None
object_pairs_hook = None
parse_constant = None
parse_float = None
parse_int  = None
s          = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]
===============================

As you can perhaps see from the bolded part (for /usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74), the value being written into the JSONField called "args" is a Python list, (shown as "s" on the last line of the traceback-with-values). I am aware of documented changes around serializers but did not think that affected me; however, given that I am using an FDW to return the data, is it that I should now be serialising returned data into a string?

Any pointers appreciated.

Thanks, Shaheed

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAHAc2jcTVRUg1Q9vVdNL6xpnZpBMCPZu8BoQKcacfEwYDFN6BA%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Django 3.1.2 JSONField handling error with an Postgres Foreign Data Wrapper-based model

Jason Johns
this is pretty interesting, and may be something to report to the devs at https://groups.google.com/g/django-developers, or open up a ticket at https://code.djangoproject.com/  I don't see any tickets with the query jsonfield foreign data that are similar with your experience, so you may have found an edge case.

On Thursday, October 15, 2020 at 5:31:09 AM UTC-4 [hidden email] wrote:

Hi,

I have a Django model working fine under Django 3.0 (i.e. "Django<3.1") which looks like this:

===================
    class Job(models.Model):
        id = models.CharField(max_length=36, primary_key=True)
        queue = models.CharField(max_length=40)
        args = JSONField()
        kwargs = JSONField()
        type = models.CharField(max_length=80)
        ...

        class Meta:
            managed = False  # <------   The table is implemented as a Postgres FDW wrapper.
            db_table = 'jobs'
===================

I am testing the update to Django 3.1.2 and hit an error in executing this line:

    jobs = list(models.Job.objects.filter(queue='celery', state='scheduled'))

The error is as follows from pytest (i.e. stack trace with local variables too):

==================
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:287: in __iter__
   self._fetch_all()
       self       = <QuerySet []>
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:1308: in _fetch_all
   self._result_cache = list(self._iterable_class(self))
       self       = <QuerySet []>
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:70: in __iter__
   for row in compiler.results_iter(results):
       annotation_col_map = {}
       compiler   = <django.db.models.sql.compiler.SQLCompiler object at 0x7f8685e49160>
       db         = 'fdw'
       init_list  = ['id', 'queue', 'args', 'kwargs', 'type', 'state', ...]
       klass_info = {'model': <class 'paiyroll.models.batch.Job'>, 'select_fields': [0, 1, 2, 3, 4, 5, ...]}
       known_related_objects = []
       model_cls  = <class 'paiyroll.models.batch.Job'>
       model_fields_end = 9
       model_fields_start = 0
       queryset   = <QuerySet []>
       related_populators = []
       results    = [[('8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery', ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {}, 'paiyroll.tasks.function_run', 'scheduled', ...)]]
       select     = [(Col(jobs, paiyroll.Job.id), ('"jobs"."id"', []), None), (Col(jobs, paiyroll.Job.queue), ('"jobs"."queue"', []), None..., paiyroll.Job.type), ('"jobs"."type"', []), None), (Col(jobs, paiyroll.Job.state), ('"jobs"."state"', []), None), ...]
       select_fields = [0, 1, 2, 3, 4, 5, ...]
       self       = <django.db.models.query.ModelIterable object at 0x7f86836f3040>
/usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py:1100: in apply_converters
   value = converter(value, expression, connection)
       connection = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f869a321670>
       converter  = <bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>
       converters = [(2, ([<bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>], Col(jobs, pa...NField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: details>>], Col(jobs, paiyroll.Job.details)))]
       convs      = [<bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>]
       expression = Col(jobs, paiyroll.Job.args)
       pos        = 2
       row        = ['8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery', ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {}, 'paiyroll.tasks.function_run', 'scheduled', ...]
       rows       = <itertools.chain object at 0x7f8683ae7520>
       self       = <django.db.models.sql.compiler.SQLCompiler object at 0x7f8685e49160>
       value      = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]
/usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74: in from_db_value
   return json.loads(value, cls=self.decoder)
       connection = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f869a321670>
       expression = Col(jobs, paiyroll.Job.args)
       self       = <django.contrib.postgres.fields.jsonb.JSONField: args>
       value      = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

s = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], cls = None
object_hook = None, parse_float = None, parse_int = None, parse_constant = None
object_pairs_hook = None, kw = {}

   def loads(s, *, cls=None, object_hook=None, parse_float=None,
           parse_int=None, parse_constant=None, object_pairs_hook=None, **kw):
       """Deserialize ``s`` (a ``str``, ``bytes`` or ``bytearray`` instance
       containing a JSON document) to a Python object.

       ``object_hook`` is an optional function that will be called with the
       result of any object literal decode (a ``dict``). The return value of
       ``object_hook`` will be used instead of the ``dict``. This feature
       can be used to implement custom decoders (e.g. JSON-RPC class hinting).

       ``object_pairs_hook`` is an optional function that will be called with the
       result of any object literal decoded with an ordered list of pairs.  The
       return value of ``object_pairs_hook`` will be used instead of the ``dict``.
       This feature can be used to implement custom decoders.  If ``object_hook``
       is also defined, the ``object_pairs_hook`` takes priority.

       ``parse_float``, if specified, will be called with the string
       of every JSON float to be decoded. By default this is equivalent to
       float(num_str). This can be used to use another datatype or parser
       for JSON floats (e.g. decimal.Decimal).

       ``parse_int``, if specified, will be called with the string
       of every JSON int to be decoded. By default this is equivalent to
       int(num_str). This can be used to use another datatype or parser
       for JSON integers (e.g. float).

       ``parse_constant``, if specified, will be called with one of the
       following strings: -Infinity, Infinity, NaN.
       This can be used to raise an exception if invalid JSON numbers
       are encountered.

       To use a custom ``JSONDecoder`` subclass, specify it with the ``cls``
       kwarg; otherwise ``JSONDecoder`` is used.

       The ``encoding`` argument is ignored and deprecated since Python 3.1.
       """
       if isinstance(s, str):
           if s.startswith('\ufeff'):
               raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)",
                                     s, 0)
       else:
           if not isinstance(s, (bytes, bytearray)):
>               raise TypeError(f'the JSON object must be str, bytes or bytearray, '
                               f'not {s.__class__.__name__}')
E               TypeError: the JSON object must be str, bytes or bytearray, not list

cls        = None
kw         = {}
object_hook = None
object_pairs_hook = None
parse_constant = None
parse_float = None
parse_int  = None
s          = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]
===============================

As you can perhaps see from the bolded part (for /usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74), the value being written into the JSONField called "args" is a Python list, (shown as "s" on the last line of the traceback-with-values). I am aware of documented changes around serializers but did not think that affected me; however, given that I am using an FDW to return the data, is it that I should now be serialising returned data into a string?

Any pointers appreciated.

Thanks, Shaheed

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/6d01a874-f810-45f7-b583-28e294d564ben%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Django 3.1.2 JSONField handling error with an Postgres Foreign Data Wrapper-based model

Shaheed Haque-2
Thanks Jason, I filed https://code.djangoproject.com/ticket/32111#ticket.

On Thursday, 15 October 2020 at 12:51:22 UTC+1 Jason wrote:
this is pretty interesting, and may be something to report to the devs at https://groups.google.com/g/django-developers, or open up a ticket at https://code.djangoproject.com/  I don't see any tickets with the query jsonfield foreign data that are similar with your experience, so you may have found an edge case.

On Thursday, October 15, 2020 at 5:31:09 AM UTC-4 [hidden email] wrote:

Hi,

I have a Django model working fine under Django 3.0 (i.e. "Django<3.1") which looks like this:

===================
    class Job(models.Model):
        id = models.CharField(max_length=36, primary_key=True)
        queue = models.CharField(max_length=40)
        args = JSONField()
        kwargs = JSONField()
        type = models.CharField(max_length=80)
        ...

        class Meta:
            managed = False  # <------   The table is implemented as a Postgres FDW wrapper.
            db_table = 'jobs'
===================

I am testing the update to Django 3.1.2 and hit an error in executing this line:

    jobs = list(models.Job.objects.filter(queue='celery', state='scheduled'))

The error is as follows from pytest (i.e. stack trace with local variables too):

==================
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:287: in __iter__
   self._fetch_all()
       self       = <QuerySet []>
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:1308: in _fetch_all
   self._result_cache = list(self._iterable_class(self))
       self       = <QuerySet []>
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:70: in __iter__
   for row in compiler.results_iter(results):
       annotation_col_map = {}
       compiler   = <django.db.models.sql.compiler.SQLCompiler object at 0x7f8685e49160>
       db         = 'fdw'
       init_list  = ['id', 'queue', 'args', 'kwargs', 'type', 'state', ...]
       klass_info = {'model': <class 'paiyroll.models.batch.Job'>, 'select_fields': [0, 1, 2, 3, 4, 5, ...]}
       known_related_objects = []
       model_cls  = <class 'paiyroll.models.batch.Job'>
       model_fields_end = 9
       model_fields_start = 0
       queryset   = <QuerySet []>
       related_populators = []
       results    = [[('8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery', ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {}, 'paiyroll.tasks.function_run', 'scheduled', ...)]]
       select     = [(Col(jobs, paiyroll.Job.id), ('"jobs"."id"', []), None), (Col(jobs, paiyroll.Job.queue), ('"jobs"."queue"', []), None..., paiyroll.Job.type), ('"jobs"."type"', []), None), (Col(jobs, paiyroll.Job.state), ('"jobs"."state"', []), None), ...]
       select_fields = [0, 1, 2, 3, 4, 5, ...]
       self       = <django.db.models.query.ModelIterable object at 0x7f86836f3040>
/usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py:1100: in apply_converters
   value = converter(value, expression, connection)
       connection = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f869a321670>
       converter  = <bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>
       converters = [(2, ([<bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>], Col(jobs, pa...NField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: details>>], Col(jobs, paiyroll.Job.details)))]
       convs      = [<bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>]
       expression = Col(jobs, paiyroll.Job.args)
       pos        = 2
       row        = ['8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery', ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {}, 'paiyroll.tasks.function_run', 'scheduled', ...]
       rows       = <itertools.chain object at 0x7f8683ae7520>
       self       = <django.db.models.sql.compiler.SQLCompiler object at 0x7f8685e49160>
       value      = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]
/usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74: in from_db_value
   return json.loads(value, cls=self.decoder)
       connection = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f869a321670>
       expression = Col(jobs, paiyroll.Job.args)
       self       = <django.contrib.postgres.fields.jsonb.JSONField: args>
       value      = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

s = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], cls = None
object_hook = None, parse_float = None, parse_int = None, parse_constant = None
object_pairs_hook = None, kw = {}

   def loads(s, *, cls=None, object_hook=None, parse_float=None,
           parse_int=None, parse_constant=None, object_pairs_hook=None, **kw):
       """Deserialize ``s`` (a ``str``, ``bytes`` or ``bytearray`` instance
       containing a JSON document) to a Python object.

       ``object_hook`` is an optional function that will be called with the
       result of any object literal decode (a ``dict``). The return value of
       ``object_hook`` will be used instead of the ``dict``. This feature
       can be used to implement custom decoders (e.g. JSON-RPC class hinting).

       ``object_pairs_hook`` is an optional function that will be called with the
       result of any object literal decoded with an ordered list of pairs.  The
       return value of ``object_pairs_hook`` will be used instead of the ``dict``.
       This feature can be used to implement custom decoders.  If ``object_hook``
       is also defined, the ``object_pairs_hook`` takes priority.

       ``parse_float``, if specified, will be called with the string
       of every JSON float to be decoded. By default this is equivalent to
       float(num_str). This can be used to use another datatype or parser
       for JSON floats (e.g. decimal.Decimal).

       ``parse_int``, if specified, will be called with the string
       of every JSON int to be decoded. By default this is equivalent to
       int(num_str). This can be used to use another datatype or parser
       for JSON integers (e.g. float).

       ``parse_constant``, if specified, will be called with one of the
       following strings: -Infinity, Infinity, NaN.
       This can be used to raise an exception if invalid JSON numbers
       are encountered.

       To use a custom ``JSONDecoder`` subclass, specify it with the ``cls``
       kwarg; otherwise ``JSONDecoder`` is used.

       The ``encoding`` argument is ignored and deprecated since Python 3.1.
       """
       if isinstance(s, str):
           if s.startswith('\ufeff'):
               raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)",
                                     s, 0)
       else:
           if not isinstance(s, (bytes, bytearray)):
>               raise TypeError(f'the JSON object must be str, bytes or bytearray, '
                               f'not {s.__class__.__name__}')
E               TypeError: the JSON object must be str, bytes or bytearray, not list

cls        = None
kw         = {}
object_hook = None
object_pairs_hook = None
parse_constant = None
parse_float = None
parse_int  = None
s          = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]
===============================

As you can perhaps see from the bolded part (for /usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74), the value being written into the JSONField called "args" is a Python list, (shown as "s" on the last line of the traceback-with-values). I am aware of documented changes around serializers but did not think that affected me; however, given that I am using an FDW to return the data, is it that I should now be serialising returned data into a string?

Any pointers appreciated.

Thanks, Shaheed

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/f1392121-70f6-4359-8d2c-10dbb157607bn%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Django 3.1.2 JSONField handling error with an Postgres Foreign Data Wrapper-based model

Jason Johns
looks like you got a response back, and there's more context in the linked ticket too.  tl;dr json is not jsonb, and django only works with the latter

On Thursday, October 15, 2020 at 10:02:30 AM UTC-4 [hidden email] wrote:
Thanks Jason, I filed https://code.djangoproject.com/ticket/32111#ticket.

On Thursday, 15 October 2020 at 12:51:22 UTC+1 Jason wrote:
this is pretty interesting, and may be something to report to the devs at https://groups.google.com/g/django-developers, or open up a ticket at https://code.djangoproject.com/  I don't see any tickets with the query jsonfield foreign data that are similar with your experience, so you may have found an edge case.

On Thursday, October 15, 2020 at 5:31:09 AM UTC-4 [hidden email] wrote:

Hi,

I have a Django model working fine under Django 3.0 (i.e. "Django<3.1") which looks like this:

===================
    class Job(models.Model):
        id = models.CharField(max_length=36, primary_key=True)
        queue = models.CharField(max_length=40)
        args = JSONField()
        kwargs = JSONField()
        type = models.CharField(max_length=80)
        ...

        class Meta:
            managed = False  # <------   The table is implemented as a Postgres FDW wrapper.
            db_table = 'jobs'
===================

I am testing the update to Django 3.1.2 and hit an error in executing this line:

    jobs = list(models.Job.objects.filter(queue='celery', state='scheduled'))

The error is as follows from pytest (i.e. stack trace with local variables too):

==================
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:287: in __iter__
   self._fetch_all()
       self       = <QuerySet []>
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:1308: in _fetch_all
   self._result_cache = list(self._iterable_class(self))
       self       = <QuerySet []>
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:70: in __iter__
   for row in compiler.results_iter(results):
       annotation_col_map = {}
       compiler   = <django.db.models.sql.compiler.SQLCompiler object at 0x7f8685e49160>
       db         = 'fdw'
       init_list  = ['id', 'queue', 'args', 'kwargs', 'type', 'state', ...]
       klass_info = {'model': <class 'paiyroll.models.batch.Job'>, 'select_fields': [0, 1, 2, 3, 4, 5, ...]}
       known_related_objects = []
       model_cls  = <class 'paiyroll.models.batch.Job'>
       model_fields_end = 9
       model_fields_start = 0
       queryset   = <QuerySet []>
       related_populators = []
       results    = [[('8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery', ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {}, 'paiyroll.tasks.function_run', 'scheduled', ...)]]
       select     = [(Col(jobs, paiyroll.Job.id), ('"jobs"."id"', []), None), (Col(jobs, paiyroll.Job.queue), ('"jobs"."queue"', []), None..., paiyroll.Job.type), ('"jobs"."type"', []), None), (Col(jobs, paiyroll.Job.state), ('"jobs"."state"', []), None), ...]
       select_fields = [0, 1, 2, 3, 4, 5, ...]
       self       = <django.db.models.query.ModelIterable object at 0x7f86836f3040>
/usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py:1100: in apply_converters
   value = converter(value, expression, connection)
       connection = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f869a321670>
       converter  = <bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>
       converters = [(2, ([<bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>], Col(jobs, pa...NField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: details>>], Col(jobs, paiyroll.Job.details)))]
       convs      = [<bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>]
       expression = Col(jobs, paiyroll.Job.args)
       pos        = 2
       row        = ['8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery', ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {}, 'paiyroll.tasks.function_run', 'scheduled', ...]
       rows       = <itertools.chain object at 0x7f8683ae7520>
       self       = <django.db.models.sql.compiler.SQLCompiler object at 0x7f8685e49160>
       value      = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]
/usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74: in from_db_value
   return json.loads(value, cls=self.decoder)
       connection = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f869a321670>
       expression = Col(jobs, paiyroll.Job.args)
       self       = <django.contrib.postgres.fields.jsonb.JSONField: args>
       value      = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

s = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], cls = None
object_hook = None, parse_float = None, parse_int = None, parse_constant = None
object_pairs_hook = None, kw = {}

   def loads(s, *, cls=None, object_hook=None, parse_float=None,
           parse_int=None, parse_constant=None, object_pairs_hook=None, **kw):
       """Deserialize ``s`` (a ``str``, ``bytes`` or ``bytearray`` instance
       containing a JSON document) to a Python object.

       ``object_hook`` is an optional function that will be called with the
       result of any object literal decode (a ``dict``). The return value of
       ``object_hook`` will be used instead of the ``dict``. This feature
       can be used to implement custom decoders (e.g. JSON-RPC class hinting).

       ``object_pairs_hook`` is an optional function that will be called with the
       result of any object literal decoded with an ordered list of pairs.  The
       return value of ``object_pairs_hook`` will be used instead of the ``dict``.
       This feature can be used to implement custom decoders.  If ``object_hook``
       is also defined, the ``object_pairs_hook`` takes priority.

       ``parse_float``, if specified, will be called with the string
       of every JSON float to be decoded. By default this is equivalent to
       float(num_str). This can be used to use another datatype or parser
       for JSON floats (e.g. decimal.Decimal).

       ``parse_int``, if specified, will be called with the string
       of every JSON int to be decoded. By default this is equivalent to
       int(num_str). This can be used to use another datatype or parser
       for JSON integers (e.g. float).

       ``parse_constant``, if specified, will be called with one of the
       following strings: -Infinity, Infinity, NaN.
       This can be used to raise an exception if invalid JSON numbers
       are encountered.

       To use a custom ``JSONDecoder`` subclass, specify it with the ``cls``
       kwarg; otherwise ``JSONDecoder`` is used.

       The ``encoding`` argument is ignored and deprecated since Python 3.1.
       """
       if isinstance(s, str):
           if s.startswith('\ufeff'):
               raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)",
                                     s, 0)
       else:
           if not isinstance(s, (bytes, bytearray)):
>               raise TypeError(f'the JSON object must be str, bytes or bytearray, '
                               f'not {s.__class__.__name__}')
E               TypeError: the JSON object must be str, bytes or bytearray, not list

cls        = None
kw         = {}
object_hook = None
object_pairs_hook = None
parse_constant = None
parse_float = None
parse_int  = None
s          = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]
===============================

As you can perhaps see from the bolded part (for /usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74), the value being written into the JSONField called "args" is a Python list, (shown as "s" on the last line of the traceback-with-values). I am aware of documented changes around serializers but did not think that affected me; however, given that I am using an FDW to return the data, is it that I should now be serialising returned data into a string?

Any pointers appreciated.

Thanks, Shaheed

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/f5e441c5-f38a-4260-8188-d002fbe83c99n%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Django 3.1.2 JSONField handling error with an Postgres Foreign Data Wrapper-based model

Shaheed Haque-2
Indeed. And to close out the discussion I can confirm that my FDW implementations were using JSON columns, and the issue was resolved by switching to JSONB instead. 

On Sun, 18 Oct 2020, 14:40 Jason, <[hidden email]> wrote:
looks like you got a response back, and there's more context in the linked ticket too.  tl;dr json is not jsonb, and django only works with the latter

On Thursday, October 15, 2020 at 10:02:30 AM UTC-4 [hidden email] wrote:
Thanks Jason, I filed https://code.djangoproject.com/ticket/32111#ticket.

On Thursday, 15 October 2020 at 12:51:22 UTC+1 Jason wrote:
this is pretty interesting, and may be something to report to the devs at https://groups.google.com/g/django-developers, or open up a ticket at https://code.djangoproject.com/  I don't see any tickets with the query jsonfield foreign data that are similar with your experience, so you may have found an edge case.

On Thursday, October 15, 2020 at 5:31:09 AM UTC-4 [hidden email] wrote:

Hi,

I have a Django model working fine under Django 3.0 (i.e. "Django<3.1") which looks like this:

===================
    class Job(models.Model):
        id = models.CharField(max_length=36, primary_key=True)
        queue = models.CharField(max_length=40)
        args = JSONField()
        kwargs = JSONField()
        type = models.CharField(max_length=80)
        ...

        class Meta:
            managed = False  # <------   The table is implemented as a Postgres FDW wrapper.
            db_table = 'jobs'
===================

I am testing the update to Django 3.1.2 and hit an error in executing this line:

    jobs = list(models.Job.objects.filter(queue='celery', state='scheduled'))

The error is as follows from pytest (i.e. stack trace with local variables too):

==================
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:287: in __iter__
   self._fetch_all()
       self       = <QuerySet []>
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:1308: in _fetch_all
   self._result_cache = list(self._iterable_class(self))
       self       = <QuerySet []>
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:70: in __iter__
   for row in compiler.results_iter(results):
       annotation_col_map = {}
       compiler   = <django.db.models.sql.compiler.SQLCompiler object at 0x7f8685e49160>
       db         = 'fdw'
       init_list  = ['id', 'queue', 'args', 'kwargs', 'type', 'state', ...]
       klass_info = {'model': <class 'paiyroll.models.batch.Job'>, 'select_fields': [0, 1, 2, 3, 4, 5, ...]}
       known_related_objects = []
       model_cls  = <class 'paiyroll.models.batch.Job'>
       model_fields_end = 9
       model_fields_start = 0
       queryset   = <QuerySet []>
       related_populators = []
       results    = [[('8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery', ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {}, 'paiyroll.tasks.function_run', 'scheduled', ...)]]
       select     = [(Col(jobs, paiyroll.Job.id), ('"jobs"."id"', []), None), (Col(jobs, paiyroll.Job.queue), ('"jobs"."queue"', []), None..., paiyroll.Job.type), ('"jobs"."type"', []), None), (Col(jobs, paiyroll.Job.state), ('"jobs"."state"', []), None), ...]
       select_fields = [0, 1, 2, 3, 4, 5, ...]
       self       = <django.db.models.query.ModelIterable object at 0x7f86836f3040>
/usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py:1100: in apply_converters
   value = converter(value, expression, connection)
       connection = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f869a321670>
       converter  = <bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>
       converters = [(2, ([<bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>], Col(jobs, pa...NField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: details>>], Col(jobs, paiyroll.Job.details)))]
       convs      = [<bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>]
       expression = Col(jobs, paiyroll.Job.args)
       pos        = 2
       row        = ['8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery', ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {}, 'paiyroll.tasks.function_run', 'scheduled', ...]
       rows       = <itertools.chain object at 0x7f8683ae7520>
       self       = <django.db.models.sql.compiler.SQLCompiler object at 0x7f8685e49160>
       value      = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]
/usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74: in from_db_value
   return json.loads(value, cls=self.decoder)
       connection = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f869a321670>
       expression = Col(jobs, paiyroll.Job.args)
       self       = <django.contrib.postgres.fields.jsonb.JSONField: args>
       value      = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

s = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], cls = None
object_hook = None, parse_float = None, parse_int = None, parse_constant = None
object_pairs_hook = None, kw = {}

   def loads(s, *, cls=None, object_hook=None, parse_float=None,
           parse_int=None, parse_constant=None, object_pairs_hook=None, **kw):
       """Deserialize ``s`` (a ``str``, ``bytes`` or ``bytearray`` instance
       containing a JSON document) to a Python object.

       ``object_hook`` is an optional function that will be called with the
       result of any object literal decode (a ``dict``). The return value of
       ``object_hook`` will be used instead of the ``dict``. This feature
       can be used to implement custom decoders (e.g. JSON-RPC class hinting).

       ``object_pairs_hook`` is an optional function that will be called with the
       result of any object literal decoded with an ordered list of pairs.  The
       return value of ``object_pairs_hook`` will be used instead of the ``dict``.
       This feature can be used to implement custom decoders.  If ``object_hook``
       is also defined, the ``object_pairs_hook`` takes priority.

       ``parse_float``, if specified, will be called with the string
       of every JSON float to be decoded. By default this is equivalent to
       float(num_str). This can be used to use another datatype or parser
       for JSON floats (e.g. decimal.Decimal).

       ``parse_int``, if specified, will be called with the string
       of every JSON int to be decoded. By default this is equivalent to
       int(num_str). This can be used to use another datatype or parser
       for JSON integers (e.g. float).

       ``parse_constant``, if specified, will be called with one of the
       following strings: -Infinity, Infinity, NaN.
       This can be used to raise an exception if invalid JSON numbers
       are encountered.

       To use a custom ``JSONDecoder`` subclass, specify it with the ``cls``
       kwarg; otherwise ``JSONDecoder`` is used.

       The ``encoding`` argument is ignored and deprecated since Python 3.1.
       """
       if isinstance(s, str):
           if s.startswith('\ufeff'):
               raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)",
                                     s, 0)
       else:
           if not isinstance(s, (bytes, bytearray)):
>               raise TypeError(f'the JSON object must be str, bytes or bytearray, '
                               f'not {s.__class__.__name__}')
E               TypeError: the JSON object must be str, bytes or bytearray, not list

cls        = None
kw         = {}
object_hook = None
object_pairs_hook = None
parse_constant = None
parse_float = None
parse_int  = None
s          = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]
===============================

As you can perhaps see from the bolded part (for /usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74), the value being written into the JSONField called "args" is a Python list, (shown as "s" on the last line of the traceback-with-values). I am aware of documented changes around serializers but did not think that affected me; however, given that I am using an FDW to return the data, is it that I should now be serialising returned data into a string?

Any pointers appreciated.

Thanks, Shaheed

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/f5e441c5-f38a-4260-8188-d002fbe83c99n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAHAc2jeZRPQNBF-V7nujUK1T2pZJZk0LV%3DHJt%2BS2ZwWW5jdqzQ%40mail.gmail.com.