Are there use cases for storing null bytes in CharField/TextField?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Are there use cases for storing null bytes in CharField/TextField?

Tim Graham-2
Does anyone know of a use case for using null bytes in CharField/TextField?

psycopg2 2.7+ raises ValueError("A string literal cannot contain NUL (0x00) characters.") when trying to save null bytes [0] and this exception is unhandled in Django which allow malicious form submissions to crash [1]. With psycopg2 < 2.7, there is no exception and null bytes are silently truncated by PostgreSQL. Other databases that I tested (SQLite, MySQL, Oracle) allow saving null bytes. This creates possible cross-database compatibility problems when moving data from those databases to PostgreSQL, e.g.[2].

I propose to have CharField and TextField strip null bytes from the value either a) only on PostgreSQL or b) on all databases. Please indicate your preference or suggest another solution.

[0] https://github.com/psycopg/psycopg2/issues/420
[1] https://code.djangoproject.com/ticket/28201 - Saving a Char/TextField with psycopg2 2.7+ raises ValueError: A string literal cannot contain NUL (0x00) characters is unhandled
[2] https://code.djangoproject.com/ticket/28117 - loaddata raises ValueError with psycopg2 backend when data contains null bytes

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Are there use cases for storing null bytes in CharField/TextField?

Adam Johnson-2
The problem with (a) - data with null bytes in strings from other databases can't be loaded into PG as per #28117 .

The problem with (b) - data currently in databases in the wild will be modified upon save 😱

(b) is incredibly destructive and could break an unknown number of applications whilst (a) doesn't affect anyone until they try to migrate null-byte-strings into PG. I vote for (a), or (c) add form-level validation to (Char/Text)Field that null bytes aren't in the submitted string (for all databases) and error when trying to save them on PG.


On 15 May 2017 at 16:54, Tim Graham <[hidden email]> wrote:
Does anyone know of a use case for using null bytes in CharField/TextField?

psycopg2 2.7+ raises ValueError("A string literal cannot contain NUL (0x00) characters.") when trying to save null bytes [0] and this exception is unhandled in Django which allow malicious form submissions to crash [1]. With psycopg2 < 2.7, there is no exception and null bytes are silently truncated by PostgreSQL. Other databases that I tested (SQLite, MySQL, Oracle) allow saving null bytes. This creates possible cross-database compatibility problems when moving data from those databases to PostgreSQL, e.g.[2].

I propose to have CharField and TextField strip null bytes from the value either a) only on PostgreSQL or b) on all databases. Please indicate your preference or suggest another solution.

[0] https://github.com/psycopg/psycopg2/issues/420
[1] https://code.djangoproject.com/ticket/28201 - Saving a Char/TextField with psycopg2 2.7+ raises ValueError: A string literal cannot contain NUL (0x00) characters is unhandled
[2] https://code.djangoproject.com/ticket/28117 - loaddata raises ValueError with psycopg2 backend when data contains null bytes

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Adam

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMyDDM1qVc3ovXb9PhzKY3jd__FURYX6Fy9r1WFrBpcpMy%2Bz%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Are there use cases for storing null bytes in CharField/TextField?

Michael Manfre
In reply to this post by Tim Graham-2
I imagine we won't hear of a use case until after the change happens and I'm some what strongly opposed to stripping potentially valid data from all databases because of a limitation of one. I'd be in favor of loaddata checking for null bytes and complaining when the backend doesn't support that feature.

Regards,
Michael Manfre

On Mon, May 15, 2017 at 11:54 AM Tim Graham <[hidden email]> wrote:
Does anyone know of a use case for using null bytes in CharField/TextField?

psycopg2 2.7+ raises ValueError("A string literal cannot contain NUL (0x00) characters.") when trying to save null bytes [0] and this exception is unhandled in Django which allow malicious form submissions to crash [1]. With psycopg2 < 2.7, there is no exception and null bytes are silently truncated by PostgreSQL. Other databases that I tested (SQLite, MySQL, Oracle) allow saving null bytes. This creates possible cross-database compatibility problems when moving data from those databases to PostgreSQL, e.g.[2].

I propose to have CharField and TextField strip null bytes from the value either a) only on PostgreSQL or b) on all databases. Please indicate your preference or suggest another solution.

[0] https://github.com/psycopg/psycopg2/issues/420
[1] https://code.djangoproject.com/ticket/28201 - Saving a Char/TextField with psycopg2 2.7+ raises ValueError: A string literal cannot contain NUL (0x00) characters is unhandled
[2] https://code.djangoproject.com/ticket/28117 - loaddata raises ValueError with psycopg2 backend when data contains null bytes

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAGdCwBsvwtr4F3j1jGo9uGTwBsjvU0ypLc%2B2q0482Peha3ejzw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Are there use cases for storing null bytes in CharField/TextField?

Luke Plant-2
In reply to this post by Adam Johnson-2

I agree with Adam, we should never silently change submitted data at the model layer. My preference would be c), a form-level validation error that prevents saving.

Luke


On 15/05/17 19:11, Adam Johnson wrote:
The problem with (a) - data with null bytes in strings from other databases can't be loaded into PG as per #28117 .

The problem with (b) - data currently in databases in the wild will be modified upon save 😱

(b) is incredibly destructive and could break an unknown number of applications whilst (a) doesn't affect anyone until they try to migrate null-byte-strings into PG. I vote for (a), or (c) add form-level validation to (Char/Text)Field that null bytes aren't in the submitted string (for all databases) and error when trying to save them on PG.


On 15 May 2017 at 16:54, Tim Graham <[hidden email]> wrote:
Does anyone know of a use case for using null bytes in CharField/TextField?

psycopg2 2.7+ raises ValueError("A string literal cannot contain NUL (0x00) characters.") when trying to save null bytes [0] and this exception is unhandled in Django which allow malicious form submissions to crash [1]. With psycopg2 < 2.7, there is no exception and null bytes are silently truncated by PostgreSQL. Other databases that I tested (SQLite, MySQL, Oracle) allow saving null bytes. This creates possible cross-database compatibility problems when moving data from those databases to PostgreSQL, e.g.[2].

I propose to have CharField and TextField strip null bytes from the value either a) only on PostgreSQL or b) on all databases. Please indicate your preference or suggest another solution.

[0] https://github.com/psycopg/psycopg2/issues/420
[1] https://code.djangoproject.com/ticket/28201 - Saving a Char/TextField with psycopg2 2.7+ raises ValueError: A string literal cannot contain NUL (0x00) characters is unhandled
[2] https://code.djangoproject.com/ticket/28117 - loaddata raises ValueError with psycopg2 backend when data contains null bytes
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Adam
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMyDDM1qVc3ovXb9PhzKY3jd__FURYX6Fy9r1WFrBpcpMy%2Bz%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1fbe9f18-f935-93eb-dd90-ffa754ad9c2b%40cantab.net.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Are there use cases for storing null bytes in CharField/TextField?

Tim Chase-7
In reply to this post by Tim Graham-2
On 2017-05-15 08:54, Tim Graham wrote:
> Does anyone know of a use case for using null bytes in
> CharField/TextField?

Is this not what BinaryField is for?  It would seem to me that
attempting to store binary NULL bytes in a CharField/TextField should
result in an error condition.

-tkc



--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20170515123050.363a2859%40bigbox.christie.dr.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Are there use cases for storing null bytes in CharField/TextField?

Claude Paroz
In reply to this post by Tim Graham-2
I also think that this should be handled at serialization level (form fields and (de)serialization framework).

Claude

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/a9dde655-4bb1-409f-883e-6f47f742f17a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Are there use cases for storing null bytes in CharField/TextField?

Jani Tiainen
In reply to this post by Tim Graham-2

Hi,

I would guess that one could use null byte to denote "empty field" in Oracle for example. (I recall seeing such a convention in one of our non-django apps). And that's to overcome limitation that Oracle doesn't have real concept of empty string so we stored single null byte to mark that.



On 15.05.2017 18:54, Tim Graham wrote:
Does anyone know of a use case for using null bytes in CharField/TextField?

psycopg2 2.7+ raises ValueError("A string literal cannot contain NUL (0x00) characters.") when trying to save null bytes [0] and this exception is unhandled in Django which allow malicious form submissions to crash [1]. With psycopg2 < 2.7, there is no exception and null bytes are silently truncated by PostgreSQL. Other databases that I tested (SQLite, MySQL, Oracle) allow saving null bytes. This creates possible cross-database compatibility problems when moving data from those databases to PostgreSQL, e.g.[2].

I propose to have CharField and TextField strip null bytes from the value either a) only on PostgreSQL or b) on all databases. Please indicate your preference or suggest another solution.

[0] https://github.com/psycopg/psycopg2/issues/420
[1] https://code.djangoproject.com/ticket/28201 - Saving a Char/TextField with psycopg2 2.7+ raises ValueError: A string literal cannot contain NUL (0x00) characters is unhandled
[2] https://code.djangoproject.com/ticket/28117 - loaddata raises ValueError with psycopg2 backend when data contains null bytes
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
Jani Tiainen

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/22fac845-6870-de4e-6fbe-eab247b8853a%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Are there use cases for storing null bytes in CharField/TextField?

Tim Graham-2
If CharField/TextField have a form validation error if null bytes are in the input, are users going to be able to understand that error and fix it? I'm not sure if it's a probable case, but I'm thinking of a non-technical user who copy/pastes some text that includes a null byte.

Perhaps a " strip_null_bytes" model field option that defaults to True would be reasonable. That could be passed to the form field to toggle where or not that validation happens. Actually, three possible behaviors might be needed: silently strip null bytes, allow null bytes (an invalid option when using PostgreSQL), prohibit null bytes.

On Tuesday, May 16, 2017 at 5:11:38 AM UTC-4, Jani Tiainen wrote:

Hi,

I would guess that one could use null byte to denote "empty field" in Oracle for example. (I recall seeing such a convention in one of our non-django apps). And that's to overcome limitation that Oracle doesn't have real concept of empty string so we stored single null byte to mark that.



On 15.05.2017 18:54, Tim Graham wrote:
Does anyone know of a use case for using null bytes in CharField/TextField?

psycopg2 2.7+ raises ValueError("A string literal cannot contain NUL (0x00) characters.") when trying to save null bytes [0] and this exception is unhandled in Django which allow malicious form submissions to crash [1]. With psycopg2 < 2.7, there is no exception and null bytes are silently truncated by PostgreSQL. Other databases that I tested (SQLite, MySQL, Oracle) allow saving null bytes. This creates possible cross-database compatibility problems when moving data from those databases to PostgreSQL, e.g.[2].

I propose to have CharField and TextField strip null bytes from the value either a) only on PostgreSQL or b) on all databases. Please indicate your preference or suggest another solution.

[0] <a href="https://github.com/psycopg/psycopg2/issues/420" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fpsycopg%2Fpsycopg2%2Fissues%2F420\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEVW66v1FcxVK4nWLzP_JWtUCzyWw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fpsycopg%2Fpsycopg2%2Fissues%2F420\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEVW66v1FcxVK4nWLzP_JWtUCzyWw&#39;;return true;">https://github.com/psycopg/psycopg2/issues/420
[1] <a href="https://code.djangoproject.com/ticket/28201" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F28201\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEICgeGqJr3UHokezoR2BxE_2FVGQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F28201\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEICgeGqJr3UHokezoR2BxE_2FVGQ&#39;;return true;">https://code.djangoproject.com/ticket/28201 - Saving a Char/TextField with psycopg2 2.7+ raises ValueError: A string literal cannot contain NUL (0x00) characters is unhandled
[2] <a href="https://code.djangoproject.com/ticket/28117" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F28117\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFuvBnzg9lmjUFQOeP2sPZt8LKTMA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F28117\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFuvBnzg9lmjUFQOeP2sPZt8LKTMA&#39;;return true;">https://code.djangoproject.com/ticket/28117 - loaddata raises ValueError with psycopg2 backend when data contains null bytes
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="pVOvNw5CAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-develop...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="pVOvNw5CAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-d...@googlegroups.com.
Visit this group at <a href="https://groups.google.com/group/django-developers" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

-- 
Jani Tiainen

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/7226ab56-ff69-415c-9955-82e8d62cdd60%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Are there use cases for storing null bytes in CharField/TextField?

Tim Graham-2
A reply from Luke came only to me:
----

Is it even possible for null bytes to be copied into a text field and for them to be submitted? The original report uses Javascript to get them in there - https://code.djangoproject.com/ticket/28201

These pages suggests it is going to be really hard for users to be entering 0x00 accidentally:

- http://stackoverflow.com/questions/6961208/how-to-input-a-null-character-into-a-web-form

- https://superuser.com/questions/946533/is-there-any-way-to-copy-null-bytes-ascii-0x00-to-the-clipboard-on-windows

If we are talking about this kind of rare condition, I think a slightly obscure error message is fine. Defaulting to stripping any data would be a bad idea IMO, for the case where a backend is currently handling and storing 0x00 chars we should try hard not to break that.

Luke
----

I guess we'll proceed with the form field / serialization enhancements.

On Friday, May 19, 2017 at 9:35:28 AM UTC-4, Tim Graham wrote:
If CharField/TextField have a form validation error if null bytes are in the input, are users going to be able to understand that error and fix it? I'm not sure if it's a probable case, but I'm thinking of a non-technical user who copy/pastes some text that includes a null byte.

Perhaps a " strip_null_bytes" model field option that defaults to True would be reasonable. That could be passed to the form field to toggle where or not that validation happens. Actually, three possible behaviors might be needed: silently strip null bytes, allow null bytes (an invalid option when using PostgreSQL), prohibit null bytes.

On Tuesday, May 16, 2017 at 5:11:38 AM UTC-4, Jani Tiainen wrote:

Hi,

I would guess that one could use null byte to denote "empty field" in Oracle for example. (I recall seeing such a convention in one of our non-django apps). And that's to overcome limitation that Oracle doesn't have real concept of empty string so we stored single null byte to mark that.



On 15.05.2017 18:54, Tim Graham wrote:
Does anyone know of a use case for using null bytes in CharField/TextField?

psycopg2 2.7+ raises ValueError("A string literal cannot contain NUL (0x00) characters.") when trying to save null bytes [0] and this exception is unhandled in Django which allow malicious form submissions to crash [1]. With psycopg2 < 2.7, there is no exception and null bytes are silently truncated by PostgreSQL. Other databases that I tested (SQLite, MySQL, Oracle) allow saving null bytes. This creates possible cross-database compatibility problems when moving data from those databases to PostgreSQL, e.g.[2].

I propose to have CharField and TextField strip null bytes from the value either a) only on PostgreSQL or b) on all databases. Please indicate your preference or suggest another solution.

[0] <a href="https://github.com/psycopg/psycopg2/issues/420" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fpsycopg%2Fpsycopg2%2Fissues%2F420\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEVW66v1FcxVK4nWLzP_JWtUCzyWw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fpsycopg%2Fpsycopg2%2Fissues%2F420\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEVW66v1FcxVK4nWLzP_JWtUCzyWw&#39;;return true;">https://github.com/psycopg/psycopg2/issues/420
[1] <a href="https://code.djangoproject.com/ticket/28201" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F28201\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEICgeGqJr3UHokezoR2BxE_2FVGQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F28201\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEICgeGqJr3UHokezoR2BxE_2FVGQ&#39;;return true;">https://code.djangoproject.com/ticket/28201 - Saving a Char/TextField with psycopg2 2.7+ raises ValueError: A string literal cannot contain NUL (0x00) characters is unhandled
[2] <a href="https://code.djangoproject.com/ticket/28117" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F28117\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFuvBnzg9lmjUFQOeP2sPZt8LKTMA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F28117\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFuvBnzg9lmjUFQOeP2sPZt8LKTMA&#39;;return true;">https://code.djangoproject.com/ticket/28117 - loaddata raises ValueError with psycopg2 backend when data contains null bytes
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/django-developers" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

-- 
Jani Tiainen

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/35ac6812-e02e-47fc-a3f0-e615e59c4de2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Are there use cases for storing null bytes in CharField/TextField?

Jon Dufresne
In reply to this post by Tim Chase-7
On Mon, May 15, 2017 at 10:30 AM, Tim Chase <[hidden email]> wrote:
On 2017-05-15 08:54, Tim Graham wrote:
> Does anyone know of a use case for using null bytes in
> CharField/TextField?

Is this not what BinaryField is for?  It would seem to me that
attempting to store binary NULL bytes in a CharField/TextField should
result in an error condition.

The null byte is also a valid Unicode code point [0].

I guess I'm a bit surprised that a valid code point can't be stored in a PostgreSQL text column. This does appear to be documented for the char(int) string function [1], although without justification.

> The NULL (0) character is not allowed because text data types cannot store such bytes.

I'm curious behind PostgreSQL's decision to prohibit this code point. If anyone has additional information to share on their reason, please pass it along.


Cheers

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CADhq2b6egRaJ8t23HRXwjfqTHjTJx%3DjRPSL8P%3D6MUTyotFsP6w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Are there use cases for storing null bytes in CharField/TextField?

Tim Graham-2
I found a PostgreSQL bug report requesting removal of the restriction. Here's the final reply:

Franklin Schmidt wrote:
> I agree that storing 0x00 in a UTF8 string is weird, but I am
> converting a huge database to postgres, and in a huge database, weird
> things happen.  Using bytea for a text field just because one in a
> million records has a 0x00 doesn't make sense to me.  I did hack
> around it in my conversion code to remove the 0x00 but I expect that
> anyone else who tries converting a big database to postgres will also
> confront this issue.

That's the right solution. If you have 0x00 bytes in your text fields, 
you're much better off cleaning them away anyway, than trying to work 
around them.
-Heikki Linnakangas

https://www.postgresql.org/message-id/200712170734.lBH7YdG9034458%40wwwmaster.postgresql.org

I also found a possible related discussion about supporting \u0000 in JSON values [1]. PostgreSQL tried to support it but had to remove that support because it caused ambiguity.

https://www.postgresql.org/message-id/[hidden email]

On Wednesday, May 31, 2017 at 8:27:49 PM UTC-4, Jon Dufresne wrote:
On Mon, May 15, 2017 at 10:30 AM, Tim Chase <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="AgdBAsWcBQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django...@....com> wrote:
On 2017-05-15 08:54, Tim Graham wrote:
> Does anyone know of a use case for using null bytes in
> CharField/TextField?

Is this not what BinaryField is for?  It would seem to me that
attempting to store binary NULL bytes in a CharField/TextField should
result in an error condition.

The null byte is also a valid Unicode code point [0].

I guess I'm a bit surprised that a valid code point can't be stored in a PostgreSQL text column. This does appear to be documented for the char(int) string function [1], although without justification.

> The NULL (0) character is not allowed because text data types cannot store such bytes.

I'm curious behind PostgreSQL's decision to prohibit this code point. If anyone has additional information to share on their reason, please pass it along.


[0] <a href="http://www.fileformat.info/info/unicode/char/0000/index.htm" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.fileformat.info%2Finfo%2Funicode%2Fchar%2F0000%2Findex.htm\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEmyXnIam5pXTQEB_CiW_FnpcxeKQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.fileformat.info%2Finfo%2Funicode%2Fchar%2F0000%2Findex.htm\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEmyXnIam5pXTQEB_CiW_FnpcxeKQ&#39;;return true;">http://www.fileformat.info/info/unicode/char/0000/index.htm
[1] <a href="https://www.postgresql.org/docs/current/static/functions-string.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.postgresql.org%2Fdocs%2Fcurrent%2Fstatic%2Ffunctions-string.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHNF-_-Gt-otYkZzXl4fMEKp9NzWg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.postgresql.org%2Fdocs%2Fcurrent%2Fstatic%2Ffunctions-string.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHNF-_-Gt-otYkZzXl4fMEKp9NzWg&#39;;return true;">https://www.postgresql.org/docs/current/static/functions-string.html

Cheers

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/450da309-c2af-43a9-8026-51d23994f8c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.