Quantcast

Are there use cases for storing null bytes in CharField/TextField?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Are there use cases for storing null bytes in CharField/TextField?

Tim Graham-2
Does anyone know of a use case for using null bytes in CharField/TextField?

psycopg2 2.7+ raises ValueError("A string literal cannot contain NUL (0x00) characters.") when trying to save null bytes [0] and this exception is unhandled in Django which allow malicious form submissions to crash [1]. With psycopg2 < 2.7, there is no exception and null bytes are silently truncated by PostgreSQL. Other databases that I tested (SQLite, MySQL, Oracle) allow saving null bytes. This creates possible cross-database compatibility problems when moving data from those databases to PostgreSQL, e.g.[2].

I propose to have CharField and TextField strip null bytes from the value either a) only on PostgreSQL or b) on all databases. Please indicate your preference or suggest another solution.

[0] https://github.com/psycopg/psycopg2/issues/420
[1] https://code.djangoproject.com/ticket/28201 - Saving a Char/TextField with psycopg2 2.7+ raises ValueError: A string literal cannot contain NUL (0x00) characters is unhandled
[2] https://code.djangoproject.com/ticket/28117 - loaddata raises ValueError with psycopg2 backend when data contains null bytes

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Are there use cases for storing null bytes in CharField/TextField?

Adam Johnson-2
The problem with (a) - data with null bytes in strings from other databases can't be loaded into PG as per #28117 .

The problem with (b) - data currently in databases in the wild will be modified upon save ­čś▒

(b) is incredibly destructive and could break an unknown number of applications whilst (a) doesn't affect anyone until they try to migrate null-byte-strings into PG. I vote for (a), or (c) add form-level validation to (Char/Text)Field that null bytes aren't in the submitted string (for all databases) and error when trying to save them on PG.


On 15 May 2017 at 16:54, Tim Graham <[hidden email]> wrote:
Does anyone know of a use case for using null bytes in CharField/TextField?

psycopg2 2.7+ raises ValueError("A string literal cannot contain NUL (0x00) characters.") when trying to save null bytes [0] and this exception is unhandled in Django which allow malicious form submissions to crash [1]. With psycopg2 < 2.7, there is no exception and null bytes are silently truncated by PostgreSQL. Other databases that I tested (SQLite, MySQL, Oracle) allow saving null bytes. This creates possible cross-database compatibility problems when moving data from those databases to PostgreSQL, e.g.[2].

I propose to have CharField and TextField strip null bytes from the value either a) only on PostgreSQL or b) on all databases. Please indicate your preference or suggest another solution.

[0] https://github.com/psycopg/psycopg2/issues/420
[1] https://code.djangoproject.com/ticket/28201 - Saving a Char/TextField with psycopg2 2.7+ raises ValueError: A string literal cannot contain NUL (0x00) characters is unhandled
[2] https://code.djangoproject.com/ticket/28117 - loaddata raises ValueError with psycopg2 backend when data contains null bytes

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Adam

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMyDDM1qVc3ovXb9PhzKY3jd__FURYX6Fy9r1WFrBpcpMy%2Bz%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Are there use cases for storing null bytes in CharField/TextField?

Michael Manfre
In reply to this post by Tim Graham-2
I imagine we won't hear of a use case until after the change happens and I'm some what strongly opposed to stripping potentially valid data from all databases because of a limitation of one. I'd be in favor of loaddata checking for null bytes and complaining when the backend doesn't support that feature.

Regards,
Michael Manfre

On Mon, May 15, 2017 at 11:54 AM Tim Graham <[hidden email]> wrote:
Does anyone know of a use case for using null bytes in CharField/TextField?

psycopg2 2.7+ raises ValueError("A string literal cannot contain NUL (0x00) characters.") when trying to save null bytes [0] and this exception is unhandled in Django which allow malicious form submissions to crash [1]. With psycopg2 < 2.7, there is no exception and null bytes are silently truncated by PostgreSQL. Other databases that I tested (SQLite, MySQL, Oracle) allow saving null bytes. This creates possible cross-database compatibility problems when moving data from those databases to PostgreSQL, e.g.[2].

I propose to have CharField and TextField strip null bytes from the value either a) only on PostgreSQL or b) on all databases. Please indicate your preference or suggest another solution.

[0] https://github.com/psycopg/psycopg2/issues/420
[1] https://code.djangoproject.com/ticket/28201 - Saving a Char/TextField with psycopg2 2.7+ raises ValueError: A string literal cannot contain NUL (0x00) characters is unhandled
[2] https://code.djangoproject.com/ticket/28117 - loaddata raises ValueError with psycopg2 backend when data contains null bytes

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAGdCwBsvwtr4F3j1jGo9uGTwBsjvU0ypLc%2B2q0482Peha3ejzw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Are there use cases for storing null bytes in CharField/TextField?

Luke Plant-2
In reply to this post by Adam Johnson-2

I agree with Adam, we should never silently change submitted data at the model layer. My preference would be c), a form-level validation error that prevents saving.

Luke


On 15/05/17 19:11, Adam Johnson wrote:
The problem with (a) - data with null bytes in strings from other databases can't be loaded into PG as per #28117 .

The problem with (b) - data currently in databases in the wild will be modified upon save ­čś▒

(b) is incredibly destructive and could break an unknown number of applications whilst (a) doesn't affect anyone until they try to migrate null-byte-strings into PG. I vote for (a), or (c) add form-level validation to (Char/Text)Field that null bytes aren't in the submitted string (for all databases) and error when trying to save them on PG.


On 15 May 2017 at 16:54, Tim Graham <[hidden email]> wrote:
Does anyone know of a use case for using null bytes in CharField/TextField?

psycopg2 2.7+ raises ValueError("A string literal cannot contain NUL (0x00) characters.") when trying to save null bytes [0] and this exception is unhandled in Django which allow malicious form submissions to crash [1]. With psycopg2 < 2.7, there is no exception and null bytes are silently truncated by PostgreSQL. Other databases that I tested (SQLite, MySQL, Oracle) allow saving null bytes. This creates possible cross-database compatibility problems when moving data from those databases to PostgreSQL, e.g.[2].

I propose to have CharField and TextField strip null bytes from the value either a) only on PostgreSQL or b) on all databases. Please indicate your preference or suggest another solution.

[0] https://github.com/psycopg/psycopg2/issues/420
[1] https://code.djangoproject.com/ticket/28201 - Saving a Char/TextField with psycopg2 2.7+ raises ValueError: A string literal cannot contain NUL (0x00) characters is unhandled
[2] https://code.djangoproject.com/ticket/28117 - loaddata raises ValueError with psycopg2 backend when data contains null bytes
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Adam
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMyDDM1qVc3ovXb9PhzKY3jd__FURYX6Fy9r1WFrBpcpMy%2Bz%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1fbe9f18-f935-93eb-dd90-ffa754ad9c2b%40cantab.net.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Are there use cases for storing null bytes in CharField/TextField?

Tim Chase-7
In reply to this post by Tim Graham-2
On 2017-05-15 08:54, Tim Graham wrote:
> Does anyone know of a use case for using null bytes in
> CharField/TextField?

Is this not what BinaryField is for?  It would seem to me that
attempting to store binary NULL bytes in a CharField/TextField should
result in an error condition.

-tkc



--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20170515123050.363a2859%40bigbox.christie.dr.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Are there use cases for storing null bytes in CharField/TextField?

Claude Paroz
In reply to this post by Tim Graham-2
I also think that this should be handled at serialization level (form fields and (de)serialization framework).

Claude

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/a9dde655-4bb1-409f-883e-6f47f742f17a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Are there use cases for storing null bytes in CharField/TextField?

Jani Tiainen
In reply to this post by Tim Graham-2

Hi,

I would guess that one could use null byte to denote "empty field" in Oracle for example. (I recall seeing such a convention in one of our non-django apps). And that's to overcome limitation that Oracle doesn't have real concept of empty string so we stored single null byte to mark that.



On 15.05.2017 18:54, Tim Graham wrote:
Does anyone know of a use case for using null bytes in CharField/TextField?

psycopg2 2.7+ raises ValueError("A string literal cannot contain NUL (0x00) characters.") when trying to save null bytes [0] and this exception is unhandled in Django which allow malicious form submissions to crash [1]. With psycopg2 < 2.7, there is no exception and null bytes are silently truncated by PostgreSQL. Other databases that I tested (SQLite, MySQL, Oracle) allow saving null bytes. This creates possible cross-database compatibility problems when moving data from those databases to PostgreSQL, e.g.[2].

I propose to have CharField and TextField strip null bytes from the value either a) only on PostgreSQL or b) on all databases. Please indicate your preference or suggest another solution.

[0] https://github.com/psycopg/psycopg2/issues/420
[1] https://code.djangoproject.com/ticket/28201 - Saving a Char/TextField with psycopg2 2.7+ raises ValueError: A string literal cannot contain NUL (0x00) characters is unhandled
[2] https://code.djangoproject.com/ticket/28117 - loaddata raises ValueError with psycopg2 backend when data contains null bytes
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
Jani Tiainen

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/22fac845-6870-de4e-6fbe-eab247b8853a%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Are there use cases for storing null bytes in CharField/TextField?

Tim Graham-2
If CharField/TextField have a form validation error if null bytes are in the input, are users going to be able to understand that error and fix it? I'm not sure if it's a probable case, but I'm thinking of a non-technical user who copy/pastes some text that includes a null byte.

Perhaps a " strip_null_bytes" model field option that defaults to True would be reasonable. That could be passed to the form field to toggle where or not that validation happens. Actually, three possible behaviors might be needed: silently strip null bytes, allow null bytes (an invalid option when using PostgreSQL), prohibit null bytes.

On Tuesday, May 16, 2017 at 5:11:38 AM UTC-4, Jani Tiainen wrote:

Hi,

I would guess that one could use null byte to denote "empty field" in Oracle for example. (I recall seeing such a convention in one of our non-django apps). And that's to overcome limitation that Oracle doesn't have real concept of empty string so we stored single null byte to mark that.



On 15.05.2017 18:54, Tim Graham wrote:
Does anyone know of a use case for using null bytes in CharField/TextField?

psycopg2 2.7+ raises ValueError("A string literal cannot contain NUL (0x00) characters.") when trying to save null bytes [0] and this exception is unhandled in Django which allow malicious form submissions to crash [1]. With psycopg2 < 2.7, there is no exception and null bytes are silently truncated by PostgreSQL. Other databases that I tested (SQLite, MySQL, Oracle) allow saving null bytes. This creates possible cross-database compatibility problems when moving data from those databases to PostgreSQL, e.g.[2].

I propose to have CharField and TextField strip null bytes from the value either a) only on PostgreSQL or b) on all databases. Please indicate your preference or suggest another solution.

[0] <a href="https://github.com/psycopg/psycopg2/issues/420" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fpsycopg%2Fpsycopg2%2Fissues%2F420\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEVW66v1FcxVK4nWLzP_JWtUCzyWw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fpsycopg%2Fpsycopg2%2Fissues%2F420\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEVW66v1FcxVK4nWLzP_JWtUCzyWw&#39;;return true;">https://github.com/psycopg/psycopg2/issues/420
[1] <a href="https://code.djangoproject.com/ticket/28201" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F28201\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEICgeGqJr3UHokezoR2BxE_2FVGQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F28201\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEICgeGqJr3UHokezoR2BxE_2FVGQ&#39;;return true;">https://code.djangoproject.com/ticket/28201 - Saving a Char/TextField with psycopg2 2.7+ raises ValueError: A string literal cannot contain NUL (0x00) characters is unhandled
[2] <a href="https://code.djangoproject.com/ticket/28117" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F28117\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFuvBnzg9lmjUFQOeP2sPZt8LKTMA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F28117\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFuvBnzg9lmjUFQOeP2sPZt8LKTMA&#39;;return true;">https://code.djangoproject.com/ticket/28117 - loaddata raises ValueError with psycopg2 backend when data contains null bytes
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="pVOvNw5CAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-develop...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="pVOvNw5CAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-d...@googlegroups.com.
Visit this group at <a href="https://groups.google.com/group/django-developers" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

-- 
Jani Tiainen

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/7226ab56-ff69-415c-9955-82e8d62cdd60%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...