Quantcast

Adding a database-agnostic JSONField into Django

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Adding a database-agnostic JSONField into Django

Raphael Hertzog
Hello,

in almost all projects I work on, I end up using a JSONField. Since
I value being able to run with any database, I'm not relying on
django.contrib.postgres.fields.JSONField. So I have been using
pypi's django-jsonfield maintained by Matthew Schinckel:
https://bitbucket.org/schinckel/django-jsonfield
(I have also packaged this for Debian)

I have recently discovered pypi's "jsonfield" maintained by Brad Jasper:
https://github.com/bradjasper/django-jsonfield

Both projects are very similar (and use the same python package name) and
both projects are actually looking for a new maintainer... since I rely on
something like this, I would be willing to try to merge the best of both
modules into a possible django.contrib.jsonfield or directly into the core.

We could use this opportunity to let the newly-integrated field use
DjangoJSONEncoder by default (see recent discussion about this) and
django.contrib.postgres could register its additionals lookups into the
generic field (assuming we use "jsonb" as underlying type for postgresql).

What do you think of this?

If inclusion into Django is not desired, then maybe we could aim to
at least merge both of those projects in a single "blessed" third-party
module that could be maintained in
https://github.com/django/django-jsonfield?

Cheers,
--
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
http://debian-handbook.info/get/

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20160623095656.GB22598%40home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Adding a database-agnostic JSONField into Django

Yoong Kang Lim

I would also love to see this in Django. Although I mostly use Postgres, it would be nice not to be locked into a single database vendor if I were to use a JSONField.

On 23 Jun 2016 7:57 PM, "Raphael Hertzog" <[hidden email]> wrote:
Hello,

in almost all projects I work on, I end up using a JSONField. Since
I value being able to run with any database, I'm not relying on
django.contrib.postgres.fields.JSONField. So I have been using
pypi's django-jsonfield maintained by Matthew Schinckel:
https://bitbucket.org/schinckel/django-jsonfield
(I have also packaged this for Debian)

I have recently discovered pypi's "jsonfield" maintained by Brad Jasper:
https://github.com/bradjasper/django-jsonfield

Both projects are very similar (and use the same python package name) and
both projects are actually looking for a new maintainer... since I rely on
something like this, I would be willing to try to merge the best of both
modules into a possible django.contrib.jsonfield or directly into the core.

We could use this opportunity to let the newly-integrated field use
DjangoJSONEncoder by default (see recent discussion about this) and
django.contrib.postgres could register its additionals lookups into the
generic field (assuming we use "jsonb" as underlying type for postgresql).

What do you think of this?

If inclusion into Django is not desired, then maybe we could aim to
at least merge both of those projects in a single "blessed" third-party
module that could be maintained in
https://github.com/django/django-jsonfield?

Cheers,
--
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
http://debian-handbook.info/get/

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20160623095656.GB22598%40home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAKL8yL63BqvXL_MO-F9NmsnMJ%3DZq58WkdvUPCJGJFwv%2Br61m%3Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Adding a database-agnostic JSONField into Django

Nick Sarbicki
Same boat for me.

I constantly need the JSON field but can't always rely on postgres.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAGuvt930QvCKazawJbH_OpHwqSpSw74TtgEKR1Tpp2Yh23WLuA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Adding a database-agnostic JSONField into Django

tedjohanssondeveloper
In reply to this post by Raphael Hertzog
1 up on this one, would be really useful for me as well!

On Thursday, 23 June 2016 10:57:07 UTC+1, Raphael Hertzog wrote:
Hello,

in almost all projects I work on, I end up using a JSONField. Since
I value being able to run with any database, I'm not relying on
django.contrib.postgres.fields.JSONField. So I have been using
pypi's django-jsonfield maintained by Matthew Schinckel:
<a href="https://bitbucket.org/schinckel/django-jsonfield" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbitbucket.org%2Fschinckel%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHijk3wPgkhqM_E-DL0nn_2W3VZiw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbitbucket.org%2Fschinckel%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHijk3wPgkhqM_E-DL0nn_2W3VZiw&#39;;return true;">https://bitbucket.org/schinckel/django-jsonfield
(I have also packaged this for Debian)

I have recently discovered pypi's "jsonfield" maintained by Brad Jasper:
<a href="https://github.com/bradjasper/django-jsonfield" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fbradjasper%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHJVwrrphjX_RizlRAZTrFNMnzztg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fbradjasper%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHJVwrrphjX_RizlRAZTrFNMnzztg&#39;;return true;">https://github.com/bradjasper/django-jsonfield

Both projects are very similar (and use the same python package name) and
both projects are actually looking for a new maintainer... since I rely on
something like this, I would be willing to try to merge the best of both
modules into a possible django.contrib.jsonfield or directly into the core.

We could use this opportunity to let the newly-integrated field use
DjangoJSONEncoder by default (see recent discussion about this) and
django.contrib.postgres could register its additionals lookups into the
generic field (assuming we use "jsonb" as underlying type for postgresql).

What do you think of this?

If inclusion into Django is not desired, then maybe we could aim to
at least merge both of those projects in a single "blessed" third-party
module that could be maintained in
<a href="https://github.com/django/django-jsonfield" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHyd3tZimha4picFB4cbWhmDQaOjg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHyd3tZimha4picFB4cbWhmDQaOjg&#39;;return true;">https://github.com/django/django-jsonfield?

Cheers,
--
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ <a href="http://debian-handbook.info/get/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fdebian-handbook.info%2Fget%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFGmQYdEp9R5BqLG0Hrr8XXBfIagg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fdebian-handbook.info%2Fget%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFGmQYdEp9R5BqLG0Hrr8XXBfIagg&#39;;return true;">http://debian-handbook.info/get/

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/517f7674-b5fe-4239-9df1-e1f94d135763%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Adding a database-agnostic JSONField into Django

Tim Graham-2
Making JSONField a core field was discussed in https://groups.google.com/d/topic/django-developers/sAgYOqBUvgI/discussion

Adam said, "my recent completion of its JSONField for MySQL 5.7+ is very similar to the contrib.postgres one, copying and adapting large parts of code from Marc Tamlyn's work. We all know how much everyone loves JSON these days. If anything, this could be a core field rather than a contrib one - Oracle and SQLite also have JSON capabilities now. JSON everywhere!"

Marc said, "I'm happy for JSONField to be made a core field on the condition that it's underlying support is more than a text blob on all our main databases. It sounds like this will soon be the case."

On Thursday, June 23, 2016 at 8:43:33 AM UTC-4, [hidden email] wrote:
1 up on this one, would be really useful for me as well!

On Thursday, 23 June 2016 10:57:07 UTC+1, Raphael Hertzog wrote:
Hello,

in almost all projects I work on, I end up using a JSONField. Since
I value being able to run with any database, I'm not relying on
django.contrib.postgres.fields.JSONField. So I have been using
pypi's django-jsonfield maintained by Matthew Schinckel:
<a href="https://bitbucket.org/schinckel/django-jsonfield" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbitbucket.org%2Fschinckel%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHijk3wPgkhqM_E-DL0nn_2W3VZiw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbitbucket.org%2Fschinckel%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHijk3wPgkhqM_E-DL0nn_2W3VZiw&#39;;return true;">https://bitbucket.org/schinckel/django-jsonfield
(I have also packaged this for Debian)

I have recently discovered pypi's "jsonfield" maintained by Brad Jasper:
<a href="https://github.com/bradjasper/django-jsonfield" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fbradjasper%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHJVwrrphjX_RizlRAZTrFNMnzztg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fbradjasper%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHJVwrrphjX_RizlRAZTrFNMnzztg&#39;;return true;">https://github.com/bradjasper/django-jsonfield

Both projects are very similar (and use the same python package name) and
both projects are actually looking for a new maintainer... since I rely on
something like this, I would be willing to try to merge the best of both
modules into a possible django.contrib.jsonfield or directly into the core.

We could use this opportunity to let the newly-integrated field use
DjangoJSONEncoder by default (see recent discussion about this) and
django.contrib.postgres could register its additionals lookups into the
generic field (assuming we use "jsonb" as underlying type for postgresql).

What do you think of this?

If inclusion into Django is not desired, then maybe we could aim to
at least merge both of those projects in a single "blessed" third-party
module that could be maintained in
<a href="https://github.com/django/django-jsonfield" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHyd3tZimha4picFB4cbWhmDQaOjg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHyd3tZimha4picFB4cbWhmDQaOjg&#39;;return true;">https://github.com/django/django-jsonfield?

Cheers,
--
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ <a href="http://debian-handbook.info/get/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fdebian-handbook.info%2Fget%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFGmQYdEp9R5BqLG0Hrr8XXBfIagg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fdebian-handbook.info%2Fget%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFGmQYdEp9R5BqLG0Hrr8XXBfIagg&#39;;return true;">http://debian-handbook.info/get/

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/889b5dba-d657-4b34-a878-028033956901%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Adding a database-agnostic JSONField into Django

Aymeric Augustin
Hello,

On 23 Jun 2016, at 15:40, Tim Graham <[hidden email]> wrote:
> Marc said, "I'm happy for JSONField to be made a core field on the condition that it's underlying support is more than a text blob on all our main databases. It sounds like this will soon be the case.”


In order to discourage people from using JSONField on databases where it cannot be implemented efficiently i.e. not with a text blob, the best solution may be to add it to Django and raise a warning when it’s misused.

I’m not sure we can do that with a check, though, as we can’t introspect database routing logic (without adding a disproportionate amount of complexity). It would have to be a runtime warning.

--
Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/45280562-6205-415F-B603-60837922D13F%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Adding a database-agnostic JSONField into Django

Brad Jasper
Current maintainer of https://github.com/bradjasper/django-jsonfield here and I would definitely support this.

-Brad

On Thu, Jun 23, 2016 at 9:28 AM, Aymeric Augustin <[hidden email]> wrote:
Hello,

On 23 Jun 2016, at 15:40, Tim Graham <[hidden email]> wrote:
> Marc said, "I'm happy for JSONField to be made a core field on the condition that it's underlying support is more than a text blob on all our main databases. It sounds like this will soon be the case.”


In order to discourage people from using JSONField on databases where it cannot be implemented efficiently i.e. not with a text blob, the best solution may be to add it to Django and raise a warning when it’s misused.

I’m not sure we can do that with a check, though, as we can’t introspect database routing logic (without adding a disproportionate amount of complexity). It would have to be a runtime warning.

--
Aymeric.


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAAmjBRJKDDT%2BAdDhbGqATChS%2BF6POK9X0fcQmQFNq33Zc7c29A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Adding a database-agnostic JSONField into Django

Raphael Hertzog
In reply to this post by Aymeric Augustin
Hi,

Le jeudi 23 juin 2016, Aymeric Augustin a écrit :
> On 23 Jun 2016, at 15:40, Tim Graham <[hidden email]> wrote:
> > Marc said, "I'm happy for JSONField to be made a core field on the
> > condition that it's underlying support is more than a text blob on all
> > our main databases. It sounds like this will soon be the case.”
>
> In order to discourage people from using JSONField on databases where it
> cannot be implemented efficiently i.e. not with a text blob, the best
> solution may be to add it to Django and raise a warning when it’s
> misused.

Why would you want to discourage people? In the recent thread "Extending
JSONField serialization", you said yourself that the use-case for a
JSONField is clearly to store and retrieve data and not to make queries on
it with smart lookups.

So while I would certainly want the generic JSONField to use the most
appropriate field type for each database, I don't think we have to wait
until all major databases have the required support. And we should
certainly not throw warnings if you use it on a database that doesn't have
any special JSON support since that is one of the selling point of this
generic module!

The generic JSONField should define the appropriate interface and have
all the logic to easily customize the behaviour for each database, but
it's also fine if it starts with using plain TEXT fields on everything
except MySQL/Postgres (which are the two databases where there are
pre-existing JSONField implementation apparently).

What would be the next logical step if I want to work on this?

Is there a need for a DEP to discuss the interface provided by this field?

Cheers,
--
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
http://debian-handbook.info/get/

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20160624070136.GA22662%40home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Adding a database-agnostic JSONField into Django

Aymeric Augustin
On 24 Jun 2016, at 09:01, Raphael Hertzog <[hidden email]> wrote:

> Why would you want to discourage people?

Well, this seemed to be the latest decision, but I don’t care much myself.

> What would be the next logical step if I want to work on this?
>

> Is there a need for a DEP to discuss the interface provided by this field?


Given the relatively isolated scope of this feature, I think that reaching
consensus in this thread would suffice. (I don’t think we’re quite there yet.)

A DEP could help, though, so if you’d like to write a short one, please go
ahead!

--
Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/C4DA8EB2-3554-4CAE-B655-4669AE6C645B%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Adding a database-agnostic JSONField into Django

Shai Berger
In reply to this post by Raphael Hertzog
On Friday 24 June 2016 10:01:36 Raphael Hertzog wrote:

> Hi,
>
> Le jeudi 23 juin 2016, Aymeric Augustin a écrit :
> > On 23 Jun 2016, at 15:40, Tim Graham <[hidden email]> wrote:
> > > Marc said, "I'm happy for JSONField to be made a core field on the
> > > condition that it's underlying support is more than a text blob on all
> > > our main databases. It sounds like this will soon be the case.”
> >
> > In order to discourage people from using JSONField on databases where it
> > cannot be implemented efficiently i.e. not with a text blob, the best
> > solution may be to add it to Django and raise a warning when it’s
> > misused.
>
> Why would you want to discourage people? In the recent thread "Extending
> JSONField serialization", you said yourself that the use-case for a
> JSONField is clearly to store and retrieve data and not to make queries on
> it with smart lookups.
>

So, what you're suggesting is actually a thin wrapper around TextField, which
deals with the JSON encoding?

> Is there a need for a DEP to discuss the interface provided by this field?

I'm not sure the scope justifies a DEP, but some discussion of the API should
happen. Personally, I'm not yet convinced that -- assuming indeed we are
talking about a thinly-wrapped TextField -- such a field needs to have a
canonical implementation (one declared so by the Django project).

Shai.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Adding a database-agnostic JSONField into Django

Raphael Hertzog
In reply to this post by Aymeric Augustin
Le vendredi 24 juin 2016, Aymeric Augustin a écrit :
> On 24 Jun 2016, at 09:01, Raphael Hertzog <[hidden email]> wrote:
> > Why would you want to discourage people?
> Well, this seemed to be the latest decision, but I don’t care much myself.

AFAIK only Marc Tamlyn insisted on such a requirement to have native JSON
support on all main databases.

While this is nice and should be encouraged, it's also the kind of
requirement which discourages people from working on it.

We have working codebase(s) for a generic JSONField. I would like to pick
one, clean it up, strip it down and make it easy to extend so that native
JSON support for more database can be easily added while still keeping
the same user facing API.

The non-portable parts (lookups mainly?) should ideally be enabled via
hooks that applications like django.contrib.postgres could rely on.

> Given the relatively isolated scope of this feature, I think that reaching
> consensus in this thread would suffice. (I don’t think we’re quite there yet.)

Yes, few core team members shared their opinions so far, but we clearly have
user interest for this feature.

> A DEP could help, though, so if you’d like to write a short one, please go
> ahead!

Well, if there's consensus I'd rather spend time on code to be reviewed, but
if others core team members would rather discuss the design at at more
abstract level before deciding whether it's a good move, then I can draft a DEP.

Cheers,
--
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
http://debian-handbook.info/get/

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20160624102639.GA21727%40home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Adding a database-agnostic JSONField into Django

Raphael Hertzog
In reply to this post by Shai Berger
Hi,

Le vendredi 24 juin 2016, Shai Berger a écrit :
> > Why would you want to discourage people? In the recent thread "Extending
> > JSONField serialization", you said yourself that the use-case for a
> > JSONField is clearly to store and retrieve data and not to make queries on
> > it with smart lookups.
>
> So, what you're suggesting is actually a thin wrapper around TextField, which
> deals with the JSON encoding?

That's the simplest answer that fullfills this requirement but I'm not
suggesting that.

I believe that a "core" JSONField should supersede all the existing
implementations:
- the external implementations which are close to your description
  (but got multiple related features over the years, not all of which are
  worth keeping)
- the django.contrib.postgres one
- the django-mysql one (that I wasn't aware of until Tim pointed out
  the older thread on this topic)

So, in my view, it builds on the simplest implementation while relying
on the native JSON database type when they are available and it should
be easy to extend for the non-portable JSON-related features that each
database might want to make available (special lookups, etc.).

Even with a "thin wrapper", as soon as you rely on the native database types,
you quickly get into non-trivial integrations problems:
- with migrations from TextField to JSONField and vice-versa
  ex: https://bitbucket.org/schinckel/django-jsonfield/issues/54/migrating-from-textfield-to-jsonfield-with
- with configuration requirements of the database connector that might have an impact
  on other parts of Django
  example where pyscopg does under the hood JSON conversion for us:
  https://bitbucket.org/schinckel/django-jsonfield/pull-requests/32/make-from_db_value-compatible-with

On top of the model field, there would also be a FormField validating the
JSON input and a widget to present the JSON in a human-readable (indented)
form.

Among the other features that would be interesting to have is the
possibility to swap in a more advanced serializer (which would be able
to deal with more types of objects than the default one).

> I'm not sure the scope justifies a DEP, but some discussion of the API should
> happen. Personally, I'm not yet convinced that -- assuming indeed we are
> talking about a thinly-wrapped TextField -- such a field needs to have a
> canonical implementation (one declared so by the Django project).

Did my answer clear your doubts?

Cheers,
--
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
http://debian-handbook.info/get/

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20160624114954.GB21727%40home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Adding a database-agnostic JSONField into Django

Cristiano Coelho
In reply to this post by Raphael Hertzog
I would like it. I honestly one use JsonField to store json data, not really to query it, and having postgres store it a very efficient way is a very nice plus compared to plain text storage. Then the postgres features to query json data are godlike but I wouldn't mind trading something to allow current json field work with every database, mostly so I can still develop on SQLite while running postgres on production.

El jueves, 23 de junio de 2016, 6:57:07 (UTC-3), Raphael Hertzog escribió:
Hello,

in almost all projects I work on, I end up using a JSONField. Since
I value being able to run with any database, I'm not relying on
django.contrib.postgres.fields.JSONField. So I have been using
pypi's django-jsonfield maintained by Matthew Schinckel:
<a href="https://bitbucket.org/schinckel/django-jsonfield" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbitbucket.org%2Fschinckel%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHijk3wPgkhqM_E-DL0nn_2W3VZiw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbitbucket.org%2Fschinckel%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHijk3wPgkhqM_E-DL0nn_2W3VZiw&#39;;return true;">https://bitbucket.org/schinckel/django-jsonfield
(I have also packaged this for Debian)

I have recently discovered pypi's "jsonfield" maintained by Brad Jasper:
<a href="https://github.com/bradjasper/django-jsonfield" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fbradjasper%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHJVwrrphjX_RizlRAZTrFNMnzztg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fbradjasper%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHJVwrrphjX_RizlRAZTrFNMnzztg&#39;;return true;">https://github.com/bradjasper/django-jsonfield

Both projects are very similar (and use the same python package name) and
both projects are actually looking for a new maintainer... since I rely on
something like this, I would be willing to try to merge the best of both
modules into a possible django.contrib.jsonfield or directly into the core.

We could use this opportunity to let the newly-integrated field use
DjangoJSONEncoder by default (see recent discussion about this) and
django.contrib.postgres could register its additionals lookups into the
generic field (assuming we use "jsonb" as underlying type for postgresql).

What do you think of this?

If inclusion into Django is not desired, then maybe we could aim to
at least merge both of those projects in a single "blessed" third-party
module that could be maintained in
<a href="https://github.com/django/django-jsonfield" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHyd3tZimha4picFB4cbWhmDQaOjg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHyd3tZimha4picFB4cbWhmDQaOjg&#39;;return true;">https://github.com/django/django-jsonfield?

Cheers,
--
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ <a href="http://debian-handbook.info/get/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fdebian-handbook.info%2Fget%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFGmQYdEp9R5BqLG0Hrr8XXBfIagg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fdebian-handbook.info%2Fget%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFGmQYdEp9R5BqLG0Hrr8XXBfIagg&#39;;return true;">http://debian-handbook.info/get/

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/2f10fa0f-4fe0-456b-90b0-925e273b2bad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Adding a database-agnostic JSONField into Django

Shai Berger
In reply to this post by Raphael Hertzog
On Friday 24 June 2016 14:49:54 Raphael Hertzog wrote:
>
> Did my answer clear your doubts?
>

It moves my opinion of the feature from -0.5 "are we sure we need this" to -0
"I won't stand in your way".

Shai.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Adding a database-agnostic JSONField into Django

Sérgio Basto
In reply to this post by Raphael Hertzog
Yeah I got similar problems , I'm working with django 1.11 and rest framework I wrote [1]

we got about 5 options for JSON Fields

https://pypi.python.org/pypi/django-jsonfield 
https://pypi.python.org/pypi/jsonfield
from rest_framework.serializers import JSONField
from django.contrib.postgres.fields import JSONField
https://pypi.python.org/pypi/django-mysql

only django-jsonfield (of bitbucket) serialize json into on dictionary , the others put me json fields in a string which is not an option ...
 

but in this thread show some integration problems with django-jsonfield , what should I use to have Mysql JSON fields ?
how do the serialization ? , I need work the content of the field .

Thanks

[1]
https://github.com/encode/django-rest-framework/issues/5111


On Thursday, June 23, 2016 at 10:57:07 AM UTC+1, Raphael Hertzog wrote:
Hello,

in almost all projects I work on, I end up using a JSONField. Since
I value being able to run with any database, I'm not relying on
django.contrib.postgres.fields.JSONField. So I have been using
pypi's django-jsonfield maintained by Matthew Schinckel:
<a href="https://bitbucket.org/schinckel/django-jsonfield" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbitbucket.org%2Fschinckel%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHijk3wPgkhqM_E-DL0nn_2W3VZiw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbitbucket.org%2Fschinckel%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHijk3wPgkhqM_E-DL0nn_2W3VZiw&#39;;return true;">https://bitbucket.org/schinckel/django-jsonfield
(I have also packaged this for Debian)

I have recently discovered pypi's "jsonfield" maintained by Brad Jasper:
<a href="https://github.com/bradjasper/django-jsonfield" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fbradjasper%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHJVwrrphjX_RizlRAZTrFNMnzztg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fbradjasper%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHJVwrrphjX_RizlRAZTrFNMnzztg&#39;;return true;">https://github.com/bradjasper/django-jsonfield

Both projects are very similar (and use the same python package name) and
both projects are actually looking for a new maintainer... since I rely on
something like this, I would be willing to try to merge the best of both
modules into a possible django.contrib.jsonfield or directly into the core.

We could use this opportunity to let the newly-integrated field use
DjangoJSONEncoder by default (see recent discussion about this) and
django.contrib.postgres could register its additionals lookups into the
generic field (assuming we use "jsonb" as underlying type for postgresql).

What do you think of this?

If inclusion into Django is not desired, then maybe we could aim to
at least merge both of those projects in a single "blessed" third-party
module that could be maintained in
<a href="https://github.com/django/django-jsonfield" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHyd3tZimha4picFB4cbWhmDQaOjg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango-jsonfield\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHyd3tZimha4picFB4cbWhmDQaOjg&#39;;return true;">https://github.com/django/django-jsonfield?

Cheers,
--
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ <a href="http://debian-handbook.info/get/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fdebian-handbook.info%2Fget%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFGmQYdEp9R5BqLG0Hrr8XXBfIagg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fdebian-handbook.info%2Fget%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFGmQYdEp9R5BqLG0Hrr8XXBfIagg&#39;;return true;">http://debian-handbook.info/get/

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/43d96948-bce3-4317-8161-e85c1eb50ded%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...