Moving database backends out of the core

classic Classic list List threaded Threaded
43 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Moving database backends out of the core

Michael Manfre
Full disclosure, I maintain django-mssql and am biased toward having all database backends treated as if they were 3rd party backends.

The current discussion about "Switch to database-level autocommit" (http://bit.ly/ZlVERI) reminded me of a past discussion regarding moving the database backends out of the core. I don't remember exactly where I heard the discussion, but I'd like to revive it. My proposal is to move all database backends out of Django, except for an sqlite reference implementation.

Django has already done a similar transition with django.contrib.localflavor, which provides us with some of the basic answers about whether this sort of thing is possible and a rough guide about how to make it happen. 


Benefits
======
  - One of the biggest benefits of this change would be that database backends will no longer be bound to Django's release cycle and Django would not need to make a dot release for a database specific issue. 

  - This will result in Django having a more flexible/robust database API. Any of the existing complexity imposed on the core by a specific database backend could be abstracted out to the API. 

  - In some instances this will simplify the core's code by pushing the complexity in to the specific database backend.

Concerns
=======

These are the current issues/concerns faced by 3rd party database backends.

  - All backends would need to support multiple versions of Django, or at least properly document which versions of Django the backend works with

  - People will need to (pip) install the database backend that they want for each project

  - Less popular database backends may get left behind

  - Each project will have its own tests and will also need to run against the entire Django test suite

Migration Path:
===========

I believe this can be accomplished without needing a multi-version deprecation cycle because unlike localflavors, there should be very little, if any, user code that imports a specific database backend directly. The only instances I can think of are the DATABSE_ENGINE path and non standard features that I've added to django-mssql; custom field types, non-standard aggregates, etc. If a multi-version deprecation cycle is needed, it will be possible to fix the import paths as localflavors.

Django 1.6 or 1.7
  - Create separate github project for each database backend and move code
  - Abstract out any backend specific code in the ORM by extending the database API
  - Move backend specific tests to appropriate project

Regards,
Michael Manfre

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Florian Apolloner
Hi,

On Tuesday, March 5, 2013 8:35:07 PM UTC+1, Michael Manfre wrote:
The current discussion about "Switch to database-level autocommit" (http://bit.ly/ZlVERI) reminded me of a past discussion regarding moving the database backends out of the core. I don't remember exactly where I heard the discussion, but I'd like to revive it. My proposal is to move all database backends out of Django, except for an sqlite reference implementation.

While I agree that moving database adapters out of core has some merit, I don't think that having sqlite as a reference implementation is a good idea: For one some features are somewhat hacky in sqlite (and people tend to copy from reference implementations, so it should be as clean as possible) and it's lack of data validation makes it imo a nogo. The reference implementation should imo also have strong support for GIS, which is somewhat okay on sqlite but quite hard to install. So if we were to do that I'd either vote for postgres or supporting postgres and sqlite inside of core (the later solely for fast tests).

Cheers,
Florian

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Shai Berger
In reply to this post by Michael Manfre
Hi,

On Tuesday 05 March 2013, Michael Manfre wrote:
> Full disclosure, I maintain django-mssql and am biased toward having all
> database backends treated as if they were 3rd party backends.
>

In recent years, I have been the main contributor to South's MSSQL and Oracle
backends. I am biased towards having MSSQL treated as an equal to the database
systems supported in core, but also towards support of backend-specific low-
level code in user apps. Also, I am a Linux user -- I use django-pyodbc for my
work on South, not django-mssql.

>
> Benefits
> ======
>   - One of the biggest benefits of this change would be that database
> backends will no longer be bound to Django's release cycle and Django would
> not need to make a dot release for a database specific issue.
>
I think you are missing something crucial: Django depends on its commonly-used
backends. A critical problem with the Postgresql or MySql backend is a
critical problem for the Django project, regardless of separation to
repositories. And where the dependency is weaker, being in core was not a
guarantee for fixes -- Django carried significant Oracle problems for quite
long, and they were just not deemed crucial.

So, IMO, you can't escape the need to have a set of "blessed" backends, which
core is committed to their correct functioning, and thus need to be, at least
to some degree, in sync with the Django main project.

>   - This will result in Django having a more flexible/robust database API.
> Any of the existing complexity imposed on the core by a specific database
> backend could be abstracted out to the API.
>
>   - In some instances this will simplify the core's code by pushing the
> complexity in to the specific database backend.
>

These benefits are, IMO, orthogonal to the separation you suggest. I can't
speak for core, but I doubt that a proposed change to make the database API
more robust and flexible will be rejected just because it is originally
motivated by a 3rd-party, rather than core, backend.

> Concerns
> =======
>
>   - Less popular database backends may get left behind
>

I am worried that the result of such change will not be that MSSQL is treated
as well as Postgres is today, but that Oracle is treated as bad as MSSQL is
today.

>
> Migration Path:
> ===========
>
> I believe this can be accomplished without needing a multi-version
> deprecation cycle because unlike localflavors, there should be very little,
> if any, user code that imports a specific database backend directly.

South certainly does, I'm sure it's not the only one.

Shai.

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Florian Apolloner
Hi Shai,

On Tuesday, March 5, 2013 10:32:29 PM UTC+1, Shai Berger wrote:
In recent years, I have been the main contributor to South's MSSQL and Oracle
backends. I am biased towards having MSSQL treated as an equal to the database
systems supported in core, but also towards support of backend-specific low-
level code in user apps. Also, I am a Linux user -- I use django-pyodbc for my
work on South, not django-mssql.

I am obviously biased against postgres as my previous post indicated, but regardless of that I think that MSSQL should stay outside of core. No core-developer I know actually uses Windows as base OS and to my knowledge no one uses MSSQL. This would put the expected support for South below what we have for Oracle currently.

And where the dependency is weaker, being in core was not a
guarantee for fixes -- Django carried significant Oracle problems for quite
long, and they were just not deemed crucial.

Now that we test on Oracle once again I am committed to have a full passing testsuite with 1.6 (whatever that means, since I don't use Oracle on a daily base I can't say what a running testsuite actually means for a project).

I am worried that the result of such change will not be that MSSQL is treated
as well as Postgres is today, but that Oracle is treated as bad as MSSQL is
today.

That might as well be true, if Oracle is outside of core I personally wouldn't put much effort in fixing stuff there (this obviously might change depending on whether I use Oracle at work or not).

Cheers,
Florian

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Florian Apolloner


On Tuesday, March 5, 2013 11:01:01 PM UTC+1, Florian Apolloner wrote:
I am obviously biased against postgres as my previous post indicated, but regardless of that I think that MSSQL should stay outside of core. No core-developer I know actually uses Windows as base OS and to my knowledge no one uses MSSQL. This would put the expected support for South below what we have for Oracle currently.

s/South/MSSQL sry

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Jonathan Slenders-2
In reply to this post by Michael Manfre


Le mardi 5 mars 2013 20:35:07 UTC+1, Michael Manfre a écrit :
Full disclosure, I maintain django-mssql and am biased toward having all database backends treated as if they were 3rd party backends.

The question is, how monolithic do we want to have Django?

If we would move database backends out of core. Why not move everything in django.db away from core? There are other ORMs for Django. And the same is true for the templating system.

For me, personally, it's really a big +1 for splitting Django up in several logical parts, and for removing the global state. (So that we could create a Django *instance*, and add a postgres *instance* to django ourself.)

I want to to see something like this:

d = Django(
        database_backend=Postgres(postgres_settings),
        urls=our_url_patters,
        templates=TemplateSystem(template_settings),
        midldeware=...,
       )

No disadvantages I think. But it's not an easy migration path.

Cheers,
Jonathan

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Andre Terra
In reply to this post by Florian Apolloner
Hi Florian,

On Tue, Mar 5, 2013 at 7:01 PM, Florian Apolloner <[hidden email]> wrote:
I am obviously biased against postgres as my previous post indicated, but regardless of that I think that MSSQL should stay outside of core. No core-developer I know actually uses Windows as base OS and to my knowledge no one uses MSSQL. This would put the expected support for South below what we have for Oracle currently.

If I may weigh in on the matter as an outsider, if we consider "The Django Project" as a "business", insofar as it aims to have as many users and be as ubiquitous as possible, there's considerable value in having MSSQL included in core. If it were up to me, I would develop on Linux using postgres and host everything on Heroku, but at work I'm restricted to corporate rules, MS SQL or Oracle, and Windows. 

I have a strong feeling that by embracing Windows as a first-class platform for developing and running Django, we're enabling current and future users to easily integrate the framework in existent business infrastructure, which can only be seen as a good thing for the project.


Cheers,
AT

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Aymeric Augustin
In reply to this post by Michael Manfre
On 5 mars 2013, at 20:35, Michael Manfre <[hidden email]> wrote:

  - Less popular database backends may get left behind

To be honest, I wouldn't make any effort to keep the MySQL and Oracle backends at feature-parity with PostgreSQL and SQLite if they weren't in core.

-- 
Aymeric.



--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Florian Apolloner
In reply to this post by Andre Terra
Hi Andre,

On Tuesday, March 5, 2013 11:39:29 PM UTC+1, Andre Terra wrote:
but at work I'm restricted to corporate rules, MS SQL or Oracle, and Windows. 

Right, that's probably one of the reasons why Oracle is in core (aside from the fact that we were completely monolithic at that time).

I have a strong feeling that by embracing Windows as a first-class platform for developing and running Django, we're enabling current and future users to easily integrate the framework in existent business infrastructure, which can only be seen as a good thing for the project.

You are obviously right that it would help, but my point about not having any core developers in that area still stands and makes it hard to support something like that. FWIW, I think that supporting Windows would be far easier than supporting MSSQL. Looking at the bug tracker we have a few bugs for windows (some are false positives): https://code.djangoproject.com/query?status=assigned&status=new&description=~windows&col=id&col=summary&col=status&col=owner&col=type&col=component&order=priority

Cheers,
Florian

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Marc Tamlyn
Seems to me that if database backends were separate from Django then the postgres backend would get a long way ahead of the others as much as the other backends would get behind - it's bound to attract the most work and be adding custom fields like hstore, arrays, json... I think this would be great for most of my use cases, but then again I also have to deploy a small proportion of sites on MySQL, so it's bound to bite me.

That said, I don't know why Oracle is included and MSSQL is not - pretty much none of the core devs currently use either (or would recommend it given the choice). Cross compatibility of Django core is really important, and should ideally be as far reaching as we can achieve. I'd rather see MSSQL pulled in, and hopefully testing of it via Travis at some point than things split out, as I think it's better for Django.

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Michael Manfre
In reply to this post by Florian Apolloner


On Tuesday, March 5, 2013 3:31:59 PM UTC-5, Florian Apolloner wrote:
Hi,

While I agree that moving database adapters out of core has some merit, I don't think that having sqlite as a reference implementation is a good idea: For one some features are somewhat hacky in sqlite (and people tend to copy from reference implementations, so it should be as clean as possible)

I think I made a mistake in using the word "reference". "Included" would be a more accurate definition of its intended status. Ideally, no database backends would be in the core, but I realize that is not entirely practical because of the need to run tests. Sqlite also provides the lowest barrier for new users to get through the tutorial.

From my experience of maintaining django-mssql, I usually have to reference all of the backends depending on the issue I'm working on. I start by looking at whichever database provides the closest behavioral match and then compare it to the others. Every database is different, so it's not like copy & paste is going to happen. Database backends are also not something a large number of people will run out and create. I imagine that even with Postgres, mysql, and oracle in separate projects they will still be maintained by a collection of developers that will most likely include core devs.
 
and it's lack of data validation makes it imo a nogo.

The lack of data validation is definitely a nogo for production sites, but imo sqlite in production is also a nogo.

The reference implementation should imo also have strong support for GIS, which is somewhat okay on sqlite but quite hard to install. So if we were to do that I'd either vote for postgres or supporting postgres and sqlite inside of core (the later solely for fast tests).

. I've never tried to install GIS for sqlite, but is the difficulty due to lack of documentation or just sheer number of steps?

Regards,
Michael

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Michael Manfre
In reply to this post by Florian Apolloner


On Tuesday, March 5, 2013 5:01:01 PM UTC-5, Florian Apolloner wrote:
Hi Shai,

On Tuesday, March 5, 2013 10:32:29 PM UTC+1, Shai Berger wrote:
In recent years, I have been the main contributor to South's MSSQL and Oracle
backends. I am biased towards having MSSQL treated as an equal to the database
systems supported in core, but also towards support of backend-specific low-
level code in user apps. Also, I am a Linux user -- I use django-pyodbc for my
work on South, not django-mssql.

I am obviously biased against postgres as my previous post indicated, but regardless of that I think that MSSQL should stay outside of core. No core-developer I know actually uses Windows as base OS and to my knowledge no one uses MSSQL. This would put the expected support for South below what we have for Oracle currently.

I agree that MSSQL should stay out of the core. I believe that adding any more database in to the core would be a large step in the wrong direction.
 
And where the dependency is weaker, being in core was not a
guarantee for fixes -- Django carried significant Oracle problems for quite
long, and they were just not deemed crucial.

Now that we test on Oracle once again I am committed to have a full passing testsuite with 1.6 (whatever that means, since I don't use Oracle on a daily base I can't say what a running testsuite actually means for a project).

Is it not possible to maintain the same standard with it as a separate project? Automated tests could still easily be accomplished and the only real difference about the backend working with Django is that the releases of the projects would not be a direct blocker for each other. 3rd party backends are only able to provide eventual support. Sometimes the new behaviors are in place ahead of the Django release, sometimes it lands shortly after. A huge benefit is that if there is a bug/change in the postgres backend, it doesn't require another Django release.
 
I am worried that the result of such change will not be that MSSQL is treated
as well as Postgres is today, but that Oracle is treated as bad as MSSQL is
today.

That might as well be true, if Oracle is outside of core I personally wouldn't put much effort in fixing stuff there (this obviously might change depending on whether I use Oracle at work or not). 

Having a backend outside of core might encourage others to step up and maintain it, especially if they depend on it. I'm sure many currently have the mentality of "the cores will take care of it, so I don't have to". 

It sounds as if Oracle is already not really a supported database and will only have a passing test suite. As you stated, "whatever that means, since I don't use Oracle on a daily base I can't say what a running testsuite actually means for a project".

Regards,
Michael

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Michael Manfre
In reply to this post by Andre Terra
Hi Andre,

On Tuesday, March 5, 2013 5:39:29 PM UTC-5, Andre Terra wrote:
If I may weigh in on the matter as an outsider, if we consider "The Django Project" as a "business", insofar as it aims to have as many users and be as ubiquitous as possible, there's considerable value in having MSSQL included in core. If it were up to me, I would develop on Linux using postgres and host everything on Heroku, but at work I'm restricted to corporate rules, MS SQL or Oracle, and Windows. 

Business rules dictate Windows and MSSQL for me as well. Despite the fact that including MSSQL in the core would mean it has a better bare minimum amount of support, I'm strongly opposed to it being included. The core already has too many databases included.
 
I have a strong feeling that by embracing Windows as a first-class platform for developing and running Django, we're enabling current and future users to easily integrate the framework in existent business infrastructure, which can only be seen as a good thing for the project.

Agreed. Windows should be treated as a first-class OS, but I don't think any of the core devs use Windows as their primary OS. I only know of one core dev that even runs the Django testsuite on Windows from time to time.

Regards,
Michael

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Florian Apolloner
In reply to this post by Michael Manfre
Hi,

On Wednesday, March 6, 2013 3:32:45 PM UTC+1, Michael Manfre wrote: 
The lack of data validation is definitely a nogo for production sites, but imo sqlite in production is also a nogo.

Right, but shipping Django with a non production db might send interesting signals to endusers ;)

The reference implementation should imo also have strong support for GIS, which is somewhat okay on sqlite but quite hard to install. So if we were to do that I'd either vote for postgres or supporting postgres and sqlite inside of core (the later solely for fast tests).

. I've never tried to install GIS for sqlite, but is the difficulty due to lack of documentation or just sheer number of steps?

Well it's not to bad, we did document it after all, but it usually requires recompiling sqlite and pysqlite (most importantly you can't pull it from your distros repos).

Regards,
Florian

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Michael Manfre

On Wed, Mar 6, 2013 at 12:47 PM, Florian Apolloner <[hidden email]> wrote:
On Wednesday, March 6, 2013 3:32:45 PM UTC+1, Michael Manfre wrote: 
The lack of data validation is definitely a nogo for production sites, but imo sqlite in production is also a nogo.

Right, but shipping Django with a non production db might send interesting signals to endusers ;)

It might, but it's all about marketing. "Django, now 10% less monolithic!" 

Regards,
Michael

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Jacob Kaplan-Moss-2
In reply to this post by Marc Tamlyn
On Wed, Mar 6, 2013 at 3:22 AM, Marc Tamlyn <[hidden email]> wrote:
> I don't know why Oracle is included and MSSQL is not [...]

It's essentially because a group of people made a concerted effort to
get the Oracle backend up to snuff (around the 1.0 timeline, I think?)
and then committed to maintaining it. Lately the original people who'd
made that commitment have dropped off a bit, but there's always been
enough attention to keep it up to date.

If someone -- preferably a group -- stepped up and committed to
keeping a MSSQL backend up-to-date in core, I'd be +1 on merging it
and giving those people commit access to keep it up to date.

[This is me speaking personally, not as a BDFL. It'd have to be a
discussion, not just my fiat.]

Jacob

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Alex Ogier
It's worth mentioning that Django appears in the Python Tools for Visual Studio (it's one of the default project templates)[1]. There's recent introductory material on deploying Django applications to Windows Azure[2] and Windows Server 2008[3], though it seems the older material at least explicitly recommends MySQL and not SQL Server. I don't know if this is the kind of thing that leads to demand among Windows folks for Windows support for Django, and if any significant fraction of them want to use MSSQL with Django, but Django is certainly becoming mainstream for them.


Best,
Alex Ogier


On Thu, Mar 7, 2013 at 12:46 PM, Jacob Kaplan-Moss <[hidden email]> wrote:
On Wed, Mar 6, 2013 at 3:22 AM, Marc Tamlyn <[hidden email]> wrote:
> I don't know why Oracle is included and MSSQL is not [...]

It's essentially because a group of people made a concerted effort to
get the Oracle backend up to snuff (around the 1.0 timeline, I think?)
and then committed to maintaining it. Lately the original people who'd
made that commitment have dropped off a bit, but there's always been
enough attention to keep it up to date.

If someone -- preferably a group -- stepped up and committed to
keeping a MSSQL backend up-to-date in core, I'd be +1 on merging it
and giving those people commit access to keep it up to date.

[This is me speaking personally, not as a BDFL. It'd have to be a
discussion, not just my fiat.]

Jacob

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.



--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Erik Romijn
Hi all,

It seems to me we are mixing a few different discussions here:
1. Should Django core have as few database backends as possible?
 1a. If yes, which ones should stay in Django core?
2. What should we do, if anything, with the current situation where
   it seems difficult to guarantee the quality of the Oracle backend?
3. Should a MSSQL Django backend be included in Django core?

Regarding question 1, I feel we should keep a fair set of backends,
like the current one, included in Django core.

Yes, we could move them to separate projects. All the same arguments
apply here as why it's not necessarily good for a project to end up
in contrib. However, in theory any component of Django could be moved
out of core and be independently maintained. The whole ORM, the
templating system, the CSRF protection, and so on. I haven't seen any
reason why we should move the MySQL backend out, but not the templating
system.

Besides that, the close integration and development of all these parts
is exactly the reason I like Django. When I download a version of Django,
or upgrade, I know that all components in there will work well together.
I can run my site on any of the supported databases, and they will all
work together with the provided admin. I can build forms on the models
I build with the ORM. The forms will nicely fit in with the templates.

When I started using Django, I had looked at several alternatives which
were more modular. However, they required me to make tons of choices,
each involving numerous consequences. If I picked ORM A, I could have
databases X, Y and Z, but no admin. For the admin I had to use ORM B,
but that did not support database Z or many to many fields. And so on.
When I tried Django, it was a relief to see I could download a single
package, and everything would just work together.

If we move database backends out of core, my big concern is that
the same will happen there. This is fine for many components we use
in many Django projects - but not for something as fundamental as the
database backend. When the admin doesn't work on any of the officially
supported backends, that should be a release blocker for Django itself,
because I think it will be a blocker for the users.

Regarding question 2, Oracle support, I think a great step forward has
been made with the addition of Oracle to the Django continuous
integration setup. That alone should help us improve the consistent
quality of Oracle support.

cheers,
Erik

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Alex Ogier
Here's something I've been thinking about:

As a rule, assuming that backends are not bug-riddled and do not have needlessly shoddy performance, nearly all of the value of an ORM is in the set of frontend features it exposes to application developers. Given a certain frontend API, then the only value in iterating the backend independently from the frontend is to fix uncaught bugs, or to improve performance, there's no value in adding features.

Also as a rule, adding features to a frontend independently from the backend has little value. Without an implementation in whatever backend you are on, it only serves as a guide for future backend development, so you might as well push for backends to catch up in the same release cycle. (Or the feature doesn't require backend support, but then it doesn't hurt to reversion the backend anyways).

So basically, there's not much value in independently versioning and maintaining frontends and backends to the ORM. This is in contrast to, say, localflavor, where an improvement in Mongolian localization can immediately help every Mongolian website in every Django version. So this primary motivation for independent maintenance is not a factor in the ORM. Hence I think the core team should include as many backends as it can handle in core (where "handle" means test that they function properly for each release). Oracle had been slipping, but from what I understand it's now in the CI server and back to passing most tests.

So I see no reason to split off backends unless the maintenance burden is such that they are chronically broken in every Django release. And every reason to add MSSQL unless the maintenance burden is too high.

Best,
Alex Ogier


On Thu, Mar 7, 2013 at 4:03 PM, Erik Romijn <[hidden email]> wrote:
Hi all,

It seems to me we are mixing a few different discussions here:
1. Should Django core have as few database backends as possible?
 1a. If yes, which ones should stay in Django core?
2. What should we do, if anything, with the current situation where
   it seems difficult to guarantee the quality of the Oracle backend?
3. Should a MSSQL Django backend be included in Django core?

Regarding question 1, I feel we should keep a fair set of backends,
like the current one, included in Django core.

Yes, we could move them to separate projects. All the same arguments
apply here as why it's not necessarily good for a project to end up
in contrib. However, in theory any component of Django could be moved
out of core and be independently maintained. The whole ORM, the
templating system, the CSRF protection, and so on. I haven't seen any
reason why we should move the MySQL backend out, but not the templating
system.

Besides that, the close integration and development of all these parts
is exactly the reason I like Django. When I download a version of Django,
or upgrade, I know that all components in there will work well together.
I can run my site on any of the supported databases, and they will all
work together with the provided admin. I can build forms on the models
I build with the ORM. The forms will nicely fit in with the templates.

When I started using Django, I had looked at several alternatives which
were more modular. However, they required me to make tons of choices,
each involving numerous consequences. If I picked ORM A, I could have
databases X, Y and Z, but no admin. For the admin I had to use ORM B,
but that did not support database Z or many to many fields. And so on.
When I tried Django, it was a relief to see I could download a single
package, and everything would just work together.

If we move database backends out of core, my big concern is that
the same will happen there. This is fine for many components we use
in many Django projects - but not for something as fundamental as the
database backend. When the admin doesn't work on any of the officially
supported backends, that should be a release blocker for Django itself,
because I think it will be a blocker for the users.

Regarding question 2, Oracle support, I think a great step forward has
been made with the addition of Oracle to the Django continuous
integration setup. That alone should help us improve the consistent
quality of Oracle support.

cheers,
Erik

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.



--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Moving database backends out of the core

Russell Keith-Magee-4


On Fri, Mar 8, 2013 at 7:57 AM, Alex Ogier <[hidden email]> wrote:
Here's something I've been thinking about:

As a rule, assuming that backends are not bug-riddled and do not have needlessly shoddy performance, nearly all of the value of an ORM is in the set of frontend features it exposes to application developers. Given a certain frontend API, then the only value in iterating the backend independently from the frontend is to fix uncaught bugs, or to improve performance, there's no value in adding features.

Also as a rule, adding features to a frontend independently from the backend has little value. Without an implementation in whatever backend you are on, it only serves as a guide for future backend development, so you might as well push for backends to catch up in the same release cycle. (Or the feature doesn't require backend support, but then it doesn't hurt to reversion the backend anyways).

So basically, there's not much value in independently versioning and maintaining frontends and backends to the ORM. This is in contrast to, say, localflavor, where an improvement in Mongolian localization can immediately help every Mongolian website in every Django version. So this primary motivation for independent maintenance is not a factor in the ORM. Hence I think the core team should include as many backends as it can handle in core (where "handle" means test that they function properly for each release). Oracle had been slipping, but from what I understand it's now in the CI server and back to passing most tests.

So I see no reason to split off backends unless the maintenance burden is such that they are chronically broken in every Django release. And every reason to add MSSQL unless the maintenance burden is too high.

And that last clause is the catch.

I completely agree with Jacob's analysis of the status quo, and I agree largely with his position on having MSSQL in the core. 

I'd have no problem seeing MSSQL in the core - it's at least as high profile as Oracle, and although the demand for a MSSQL backend is restricted to a particular subset of the community, there is at least *some* demand for it to exist. Having MSSQL in core would allow us to hold our head high and say we support Microsoft platforms. Microsoft is even a member of the DSF at this point, so they're at least notionally interested in Django as a platform. 

The maintenance burden is the problem. Historically, we've already had to *remove* a MSSQL backend because of bit rot (see [1]) - this isn't a pattern I want to repeat in the future. I don't want to add a backend to core unless I'm *certain* that it will see long term maintenance.


There is, however, a possible middle ground, following the example set by Flask: we introduce to Django a list of "officially recognised" extensions. These extensions are still maintained as external projects, but the core team promise to include these projects as part of the testing regimen for a release. If the MSSQL backend were recognised like this, we would run the full Django test suite against the MSSQL backend, and if that testing revealed a bug in the test suite which was identified as being due to a change in Django's codebase, that bug would be a release blocker (Bugs in the backend itself would still be the backend's responsibility, and not release blocking on Django)

Looking outside database backends, this could also apply to high-profile projects like haystack, django-registration, and so on. This would also serve to highlight projects in our community that are 'defacto standards', or good examples of reusability, without needing to expand the core team or the size of contrib, and show that the core project is explicitly interested in the broader ecosystem.

Yours,
Russ Magee %-)

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
123