Integrate dj-database-url into Django

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Integrate dj-database-url into Django

Kenneth Reitz-3
dj-database-url gives Django developers two main things:

1. the ability to represent their database settings via a string (a-la sqlalchemy); very useful for environmnent variables and 12factor apps.
2. it will automatically use the DATABASE_URL environment variable, if present.

I'm not sure if #2 is appropriate for Django Core (but it would be nice, as Gunicorn supports both PORT and WEB_CONCURRENCY), but I know #1 is perfect.

This plan has previously been discussed (at a conference, DjangoCon EU in Zurich, long ago) and approved by JKM, before his role at Heroku, if that is helpful information.

I think this change would vastly improve the usability of Django, and would be an excellent and simple move for the project.

Many thanks for your consideration. <3

--
Kenneth Reitz

https://code.djangoproject.com/ticket/28236

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/057ced98-4471-4939-960b-900ec39f54b7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

Florian Apolloner
Hi Kenneth,

I think there are a few things to consider before we can merge/reuse dj-database-url.

 * How are third party backends going to fit into this scheme?
 * What about other options like CACHES which very much suffers from the same problem.

Cheers,
Florian

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/e28b5354-201c-41d0-9b7d-aadeb0024200%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

Kenneth Reitz-3
I don't think the code will be able to be re-used as much as the idea/ux/functionality that it provides.

I can imagine any backend being able to convert from a url string with some methods (e.g. `detect_url` (bool return) and `parse_url` (settings dict return type).)

Perhaps the same scheme could be applied to caches as well.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/aba85896-0271-4358-aba4-c2e4cab186b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

Kenneth Reitz-3
Ah, I see what you're saying about third-party backends.

I imagine that you'd have to be able to merge a url string with a settings dict (one overrides the other). This is how usage of that functionality is manually provided in dj-database-url.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9043e51f-843d-4395-9cb1-9ef138108eb7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

Tom Forbes
In reply to this post by Kenneth Reitz-3
My two cents: connection strings/database URI's are a feature I've sorely missed in Django.

Built-in functionality to convert environment variables like DJANGO_DB_DEFAULT (or more generally DJANGO_DB_*key*) into the relevant DATABASE setting would make some deployment situations a lot simpler. Currently, unless you use dj-database-uri you have to define a bunch of ad-hoc DB_USER/DB_PASSWORD etc env variables and price the dictionary together yourself.

How does this library complex keys like OPTIONS, TEST or DEPENDENCIES?

To help support third part backends: perhaps the scheme portion of the URI could be either a relative import from django.db.backends or an absolute import to a third party library? It seems URI schemes can have dots and underscores in them, so they can be python package paths.

I.e sqlite3://xyz would resolve go django.db.backends.sqlite3, but sqlserver_ado://xyz would resolve to the third party django-mssql engine via an absolute import.

Not sure how to handle arbitrary extra settings though.

On 24 May 2017 19:05, "Kenneth Reitz" <[hidden email]> wrote:
dj-database-url gives Django developers two main things:

1. the ability to represent their database settings via a string (a-la sqlalchemy); very useful for environmnent variables and 12factor apps.
2. it will automatically use the DATABASE_URL environment variable, if present.

I'm not sure if #2 is appropriate for Django Core (but it would be nice, as Gunicorn supports both PORT and WEB_CONCURRENCY), but I know #1 is perfect.

This plan has previously been discussed (at a conference, DjangoCon EU in Zurich, long ago) and approved by JKM, before his role at Heroku, if that is helpful information.

I think this change would vastly improve the usability of Django, and would be an excellent and simple move for the project.

Many thanks for your consideration. <3

--
Kenneth Reitz

https://code.djangoproject.com/ticket/28236

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/057ced98-4471-4939-960b-900ec39f54b7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJPogT4y-E4v5giF-6ajbQ6PSpZvpAPZO0BzK%2BLT9uL1xA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

Aymeric Augustin
Hello,

I'm wondering what the exact definition of the URL format is. Is it specified somewhere? Or is it just:

[engine]://[username]:[password]@[host]:[port]/[name]

where we create arbitrary [engine] values in an ad-hoc fashion?

On 24 May 2017, at 21:21, Tom Forbes <[hidden email]> wrote:

My two cents: connection strings/database URI's are a feature I've sorely missed in Django.

Built-in functionality to convert environment variables like DJANGO_DB_DEFAULT (or more generally DJANGO_DB_*key*) into the relevant DATABASE setting would make some deployment situations a lot simpler. Currently, unless you use dj-database-uri you have to define a bunch of ad-hoc DB_USER/DB_PASSWORD etc env variables and price the dictionary together yourself.

Fully agreed. While relatively minor, it's an annoyance.

How does this library complex keys like OPTIONS, TEST or DEPENDENCIES?

I don't think it's reasonable to cram them in a URL.

dj-database-url allows passing options as extra keyword arguments. Other values should be explicitly added in the settings module, by updating the dict generated from the URL.

To help support third part backends: perhaps the scheme portion of the URI could be either a relative import from django.db.backends or an absolute import to a third party library? It seems URI schemes can have dots and underscores in them, so they can be python package paths.

I.e <a href="sqlite3://xyz" class="">sqlite3://xyz would resolve go django.db.backends.sqlite3, but sqlserver_ado://xyz would resolve to the third party django-mssql engine via an absolute import.

I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.

I'd rather specify the database engine explicitly when calling dj-database-url if it's a third-party engine. There's an open question about what to do with the [engine] part of the URL in that case. Ignoring it entirely is the easiest.

Best regards,

-- 
Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/A0870D90-75DE-496B-A838-AC7D7488A67D%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

Florian Apolloner
On Thursday, May 25, 2017 at 9:46:56 AM UTC+2, Aymeric Augustin wrote:
I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.

One possibility would be to use entrypoints in setuptools, this way 3rd party backends could specify a name which then has a fixed & verified import path.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/576a3961-6c84-4438-b434-4beaddab38b2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

Tom Forbes
In reply to this post by Aymeric Augustin
> I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.

Good point, it's not wise to enable this even without a clear threat model. Django does import the settings based on an environment variable, but it's relative and if you can use that to do anything nasty then you're most likely already through the airtight hatch (so to speak). However importing potentially global modules could be bad news.

Ignoring it is always an option, but could we not specify that the third party database provider has to be in the `INSTALLED_APPS`? That could provide some form of whitelisting so not any old module is imported. Not sure about any issues that may arise from this though.

> One possibility would be to use entrypoints in setuptools, this way 3rd party backends could specify a name which then has a fixed & verified import path.

This seems like it could get complex, and be quite unlike anything else in Django.

Perhaps just supporting this for first-party database backends is easiest?

On Thu, May 25, 2017 at 8:46 AM, Aymeric Augustin <[hidden email]> wrote:
Hello,

I'm wondering what the exact definition of the URL format is. Is it specified somewhere? Or is it just:

[engine]://[username]:[password]@[host]:[port]/[name]

where we create arbitrary [engine] values in an ad-hoc fashion?

On 24 May 2017, at 21:21, Tom Forbes <[hidden email]> wrote:

My two cents: connection strings/database URI's are a feature I've sorely missed in Django.

Built-in functionality to convert environment variables like DJANGO_DB_DEFAULT (or more generally DJANGO_DB_*key*) into the relevant DATABASE setting would make some deployment situations a lot simpler. Currently, unless you use dj-database-uri you have to define a bunch of ad-hoc DB_USER/DB_PASSWORD etc env variables and price the dictionary together yourself.

Fully agreed. While relatively minor, it's an annoyance.

How does this library complex keys like OPTIONS, TEST or DEPENDENCIES?

I don't think it's reasonable to cram them in a URL.

dj-database-url allows passing options as extra keyword arguments. Other values should be explicitly added in the settings module, by updating the dict generated from the URL.

To help support third part backends: perhaps the scheme portion of the URI could be either a relative import from django.db.backends or an absolute import to a third party library? It seems URI schemes can have dots and underscores in them, so they can be python package paths.

I.e sqlite3://xyz would resolve go django.db.backends.sqlite3, but sqlserver_ado://xyz would resolve to the third party django-mssql engine via an absolute import.

I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.

I'd rather specify the database engine explicitly when calling dj-database-url if it's a third-party engine. There's an open question about what to do with the [engine] part of the URL in that case. Ignoring it entirely is the easiest.

Best regards,

-- 
Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/A0870D90-75DE-496B-A838-AC7D7488A67D%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJO%3DGXGqV%3DeAbDzSr0_WWsDi%2Bv%2B-G7XhC%3DwnWvK1KXuukA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

Tom Forbes
Edit: DJANGO_SETTINGS_MODULE isn't relative, it will import any arbitrary module you give it. If we accept that then I think we are accepting the risk of imports via an attacker controlling environment variables whilst Django starts up?

On Sat, May 27, 2017 at 8:49 PM, Tom Forbes <[hidden email]> wrote:
> I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.

Good point, it's not wise to enable this even without a clear threat model. Django does import the settings based on an environment variable, but it's relative and if you can use that to do anything nasty then you're most likely already through the airtight hatch (so to speak). However importing potentially global modules could be bad news.

Ignoring it is always an option, but could we not specify that the third party database provider has to be in the `INSTALLED_APPS`? That could provide some form of whitelisting so not any old module is imported. Not sure about any issues that may arise from this though.

> One possibility would be to use entrypoints in setuptools, this way 3rd party backends could specify a name which then has a fixed & verified import path.

This seems like it could get complex, and be quite unlike anything else in Django.

Perhaps just supporting this for first-party database backends is easiest?

On Thu, May 25, 2017 at 8:46 AM, Aymeric Augustin <[hidden email]> wrote:
Hello,

I'm wondering what the exact definition of the URL format is. Is it specified somewhere? Or is it just:

[engine]://[username]:[password]@[host]:[port]/[name]

where we create arbitrary [engine] values in an ad-hoc fashion?

On 24 May 2017, at 21:21, Tom Forbes <[hidden email]> wrote:

My two cents: connection strings/database URI's are a feature I've sorely missed in Django.

Built-in functionality to convert environment variables like DJANGO_DB_DEFAULT (or more generally DJANGO_DB_*key*) into the relevant DATABASE setting would make some deployment situations a lot simpler. Currently, unless you use dj-database-uri you have to define a bunch of ad-hoc DB_USER/DB_PASSWORD etc env variables and price the dictionary together yourself.

Fully agreed. While relatively minor, it's an annoyance.

How does this library complex keys like OPTIONS, TEST or DEPENDENCIES?

I don't think it's reasonable to cram them in a URL.

dj-database-url allows passing options as extra keyword arguments. Other values should be explicitly added in the settings module, by updating the dict generated from the URL.

To help support third part backends: perhaps the scheme portion of the URI could be either a relative import from django.db.backends or an absolute import to a third party library? It seems URI schemes can have dots and underscores in them, so they can be python package paths.

I.e sqlite3://xyz would resolve go django.db.backends.sqlite3, but sqlserver_ado://xyz would resolve to the third party django-mssql engine via an absolute import.

I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.

I'd rather specify the database engine explicitly when calling dj-database-url if it's a third-party engine. There's an open question about what to do with the [engine] part of the URL in that case. Ignoring it entirely is the easiest.

Best regards,

-- 
Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/A0870D90-75DE-496B-A838-AC7D7488A67D%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJN80HNFnKkZQ5rFaoH5jqiEBFSfdYZpqCrw0C0E-q9f_w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

Tim Allen
I've recently been introduced to `django-environ`, a similar library that has additional features to DB connect URLs that we may want to consider: https://github.com/joke2k/django-environ

It has the same issue with third party DB engines; for example, I recently issued a PR to include `pyodbc` as an option. It has some nice newcomer friendly features, such as a cleaner way of determining BASE_URL / SITE_ROOT. Have a look at the URL for a good "before" and "after" settings file example.

Having full support for .env files would make Django projects easier to bring in line with 12-factor.

Regards,

Tim

On Saturday, May 27, 2017 at 3:52:23 PM UTC-4, Tom Forbes wrote:
Edit: DJANGO_SETTINGS_MODULE isn't relative, it will import any arbitrary module you give it. If we accept that then I think we are accepting the risk of imports via an attacker controlling environment variables whilst Django starts up?

On Sat, May 27, 2017 at 8:49 PM, Tom Forbes <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="lJV1bWT2BAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">t...@...> wrote:
> I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.

Good point, it's not wise to enable this even without a clear threat model. Django does import the settings based on an environment variable, but it's relative and if you can use that to do anything nasty then you're most likely already through the airtight hatch (so to speak). However importing potentially global modules could be bad news.

Ignoring it is always an option, but could we not specify that the third party database provider has to be in the `INSTALLED_APPS`? That could provide some form of whitelisting so not any old module is imported. Not sure about any issues that may arise from this though.

> One possibility would be to use entrypoints in setuptools, this way 3rd party backends could specify a name which then has a fixed & verified import path.

This seems like it could get complex, and be quite unlike anything else in Django.

Perhaps just supporting this for first-party database backends is easiest?

On Thu, May 25, 2017 at 8:46 AM, Aymeric Augustin <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="lJV1bWT2BAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">aymeric....@polytechnique.org> wrote:
Hello,

I'm wondering what the exact definition of the URL format is. Is it specified somewhere? Or is it just:

[engine]://[username]:[password]@[host]:[port]/[name]

where we create arbitrary [engine] values in an ad-hoc fashion?

On 24 May 2017, at 21:21, Tom Forbes <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="lJV1bWT2BAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">t...@...> wrote:

My two cents: connection strings/database URI's are a feature I've sorely missed in Django.

Built-in functionality to convert environment variables like DJANGO_DB_DEFAULT (or more generally DJANGO_DB_*key*) into the relevant DATABASE setting would make some deployment situations a lot simpler. Currently, unless you use dj-database-uri you have to define a bunch of ad-hoc DB_USER/DB_PASSWORD etc env variables and price the dictionary together yourself.

Fully agreed. While relatively minor, it's an annoyance.

How does this library complex keys like OPTIONS, TEST or DEPENDENCIES?

I don't think it's reasonable to cram them in a URL.

dj-database-url allows passing options as extra keyword arguments. Other values should be explicitly added in the settings module, by updating the dict generated from the URL.

To help support third part backends: perhaps the scheme portion of the URI could be either a relative import from django.db.backends or an absolute import to a third party library? It seems URI schemes can have dots and underscores in them, so they can be python package paths.

I.e sqlite3://xyz would resolve go django.db.backends.sqlite3, but sqlserver_ado://xyz would resolve to the third party django-mssql engine via an absolute import.

I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.

I'd rather specify the database engine explicitly when calling dj-database-url if it's a third-party engine. There's an open question about what to do with the [engine] part of the URL in that case. Ignoring it entirely is the easiest.

Best regards,

-- 
Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="lJV1bWT2BAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-develop...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="lJV1bWT2BAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-d...@googlegroups.com.
Visit this group at <a href="https://groups.google.com/group/django-developers" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/A0870D90-75DE-496B-A838-AC7D7488A67D%40polytechnique.org?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/A0870D90-75DE-496B-A838-AC7D7488A67D%40polytechnique.org?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/A0870D90-75DE-496B-A838-AC7D7488A67D%40polytechnique.org?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/django-developers/A0870D90-75DE-496B-A838-AC7D7488A67D%40polytechnique.org.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/95a17c45-7421-43e8-96ce-f9cc81bc82a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

linus
There's also https://github.com/doismellburning/django12factor - and it simplifies all the boilerplate, that is still required in django-environ, into:

import django12factor
globals().update(django12factor.factorise())

It would be quite nice if Django supported 12factor configuration natively, or at least with a single function call like this one.

Regards,
Linus

On Sunday, May 28, 2017 at 12:09:57 AM UTC+2, Tim Allen wrote:
I've recently been introduced to `django-environ`, a similar library that has additional features to DB connect URLs that we may want to consider: <a href="https://github.com/joke2k/django-environ" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjoke2k%2Fdjango-environ\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE64md95IvBjolRbUISX6LvvikEpg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjoke2k%2Fdjango-environ\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE64md95IvBjolRbUISX6LvvikEpg&#39;;return true;">https://github.com/joke2k/django-environ

It has the same issue with third party DB engines; for example, I recently issued a PR to include `pyodbc` as an option. It has some nice newcomer friendly features, such as a cleaner way of determining BASE_URL / SITE_ROOT. Have a look at the URL for a good "before" and "after" settings file example.

Having full support for .env files would make Django projects easier to bring in line with 12-factor.

Regards,

Tim

On Saturday, May 27, 2017 at 3:52:23 PM UTC-4, Tom Forbes wrote:
Edit: DJANGO_SETTINGS_MODULE isn't relative, it will import any arbitrary module you give it. If we accept that then I think we are accepting the risk of imports via an attacker controlling environment variables whilst Django starts up?

On Sat, May 27, 2017 at 8:49 PM, Tom Forbes <[hidden email]> wrote:
> I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.

Good point, it's not wise to enable this even without a clear threat model. Django does import the settings based on an environment variable, but it's relative and if you can use that to do anything nasty then you're most likely already through the airtight hatch (so to speak). However importing potentially global modules could be bad news.

Ignoring it is always an option, but could we not specify that the third party database provider has to be in the `INSTALLED_APPS`? That could provide some form of whitelisting so not any old module is imported. Not sure about any issues that may arise from this though.

> One possibility would be to use entrypoints in setuptools, this way 3rd party backends could specify a name which then has a fixed & verified import path.

This seems like it could get complex, and be quite unlike anything else in Django.

Perhaps just supporting this for first-party database backends is easiest?

On Thu, May 25, 2017 at 8:46 AM, Aymeric Augustin <[hidden email]> wrote:
Hello,

I'm wondering what the exact definition of the URL format is. Is it specified somewhere? Or is it just:

[engine]://[username]:[password]@[host]:[port]/[name]

where we create arbitrary [engine] values in an ad-hoc fashion?

On 24 May 2017, at 21:21, Tom Forbes <[hidden email]> wrote:

My two cents: connection strings/database URI's are a feature I've sorely missed in Django.

Built-in functionality to convert environment variables like DJANGO_DB_DEFAULT (or more generally DJANGO_DB_*key*) into the relevant DATABASE setting would make some deployment situations a lot simpler. Currently, unless you use dj-database-uri you have to define a bunch of ad-hoc DB_USER/DB_PASSWORD etc env variables and price the dictionary together yourself.

Fully agreed. While relatively minor, it's an annoyance.

How does this library complex keys like OPTIONS, TEST or DEPENDENCIES?

I don't think it's reasonable to cram them in a URL.

dj-database-url allows passing options as extra keyword arguments. Other values should be explicitly added in the settings module, by updating the dict generated from the URL.

To help support third part backends: perhaps the scheme portion of the URI could be either a relative import from django.db.backends or an absolute import to a third party library? It seems URI schemes can have dots and underscores in them, so they can be python package paths.

I.e sqlite3://xyz would resolve go django.db.backends.sqlite3, but sqlserver_ado://xyz would resolve to the third party django-mssql engine via an absolute import.

I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.

I'd rather specify the database engine explicitly when calling dj-database-url if it's a third-party engine. There's an open question about what to do with the [engine] part of the URL in that case. Ignoring it entirely is the easiest.

Best regards,

-- 
Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/django-developers" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/A0870D90-75DE-496B-A838-AC7D7488A67D%40polytechnique.org?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/A0870D90-75DE-496B-A838-AC7D7488A67D%40polytechnique.org?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/A0870D90-75DE-496B-A838-AC7D7488A67D%40polytechnique.org?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/django-developers/A0870D90-75DE-496B-A838-AC7D7488A67D%40polytechnique.org.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/69f0a647-42cf-4ef6-9ad7-bbe5e8fa779b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

Curtis Maloney-2
My django-classy-settings also supports env-based settings, in a
declarative manner with defaults.

What I've yet to crack (though the discussion is underway) is good
formats for structured settings like caches and databases.

Generally I use dj-database-url for my own projects :)

I like the idea of having it scan INSTALLED_APPS to allow registering
custom DB backend protocols... though I think the modern approach is
AppConfig.ready ?

--
Curtis

On 29/05/17 08:23, [hidden email] wrote:

> There's also https://github.com/doismellburning/django12factor - and it
> simplifies all the boilerplate, that is still required in
> django-environ, into:
>
> import django12factor
> globals().update(django12factor.factorise())
>
>
> It would be quite nice if Django supported 12factor configuration
> natively, or at least with a single function call like this one.
>
> Regards,
> Linus
>
> On Sunday, May 28, 2017 at 12:09:57 AM UTC+2, Tim Allen wrote:
>
>     I've recently been introduced to `django-environ`, a similar library
>     that has additional features to DB connect URLs that we may want to
>     consider: https://github.com/joke2k/django-environ
>     <https://github.com/joke2k/django-environ>
>
>     It has the same issue with third party DB engines; for example, I
>     recently issued a PR to include `pyodbc` as an option. It has some
>     nice newcomer friendly features, such as a cleaner way of
>     determining BASE_URL / SITE_ROOT. Have a look at the URL for a good
>     "before" and "after" settings file example.
>
>     Having full support for .env files would make Django projects easier
>     to bring in line with 12-factor.
>
>     Regards,
>
>     Tim
>
>     On Saturday, May 27, 2017 at 3:52:23 PM UTC-4, Tom Forbes wrote:
>
>         Edit: DJANGO_SETTINGS_MODULE isn't relative, it will import any
>         arbitrary module you give it. If we accept that then I think we
>         are accepting the risk of imports via an attacker controlling
>         environment variables whilst Django starts up?
>
>         On Sat, May 27, 2017 at 8:49 PM, Tom Forbes <[hidden email]> wrote:
>
>             > I'm wary of possible security ramifications: if we do this, changing a
>             configuration value will import an arbitrary module, which
>             could make it easier to run arbitrary code in some
>             scenarios. I don't have a clear threat model in mind here,
>             though.
>
>             Good point, it's not wise to enable this even without a
>             clear threat model. Django does import the settings based on
>             an environment variable, but it's relative and if you can
>             use that to do anything nasty then you're most likely
>             already through the airtight hatch (so to speak). However
>             importing potentially global modules could be bad news.
>
>             Ignoring it is always an option, but could we not specify
>             that the third party database provider has to be in the
>             `INSTALLED_APPS`? That could provide some form of
>             whitelisting so not any old module is imported. Not sure
>             about any issues that may arise from this though.
>
>             > One possibility would be to use entrypoints in setuptools, this
>             way 3rd party backends could specify a name which then has a
>             fixed & verified import path.
>
>             This seems like it could get complex, and be quite unlike
>             anything else in Django.
>
>             Perhaps just supporting this for first-party database
>             backends is easiest?
>
>             On Thu, May 25, 2017 at 8:46 AM, Aymeric Augustin
>             <[hidden email]> wrote:
>
>                 Hello,
>
>                 I'm wondering what the exact definition of the URL
>                 format is. Is it specified somewhere? Or is it just:
>
>                 [engine]://[username]:[password]@[host]:[port]/[name]
>
>                 where we create arbitrary [engine] values in an ad-hoc
>                 fashion?
>
>>                 On 24 May 2017, at 21:21, Tom Forbes <[hidden email]>
>>                 wrote:
>>
>>                 My two cents: connection strings/database URI's are a
>>                 feature I've sorely missed in Django.
>>
>>                 Built-in functionality to convert environment
>>                 variables like DJANGO_DB_DEFAULT (or more generally
>>                 DJANGO_DB_*key*) into the relevant DATABASE setting
>>                 would make some deployment situations a lot simpler.
>>                 Currently, unless you use dj-database-uri you have to
>>                 define a bunch of ad-hoc DB_USER/DB_PASSWORD etc env
>>                 variables and price the dictionary together yourself.
>
>                 Fully agreed. While relatively minor, it's an annoyance.
>
>>                 How does this library complex keys like OPTIONS, TEST
>>                 or DEPENDENCIES?
>
>                 I don't think it's reasonable to cram them in a URL.
>
>                 dj-database-url allows passing options as extra keyword
>                 arguments. Other values should be explicitly added in
>                 the settings module, by updating the dict generated from
>                 the URL.
>
>>                 To help support third part backends: perhaps the
>>                 scheme portion of the URI could be either a relative
>>                 import from django.db.backends or an absolute import
>>                 to a third party library? It seems URI schemes can
>>                 have dots and underscores in them, so they can be
>>                 python package paths.
>>
>>                 I.e sqlite3://xyz would resolve go
>>                 django.db.backends.sqlite3, but sqlserver_ado://xyz
>>                 would resolve to the third party django-mssql engine
>>                 via an absolute import.
>
>                 I'm wary of possible security ramifications: if we do
>                 this, changing a configuration value will import an
>                 arbitrary module, which could make it easier to run
>                 arbitrary code in some scenarios. I don't have a clear
>                 threat model in mind here, though.
>
>                 I'd rather specify the database engine explicitly when
>                 calling dj-database-url if it's a third-party engine.
>                 There's an open question about what to do with the
>                 [engine] part of the URL in that case. Ignoring it
>                 entirely is the easiest.
>
>                 Best regards,
>
>                 --
>                 Aymeric.
>
>                 --
>                 You received this message because you are subscribed to
>                 the Google Groups "Django developers (Contributions to
>                 Django itself)" group.
>                 To unsubscribe from this group and stop receiving emails
>                 from it, send an email to
>                 [hidden email].
>                 To post to this group, send email to
>                 [hidden email].
>                 Visit this group at
>                 https://groups.google.com/group/django-developers
>                 <https://groups.google.com/group/django-developers>.
>                 To view this discussion on the web visit
>                 https://groups.google.com/d/msgid/django-developers/A0870D90-75DE-496B-A838-AC7D7488A67D%40polytechnique.org
>                 <https://groups.google.com/d/msgid/django-developers/A0870D90-75DE-496B-A838-AC7D7488A67D%40polytechnique.org?utm_medium=email&utm_source=footer>.
>                 For more options, visit
>                 https://groups.google.com/d/optout
>                 <https://groups.google.com/d/optout>.
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [hidden email]
> <mailto:[hidden email]>.
> To post to this group, send email to [hidden email]
> <mailto:[hidden email]>.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/69f0a647-42cf-4ef6-9ad7-bbe5e8fa779b%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/69f0a647-42cf-4ef6-9ad7-bbe5e8fa779b%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/da76c374-083d-4e6d-f826-14407b4c0ad9%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

Shai Berger
If I understand things correctly, registration of new url-schemas to enable
something like dj-database-url to support 3rd parties should be a trivial
matter; just add an entry to some dictionary, with the schema as key, and the
dotted-path of the backend as value.

IMO, scanning INSTALLED_APPS, or relying on ready(), is a lot of over-
engineering for this. Just do the registration from the settings file, before
invoking the url-parsing mechanism:

        from django.database_url import register_backend, configure_db

        register_backend('pyodbc-azure', 'sql_server.pyodbc')

        DATABASES = {'default' : configure_db() }

The only barrier I see is namespacing -- the functions mentioned above cannot
live in django.conf (would cause a circular import if imported from the
settings module) nor django.db (which shouldn't be imported too early), and
that's why I wrote `django.database_url` above.

What am I missing?
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

Kenneth Reitz-4
My thoughts are the same as Shai's below. 

On Monday, May 29, 2017 at 4:00:09 PM UTC-4, Shai Berger wrote:
If I understand things correctly, registration of new url-schemas to enable
something like dj-database-url to support 3rd parties should be a trivial
matter; just add an entry to some dictionary, with the schema as key, and the
dotted-path of the backend as value.

IMO, scanning INSTALLED_APPS, or relying on ready(), is a lot of over-
engineering for this. Just do the registration from the settings file, before
invoking the url-parsing mechanism:

        from django.database_url import register_backend, configure_db

        register_backend('pyodbc-azure', 'sql_server.pyodbc')

        DATABASES = {'default' : configure_db() }

The only barrier I see is namespacing -- the functions mentioned above cannot
live in django.conf (would cause a circular import if imported from the
settings module) nor django.db (which shouldn't be imported too early), and
that's why I wrote `django.database_url` above.

What am I missing?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/065601c3-9488-42aa-8b23-c7f906c26b5a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

Tom Forbes

I spent some time working on this today and fixed up my branch that implements this. I think I’ve got a nice API that passes all of the dj-database-url tests.

As there is a bit of database-specific shenanigans we have to perform on URLs I added a config_from_url static method to database adapters that takes the URL and returns a valid django configuration dictionary. As it’s a method inside the adapter itself we can override it to apply any db-specific fixups and third party backends can hook into the process as well via the same mechanism.

It expands on dj-database-url by also allowing the configuration of caches using the same method, which I think is pretty neat.

It uses a registration process described below, users just call register_db_backend or register_cache_backend with a path to their third party engine in the settings to link a scheme to a backend dotted path.

I had one question that I’m not sure the answer to: for databases I added the config_from_url to the .base.DatabaseWrapper class inside each built in adapter. I’ve kind of assumed each third party database would have the same structure, how much of a safe assumption is this?



On 8 June 2017 at 18:40:07, Kenneth Reitz ([hidden email]) wrote:

My thoughts are the same as Shai's below. 

On Monday, May 29, 2017 at 4:00:09 PM UTC-4, Shai Berger wrote:
If I understand things correctly, registration of new url-schemas to enable
something like dj-database-url to support 3rd parties should be a trivial
matter; just add an entry to some dictionary, with the schema as key, and the
dotted-path of the backend as value.

IMO, scanning INSTALLED_APPS, or relying on ready(), is a lot of over-
engineering for this. Just do the registration from the settings file, before
invoking the url-parsing mechanism:

        from django.database_url import register_backend, configure_db

        register_backend('pyodbc-azure', 'sql_server.pyodbc')

        DATABASES = {'default' : configure_db() }

The only barrier I see is namespacing -- the functions mentioned above cannot
live in django.conf (would cause a circular import if imported from the
settings module) nor django.db (which shouldn't be imported too early), and
that's why I wrote `django.database_url` above.

What am I missing?
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/065601c3-9488-42aa-8b23-c7f906c26b5a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJOpvhXUQCoLpt0Om%2BCgg1VgnWGvAHLQFPnqp%3DC1p%3DxVeQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

Michael Manfre
On Sun, Feb 4, 2018 at 12:59 PM Tom Forbes <[hidden email]> wrote:

I had one question that I’m not sure the answer to: for databases I added the config_from_url to the .base.DatabaseWrapper class inside each built in adapter. I’ve kind of assumed each third party database would have the same structure, how much of a safe assumption is this?

Every 3rd party database backend adheres to the internal API and must provide its own or an inherited DatabaseWrapper. Adding config_from_url seems like the best place to me.

Regards,
Michael Manfre

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAGdCwBso9cvaL0OgvFBtZjtrvs7DGunEEmJ-k0O2Kp0GPFhGkA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

gwahl
I'd like to approach this as 'support database urls in django', rather than 'copy/paste dj-database-url into django'. For postgres (I'm not sure about other backends), a database url can be passed directly to psycopg2. The postgres connection string format actually supports more features than is possible with django's HOST/USER/PORT... style settings. For example, you can pass multiple hosts and psycopg2 will attempt to connect to one of them: https://paquier.xyz/postgresql-2/postgres-10-multi-host-connstr/. Any attempt to parse the url in django will result in a loss of those features.

The only problem I see is that we have to parse the database backend out of the url, and you can't pass a url like 'postgis://....' to psyscopg2. I'd like to be able to do something like:

DATABASES = {
    'default': {
        'DSN': 'postgres://....',
        'ENGINE': 'django.contrib.gis.db.backends.postgis',
    },
}

And let psycopg2 handle the DSN.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/38553bb5-59b0-4772-a17a-282f795f1548%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

Luke M
I'd love to see native dj-database-url support in Django. I suppose it
would have to support all the back-ends, no? It might lead to confusion
otherwise.

On  0, [hidden email] wrote:

>   I'd like to approach this as 'support database urls in django', rather
>   than 'copy/paste dj-database-url into django'. For postgres (I'm not sure
>   about other backends), a database url can be passed directly to psycopg2.
>   The postgres connection string format actually supports more features than
>   is possible with django's HOST/USER/PORT... style settings. For example,
>   you can pass multiple hosts and psycopg2 will attempt to connect to one of
>   them: https://paquier.xyz/postgresql-2/postgres-10-multi-host-connstr/.
>   Any attempt to parse the url in django will result in a loss of those
>   features.
>   The only problem I see is that we have to parse the database backend out
>   of the url, and you can't pass a url like 'postgis://....' to psyscopg2.
>   I'd like to be able to do something like:
>   DATABASES = {
>       'default': {
>           'DSN': 'postgres://....',
>           'ENGINE': 'django.contrib.gis.db.backends.postgis',
>       },
>   }
>   And let psycopg2 handle the DSN.
>
>   --
>   You received this message because you are subscribed to the Google Groups
>   "Django developers (Contributions to Django itself)" group.
>   To unsubscribe from this group and stop receiving emails from it, send an
>   email to [1][hidden email].
>   To post to this group, send email to
>   [2][hidden email].
>   Visit this group at [3]https://groups.google.com/group/django-developers.
>   To view this discussion on the web visit
>   [4]https://groups.google.com/d/msgid/django-developers/38553bb5-59b0-4772-a17a-282f795f1548%40googlegroups.com.
>   For more options, visit [5]https://groups.google.com/d/optout.
>
>References
>
>   Visible links
>   1. mailto:[hidden email]
>   2. mailto:[hidden email]
>   3. https://groups.google.com/group/django-developers
>   4. https://groups.google.com/d/msgid/django-developers/38553bb5-59b0-4772-a17a-282f795f1548%40googlegroups.com?utm_medium=email&utm_source=footer
>   5. https://groups.google.com/d/optout

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20180728075639.b6i5dzhund3g2xfv%40lostatsea.lostatsea.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

Maciej Urbański
In reply to this post by gwahl
I would agree that DSN support seems like a nicer alternative to just copying dj-database-url, because it not only focuses on 12factor configuration in enviroment variables, but also enables some additional flexibility for the database connection option passing.

As for 12factor, I think https://gist.github.com/telent/9742059 is a good read as to why exposing as enviroment variables maybe not the best motivation. Having to accommodate settings, like database connection information, just so they can be fitted into single string put conveyable by enviroment variable is a case in point. We likely can do the same for Cache addresses as mentioned previously, although we may end up inventing new URI schemes do to that.., but django overall has multitude of other options that are not as easy to stringify.

On Friday, 27 July 2018 19:14:12 UTC+2, [hidden email] wrote:
I'd like to approach this as 'support database urls in django', rather than 'copy/paste dj-database-url into django'. For postgres (I'm not sure about other backends), a database url can be passed directly to psycopg2. The postgres connection string format actually supports more features than is possible with django's HOST/USER/PORT... style settings. For example, you can pass multiple hosts and psycopg2 will attempt to connect to one of them: <a href="https://paquier.xyz/postgresql-2/postgres-10-multi-host-connstr/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fpaquier.xyz%2Fpostgresql-2%2Fpostgres-10-multi-host-connstr%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHCUH6H0yRWHsG-zWbeGqqNaj6KkA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fpaquier.xyz%2Fpostgresql-2%2Fpostgres-10-multi-host-connstr%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHCUH6H0yRWHsG-zWbeGqqNaj6KkA&#39;;return true;">https://paquier.xyz/postgresql-2/postgres-10-multi-host-connstr/. Any attempt to parse the url in django will result in a loss of those features.

The only problem I see is that we have to parse the database backend out of the url, and you can't pass a url like 'postgis://....' to psyscopg2. I'd like to be able to do something like:

DATABASES = {
    'default': {
        'DSN': 'postgres://....',
        'ENGINE': 'django.contrib.gis.db.backends.postgis',
    },
}

And let psycopg2 handle the DSN.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1a55cc1c-ba9c-4950-ab94-50da8eec7d06%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate dj-database-url into Django

Tom Forbes
So in the PR I proposed I only bits I took verbatim from dj-database-url are the tests. The rest is re-implemented. I think it's a pretty good POC but I haven't touched it in a while.

In any case we have to implement our own parsing for backends that do not support passing in a URL to the connection library. 

Making postgres skip our parsing step and passing it in directly is an implementation detail and there are much more important questions around the API design to answer before this has any chance of being included.

On Sat, 28 Jul 2018, 12:57 Maciej Urbański, <[hidden email]> wrote:
I would agree that DSN support seems like a nicer alternative to just copying dj-database-url, because it not only focuses on 12factor configuration in enviroment variables, but also enables some additional flexibility for the database connection option passing.

As for 12factor, I think https://gist.github.com/telent/9742059 is a good read as to why exposing as enviroment variables maybe not the best motivation. Having to accommodate settings, like database connection information, just so they can be fitted into single string put conveyable by enviroment variable is a case in point. We likely can do the same for Cache addresses as mentioned previously, although we may end up inventing new URI schemes do to that.., but django overall has multitude of other options that are not as easy to stringify.

On Friday, 27 July 2018 19:14:12 UTC+2, [hidden email] wrote:
I'd like to approach this as 'support database urls in django', rather than 'copy/paste dj-database-url into django'. For postgres (I'm not sure about other backends), a database url can be passed directly to psycopg2. The postgres connection string format actually supports more features than is possible with django's HOST/USER/PORT... style settings. For example, you can pass multiple hosts and psycopg2 will attempt to connect to one of them: https://paquier.xyz/postgresql-2/postgres-10-multi-host-connstr/. Any attempt to parse the url in django will result in a loss of those features.

The only problem I see is that we have to parse the database backend out of the url, and you can't pass a url like 'postgis://....' to psyscopg2. I'd like to be able to do something like:

DATABASES = {
    'default': {
        'DSN': 'postgres://....',
        'ENGINE': 'django.contrib.gis.db.backends.postgis',
    },
}

And let psycopg2 handle the DSN.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1a55cc1c-ba9c-4950-ab94-50da8eec7d06%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJMa1xSzBeGUYygG7nxfCVz8jUPNEiSEJhbAqLDTwga9BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
12