App Engine support

classic Classic list List threaded Threaded
41 messages Options
123
Reply | Threaded
Open this post in threaded view
|

App Engine support

Waldemar Kornewald-4

Hi,
now that 1.1 is out we can finally discuss App Engine support. Is
anyone planning to work on this feature or working on it for real or
are there just announcements that someone might consider doing it?

BTW, I've reworked the mini-spec a little bit to improve id/key_name
handling. Here's the link, as a reminder:
http://code.djangoproject.com/wiki/AppEngine

Bye,
Waldemar Kornewald
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

Alex Gaynor

On Mon, Aug 10, 2009 at 7:19 AM, Waldemar Kornewald<[hidden email]> wrote:

>
> Hi,
> now that 1.1 is out we can finally discuss App Engine support. Is
> anyone planning to work on this feature or working on it for real or
> are there just announcements that someone might consider doing it?
>
> BTW, I've reworked the mini-spec a little bit to improve id/key_name
> handling. Here's the link, as a reminder:
> http://code.djangoproject.com/wiki/AppEngine
>
> Bye,
> Waldemar Kornewald
> >
>

As far as I know no one is working on this.  Personally, I'm of the
opinion that writing a basic backend (ie one that doesn't do JOIN
emulation) shouldn't be an overly arduos task and is probably possible
with no modifications to the existing Query class.  Writing a more
complete version (with JOIN emulation) may require some changes,
however I very much doubt the exact nature of these changes will
become apparent until someone actually starts to do the work.

In conclusion, no on is currently working on, but for all the people
who seem to ask for this I've seen almost no code written, which
suprises me since this is something that can exist 100% external to
Django (and probably should).

Alex

--
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

Russell Keith-Magee-2

On Mon, Aug 10, 2009 at 10:56 PM, Alex Gaynor<[hidden email]> wrote:

>
> On Mon, Aug 10, 2009 at 7:19 AM, Waldemar Kornewald<[hidden email]> wrote:
>>
>> Hi,
>> now that 1.1 is out we can finally discuss App Engine support. Is
>> anyone planning to work on this feature or working on it for real or
>> are there just announcements that someone might consider doing it?
>>
>> BTW, I've reworked the mini-spec a little bit to improve id/key_name
>> handling. Here's the link, as a reminder:
>> http://code.djangoproject.com/wiki/AppEngine
>>
>> Bye,
>> Waldemar Kornewald
>> >
>>
>
> As far as I know no one is working on this.  Personally, I'm of the
> opinion that writing a basic backend (ie one that doesn't do JOIN
> emulation) shouldn't be an overly arduos task and is probably possible
> with no modifications to the existing Query class.  Writing a more
> complete version (with JOIN emulation) may require some changes,
> however I very much doubt the exact nature of these changes will
> become apparent until someone actually starts to do the work.
>
> In conclusion, no on is currently working on, but for all the people
> who seem to ask for this I've seen almost no code written, which
> suprises me since this is something that can exist 100% external to
> Django (and probably should).

I'm not completely convinced this is completely true with the current
codebase, but it's certainly 95% true. The ORM was designed to be SQL
agnostic, but there are a couple of places in where, in the name of
'getting the job done', some shortcuts were taken that means that
Query assumes the presence of a SQL backend or SQL-like data
structures. The as_sql() call on Query is the one obvious example of
this, but there are some more subtle places where there is some
conceptual leakage.

However, Alex's comments are otherwise accurate. I'm aware of a few
projects where people are playing with integrating various non-SQL
data stores with Django (including AppEngine, MongoDB, Cassandra,
Couch and others). They are at varying levels of maturity, and have
varying levels of feature completeness.

My goal for supporting these backends at a project level is _not_ to
add them to the Django core - at least, not immediately, and certainly
not in the v1.2 timeframe. Instead, these projects _should_ be able to
be built and develop external to the Django core. In time, it may be
appropriate to add them to the core - but that is a long term goal,
not a short term goal.

If we were to phrase this as a v1.2 feature goal, it would not be "Add
AppEngine Support". Rather, it would be "make the Query backend
compatible with non-SQL query languages". This is likely to involve a
number of small but subtle changes to the ORM, but we can address
these on a case-by-case basis as the AppEngine (or any other non-SQL)
backend encounters difficulties.

Yours,
Russ Magee %-)

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

Malcolm Tredinnick

On Tue, 2009-08-11 at 07:55 +0800, Russell Keith-Magee wrote:

> On Mon, Aug 10, 2009 at 10:56 PM, Alex Gaynor<[hidden email]> wrote:
> >
> > On Mon, Aug 10, 2009 at 7:19 AM, Waldemar Kornewald<[hidden email]> wrote:
> >>
> >> Hi,
> >> now that 1.1 is out we can finally discuss App Engine support. Is
> >> anyone planning to work on this feature or working on it for real or
> >> are there just announcements that someone might consider doing it?
> >>
> >> BTW, I've reworked the mini-spec a little bit to improve id/key_name
> >> handling. Here's the link, as a reminder:
> >> http://code.djangoproject.com/wiki/AppEngine
> >>
> >> Bye,
> >> Waldemar Kornewald
> >> >
> >>
> >
> > As far as I know no one is working on this.  Personally, I'm of the
> > opinion that writing a basic backend (ie one that doesn't do JOIN
> > emulation) shouldn't be an overly arduos task and is probably possible
> > with no modifications to the existing Query class.  Writing a more
> > complete version (with JOIN emulation) may require some changes,
> > however I very much doubt the exact nature of these changes will
> > become apparent until someone actually starts to do the work.
> >
> > In conclusion, no on is currently working on, but for all the people
> > who seem to ask for this I've seen almost no code written, which
> > suprises me since this is something that can exist 100% external to
> > Django (and probably should).
>
> I'm not completely convinced this is completely true with the current
> codebase, but it's certainly 95% true. The ORM was designed to be SQL
> agnostic, but there are a couple of places in where, in the name of
> 'getting the job done', some shortcuts were taken that means that
> Query assumes the presence of a SQL backend or SQL-like data
> structures. The as_sql() call on Query is the one obvious example of
> this, but there are some more subtle places where there is some
> conceptual leakage.

Hmm... I would expect a backend to completely replace the Query class,
not work around something like as_sql(). One part of making non-db
backends supportable is to introduce a way to say "instead of using
django.db.models.sql.*, use this other namespace for the Query class and
everything it uses". That's why all the SQL stuff (modulo a few bits
that haven't been done yet) is in the one namespace and the public API
is entirely outside that namespace.

A non-db backend ends up implementing at the QuerySet -> Query
internal-API layer and that's where most of the tweaking will occur, I
would imagine. The object structure for non-relational and
other-storage-system backends is quite a bit different to the object
structure required to create a relational-algebra-based tree (which is
what Query does).

[...]
> My goal for supporting these backends at a project level is _not_ to
> add them to the Django core - at least, not immediately, and certainly
> not in the v1.2 timeframe. Instead, these projects _should_ be able to
> be built and develop external to the Django core. In time, it may be
> appropriate to add them to the core - but that is a long term goal,
> not a short term goal.

Definitely agreed. I think we know enough at this point to be able to
provide fairly generic support for this kind of thing. We've been
heading in that direction for a couple of years now, with that intent.

Regards,
Malcolm


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

Russell Keith-Magee-2

On Tue, Aug 11, 2009 at 9:53 AM, Malcolm
Tredinnick<[hidden email]> wrote:

>
> On Tue, 2009-08-11 at 07:55 +0800, Russell Keith-Magee wrote:
>> On Mon, Aug 10, 2009 at 10:56 PM, Alex Gaynor<[hidden email]> wrote:
>> >
>> > On Mon, Aug 10, 2009 at 7:19 AM, Waldemar Kornewald<[hidden email]> wrote:
>> >>
>> >> Hi,
>> >> now that 1.1 is out we can finally discuss App Engine support. Is
>> >> anyone planning to work on this feature or working on it for real or
>> >> are there just announcements that someone might consider doing it?
>> >>
>> >> BTW, I've reworked the mini-spec a little bit to improve id/key_name
>> >> handling. Here's the link, as a reminder:
>> >> http://code.djangoproject.com/wiki/AppEngine
>> >>
>> >> Bye,
>> >> Waldemar Kornewald
>> >> >
>> >>
>> >
>> > As far as I know no one is working on this.  Personally, I'm of the
>> > opinion that writing a basic backend (ie one that doesn't do JOIN
>> > emulation) shouldn't be an overly arduos task and is probably possible
>> > with no modifications to the existing Query class.  Writing a more
>> > complete version (with JOIN emulation) may require some changes,
>> > however I very much doubt the exact nature of these changes will
>> > become apparent until someone actually starts to do the work.
>> >
>> > In conclusion, no on is currently working on, but for all the people
>> > who seem to ask for this I've seen almost no code written, which
>> > suprises me since this is something that can exist 100% external to
>> > Django (and probably should).
>>
>> I'm not completely convinced this is completely true with the current
>> codebase, but it's certainly 95% true. The ORM was designed to be SQL
>> agnostic, but there are a couple of places in where, in the name of
>> 'getting the job done', some shortcuts were taken that means that
>> Query assumes the presence of a SQL backend or SQL-like data
>> structures. The as_sql() call on Query is the one obvious example of
>> this, but there are some more subtle places where there is some
>> conceptual leakage.
>
> Hmm... I would expect a backend to completely replace the Query class,
> not work around something like as_sql().

Sorry - I meant QuerySet, not Query (i.e., the as_sql() call on
QuerySet is an example of leakage). Chalk that one up to a flu-addled
brain. I agree that a non-sql backend should be implementing
appengine.Query, which _should_ be a drop-in for sql.Query - although,
as noted, there might be a couple of places where some tweaks are
needed.

Yours,
Russ Magee %-)

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

Malcolm Tredinnick
In reply to this post by Waldemar Kornewald-4

On Mon, 2009-08-10 at 05:19 -0700, Waldemar Kornewald wrote:
> Hi,
> now that 1.1 is out we can finally discuss App Engine support. Is
> anyone planning to work on this feature or working on it for real or
> are there just announcements that someone might consider doing it?
>
> BTW, I've reworked the mini-spec a little bit to improve id/key_name
> handling. Here's the link, as a reminder:
> http://code.djangoproject.com/wiki/AppEngine
>

Further to the other replies, Russell and I have made, I took a quick
look at this wiki page and there's some confusion in the summary section
there.

The only that seems to be possibly *required* to be changed in Django to
support app-engine as a storage backend is the ManyToManyField change (I
say "possibly" because I haven't thought around the issue much, so don't
know if there are alternatives).

Things like a ListField aren't required. They could easily enough be
done as a standard custom field, but support for every field that's
possible in a particular database backend isn't our goal at the moment
and we don't add stuff to Django for that reason. Django runs on
app-engine would mean that existing applications run with the existing
field types. Not that every possible app-engine storage structure is
emulated in Django.

Batch delete() is already pretty much what happens. Batch updates
already exist and batch saving is an often enough requested feature that
it's likely to happen in the near future (I can imagine what an API for
that might look like, for example).

Changing the permission and content-type structure is one proposed
solution to a particular problem, but I'm not sure it's the only one. I
think that's something an external storage engine has to work out how to
solve, not require a huge internal change in Django (which we can't do
for practical reasons, as Permission and content types are normal models
right now, so they are fully supported and changing that would be
backwards incompatible, not to mention damn inconvenient in code).

Regards,
Malcolm



> Bye,
> Waldemar Kornewald
> >
>


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

Russell Keith-Magee-2

On Tue, Aug 11, 2009 at 10:07 AM, Malcolm
Tredinnick<[hidden email]> wrote:

>
> On Mon, 2009-08-10 at 05:19 -0700, Waldemar Kornewald wrote:
>> Hi,
>> now that 1.1 is out we can finally discuss App Engine support. Is
>> anyone planning to work on this feature or working on it for real or
>> are there just announcements that someone might consider doing it?
>>
>> BTW, I've reworked the mini-spec a little bit to improve id/key_name
>> handling. Here's the link, as a reminder:
>> http://code.djangoproject.com/wiki/AppEngine
>>
>
> Further to the other replies, Russell and I have made, I took a quick
> look at this wiki page and there's some confusion in the summary section
> there.
>
> The only that seems to be possibly *required* to be changed in Django to
> support app-engine as a storage backend is the ManyToManyField change (I
> say "possibly" because I haven't thought around the issue much, so don't
> know if there are alternatives).

This one is slightly topical. Alex has a github branch that refactors
the m2m code to get the SQL out of the related field model [1]. In
order to do this, it introduces a dummy model for m2m fields. This is
needed for Alex's multi-db work, and it was my intention to target the
patch for Django's trunk in the near-ish future. If there is overlap
into the requirements for AppEngine/non-sql support, I'd be interested
to hear any feedback.

[1] http://github.com/alex/django/tree/m2m-refactor

Russ %-)

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

Jacob Kaplan-Moss-2
In reply to this post by Malcolm Tredinnick

On Mon, Aug 10, 2009 at 9:07 PM, Malcolm
Tredinnick<[hidden email]> wrote:
> The only that seems to be possibly *required* to be changed in Django to
> support app-engine as a storage backend is the ManyToManyField change (I
> say "possibly" because I haven't thought around the issue much, so don't
> know if there are alternatives).

That's true, in a sense, if you define "support AppEngine" (or other
non-relational backends) as "let you define models and query 'em."
However... what people usually *mean* when they ask "does Django run
on AppEngine?" is "does the *Django admin* run on AppEngine?"

I suspect that simply supporting non-relational backends won't really
make most real-world users happy, but will just sorta kick the can
down the road.

That's why I, for one, won't say that Django "supports" non-relational
backends until (most of) django.contrib runs on NOSQL_BACKEND
unmodified, and until it's easy to write reusable apps against an API
that'll run there.

So, for me, the first step is to separate the QuerySet API into "the
set of operations that we can expect *any* backend to support" and
then "the set of operations that realistically only a relational
backend will support." For example, ``get(pk=N)`` should be expected
to work anywhere, but ``filter(foo__bar__baz__startswith='spam')``
probably is only gonna work in SQL. Likewise, ``select_related()``
means something specific on a SQL backend, but probably would be a
noop against something like CouchDB.

Given the work I've done in this area to date I expect the first set
of operations to be sufficiently powerful that any app generic enough
to be a contrib candidate could be written against this scaled-down
QuerySet API. If that's true, it should in theory be possible to get
the admin (and auth, etc.) running on non-relational stores.

[My hope is to put a lot more work into this in the near future, but
given my unpredictable work schedule I can't promise anything.]

Anyway, this is mostly a brain dump... but the main takeaway I want
people to get is that simply supporting non-relational backends in a
first-class manner isn't enough; most average users won't give a shit
until the admin works, too.

Jacob

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

Malcolm Tredinnick
In reply to this post by Russell Keith-Magee-2

On Tue, 2009-08-11 at 10:58 +0800, Russell Keith-Magee wrote:
[...]
> This one is slightly topical. Alex has a github branch that refactors
> the m2m code to get the SQL out of the related field model [1]. In
> order to do this, it introduces a dummy model for m2m fields. This is
> needed for Alex's multi-db work, and it was my intention to target the
> patch for Django's trunk in the near-ish future. If there is overlap
> into the requirements for AppEngine/non-sql support, I'd be interested
> to hear any feedback.

I'll have a look at that as soon as I get a chance. It's
build-your-own-diff country, so hopefully it's fairly self-contained.

Regards,
Malcolm



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

Malcolm Tredinnick
In reply to this post by Jacob Kaplan-Moss-2

On Mon, 2009-08-10 at 22:31 -0500, Jacob Kaplan-Moss wrote:
> On Mon, Aug 10, 2009 at 9:07 PM, Malcolm
> Tredinnick<[hidden email]> wrote:
> > The only that seems to be possibly *required* to be changed in Django to
> > support app-engine as a storage backend is the ManyToManyField change (I
> > say "possibly" because I haven't thought around the issue much, so don't
> > know if there are alternatives).
>
> That's true, in a sense, if you define "support AppEngine" (or other
> non-relational backends) as "let you define models and query 'em."

It's true in a broader sense than that, too.

> However... what people usually *mean* when they ask "does Django run
> on AppEngine?" is "does the *Django admin* run on AppEngine?"

Failure to phrase their question correctly isn't my problem. :-)

That being said, I think we're in mostly violent agreement here as to
the end goal. However, it's walk before running. The goal would be to
support unchanged public API and that influences the design.

However, you *will* have to set up the settings appropriately, so cruft
like the email backends that Walther has in his summary list isn't a
major core change to Django.

> I suspect that simply supporting non-relational backends won't really
> make most real-world users happy, but will just sorta kick the can
> down the road.
>
> That's why I, for one, won't say that Django "supports" non-relational
> backends until (most of) django.contrib runs on NOSQL_BACKEND
> unmodified, and until it's easy to write reusable apps against an API
> that'll run there.
>
> So, for me, the first step is to separate the QuerySet API into "the
> set of operations that we can expect *any* backend to support" and
> then "the set of operations that realistically only a relational
> backend will support." For example, ``get(pk=N)`` should be expected
> to work anywhere, but ``filter(foo__bar__baz__startswith='spam')``
> probably is only gonna work in SQL.

I'd disagree on that last point. It might work out to be multiple
operations and be non-atomic, but that's once of the prices you pay when
moving to a key/value-based storage system. There's no reason the
function call can't return the ultimately right results over a
non-changing dataset, though. Making that so is also something that
won't add a single line of cruft to the SQL backend, since it already
has the support for that. It all goes into the backend's sql/*
replacement module.

> Likewise, ``select_related()``
> means something specific on a SQL backend, but probably would be a
> noop against something like CouchDB.

Right. Which is definitely "works" in my book.

Regards,
Malcolm


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

Waldemar Kornewald-4
In reply to this post by Russell Keith-Magee-2

Hi,

On Aug 11, 1:55 am, Russell Keith-Magee <[hidden email]>
wrote:

> > In conclusion, no on is currently working on, but for all the people
> > who seem to ask for this I've seen almost no code written, which
> > suprises me since this is something that can exist 100% external to
> > Django (and probably should).
>
> I'm not completely convinced this is completely true with the current
> codebase, but it's certainly 95% true. The ORM was designed to be SQL
> agnostic, but there are a couple of places in where, in the name of
> 'getting the job done', some shortcuts were taken that means that
> Query assumes the presence of a SQL backend or SQL-like data
> structures. The as_sql() call on Query is the one obvious example of
> this, but there are some more subtle places where there is some
> conceptual leakage.

There are other issues which IMHO prevent a clean and efficient
backend implementation:

Transactions:
There are no begin/end_transaction functions in App Engine. You can
only run a whole function in a transaction, like @commit_on_success,
but with deeper restrictions on which rows can be accessed within the
transaction (only within one single entity group). Also, you can only
run queries with a pk within a transaction and such queries lock like
SELECT FOR UPDATE. In the future you might be able to run distributed
transactions, but that will be very inefficient, so you should still
design your transaction code carefully. Conclusion: With the current
API, a backend can't implement transaction support. Actually, Django
would need a transaction decorator which modifies all queries to run a
SELECT FOR UPDATE in order to correctly model App Engine-like behavior
(having to specify .for_update() on each query individually would be
too annoying, so a decorator would make that easier).

Delete:
When deleting a model instance all related objects are deleted, too.
The problem is that if you delete within a transaction you can't run
queries. Thus, if you want to delete related objects you need some
kind of post_delete_committed signal handler which gets called after
an object has been deleted and after the transaction has been
committed successfully. Of course, this won't guarantee a consistent
DB state because the related objects aren't deleted within a
transaction. Moreover, when batch-deleting a lot of instances you'll
cause too many queries for related objects, so this is very
inefficient. In the end, I'd say that the best solution is to provide
an optional delete_related_objects handler which can be installed on
individual models (for which you know that this problem won't occur)
via a post_delete_committed signal. AFAIK, with the current API, a
backend can't disable/override the deletion of related objects.

Save:
The save_base() function currently checks if the instance has a pk. If
so, it runs a query to check if the pk already exists in the DB, so it
knows whether to do an _insert() or _update(). This distinction isn't
needed in App Engine because there's no difference between insert and
update. So, it would be more efficient to remove that check. Again,
this can't yet be overridden by the backend.

And these are just the first few issues we've run into when analyzing
the source.

It would be much easier to simply allow for overriding Model and
QuerySet at a high level.

> However, Alex's comments are otherwise accurate. I'm aware of a few
> projects where people are playing with integrating various non-SQL
> data stores with Django (including AppEngine, MongoDB, Cassandra,
> Couch and others). They are at varying levels of maturity, and have
> varying levels of feature completeness.

Where are those projects? In particular, could you please give me the
contact information and repository of the App Engine backend
developers? We'd like to get involved and help.

> My goal for supporting these backends at a project level is _not_ to
> add them to the Django core - at least, not immediately, and certainly
> not in the v1.2 timeframe. Instead, these projects _should_ be able to
> be built and develop external to the Django core. In time, it may be
> appropriate to add them to the core - but that is a long term goal,
> not a short term goal.

That's definitely OK. But if you (the Django developers) hope to
support most of the contrib apps 1:1 without any special checks for
non-relational backends I think this goal will be very difficult to
achieve. I had to disable result sorting in the admin interface, for
example, because that would require too many composite datastore
indexes and there are almost endless combinations when you can filter
and sort and search results in the admin interface. Who knows, maybe
when Google adds full-text search this limitation will go away...

Bye,
Waldemar Kornewald
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

Malcolm Tredinnick

On Tue, 2009-08-11 at 00:03 -0700, Waldemar Kornewald wrote:

> Hi,
>
> On Aug 11, 1:55 am, Russell Keith-Magee <[hidden email]>
> wrote:
> > > In conclusion, no on is currently working on, but for all the people
> > > who seem to ask for this I've seen almost no code written, which
> > > suprises me since this is something that can exist 100% external to
> > > Django (and probably should).
> >
> > I'm not completely convinced this is completely true with the current
> > codebase, but it's certainly 95% true. The ORM was designed to be SQL
> > agnostic, but there are a couple of places in where, in the name of
> > 'getting the job done', some shortcuts were taken that means that
> > Query assumes the presence of a SQL backend or SQL-like data
> > structures. The as_sql() call on Query is the one obvious example of
> > this, but there are some more subtle places where there is some
> > conceptual leakage.
>
> There are other issues which IMHO prevent a clean and efficient
> backend implementation:

[... various individual items snipped...]
>
> And these are just the first few issues we've run into when analyzing
> the source.

Most of those are the kind of incremental changes that are part of
making the backend stuff generic. For example, pushing some of the logic
of the insert/update stuff down into the Query layer is a solution to
the last point. A lot of the delete stuff is already part of the query
layer, so it's not much extra to add a layer on the extra stuff.

This is definitely the sort of stuff that is good to know, but initially
I think it's good to keep phrasing as being specific problems, rather
than targeting particular solutions, since there is clearly some
difference in the way you want to go to the way we would like move and
it's primarily important to understand the problems we're solving,
rather than spinning around on prematurely debating micro-solutions.

> It would be much easier to simply allow for overriding Model and
> QuerySet at a high level.

By "easier" you mean that you would only have to write *all* of the
public API again for each backend, instead of reusing as much common
stuff as possible?! That's the solution that maybe gets you GAE support
early on at the cost of a bunch of developer work for this particular
individual item, but doesn't pay-off for long-term reusability.

Doing the easiest thing possible here at the expense of pain down the
track is a false goal. Our job as framework maintainers is to do the
hard stuff so that everybody else says "that was much easier than I
thought it would be" when they come to using it. We've done it a number
of times in the past and I take no end of pleasure in repeatedly hearing
people surprised at discovering how tricky things are under the cover so
that their code just works, no matter whether they're using the fully
public API or some of the semi-stable internal bits like extending
QuerySets or writing custom Q-objects, or writing a new database
backends. *That* is the brass ring. Making Query and friends replaceable
means that bug fixes and enhancements in the common code are seen by
everybody. Making Model and QuerySet replaceable means that there will
hardly ever be parity between the alternate storage backends as the
different maintainers add different sets of features.

[...]
>  But if you (the Django developers) hope to
> support most of the contrib apps 1:1 without any special checks for
> non-relational backends I think this goal will be very difficult to
> achieve. I had to disable result sorting in the admin interface, for
> example, because that would require too many composite datastore
> indexes and there are almost endless combinations when you can filter
> and sort and search results in the admin interface. Who knows, maybe
> when Google adds full-text search this limitation will go away...

This is why I also disagreed slightly with Jacob's end-goal. Because
people's dreams are a little ahead of reality. Django cannot standardise
on lowest-common-denominator, even for contrib  apps. So I think there
will be things that simply don't work in certain cases because the
backend isn't up to the task. But it won't be entire apps. It will be
small features in individual apps. Where we draw that fuzzy line is
obviously something to evaluate as we go along. For now, let's
definitely think big and get as much of the way there as we can.

Regards,
Malcolm



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

terrex
In reply to this post by Waldemar Kornewald-4

There is another work on this, apart from http://code.google.com/p/app-engine-patch/
, which is at http://code.google.com/p/google-app-engine-django/
Regards,

On Aug 10, 12:19 pm, Waldemar Kornewald <[hidden email]> wrote:

> Hi,
> now that 1.1 is out we can finally discuss App Engine support. Is
> anyone planning to work on this feature or working on it for real or
> are there just announcements that someone might consider doing it?
>
> BTW, I've reworked the mini-spec a little bit to improve id/key_name
> handling. Here's the link, as a reminder:http://code.djangoproject.com/wiki/AppEngine
>
> Bye,
> Waldemar Kornewald

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

Waldemar Kornewald-4
In reply to this post by Malcolm Tredinnick

On Aug 11, 10:01 am, Malcolm Tredinnick <[hidden email]>
wrote:

> On Tue, 2009-08-11 at 00:03 -0700, Waldemar Kornewald wrote:
> > And these are just the first few issues we've run into when analyzing
> > the source.
>
> Most of those are the kind of incremental changes that are part of
> making the backend stuff generic. For example, pushing some of the logic
> of the insert/update stuff down into the Query layer is a solution to
> the last point. A lot of the delete stuff is already part of the query
> layer, so it's not much extra to add a layer on the extra stuff.
>
> This is definitely the sort of stuff that is good to know, but initially
> I think it's good to keep phrasing as being specific problems, rather
> than targeting particular solutions, since there is clearly some
> difference in the way you want to go to the way we would like move and
> it's primarily important to understand the problems we're solving,
> rather than spinning around on prematurely debating micro-solutions.

I thought the high-level consensus already was to move more stuff to
the QuerySet/Query level and allow backends to override Query? What
information do you need for refactoring the backend API?

Bye,
Waldemar Kornewald
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

Mitchell Garnaat

As a heavy SimpleDB user, I would love to see a way to use Django and
SimpleDB together and I would certainly be willing to devote time to
helping make that happen.  I think I can contribute on the SimpleDB
side (boto has supported SimpleDB for over a year and I'm very
familiar with the service) but every time I sit down to try to figure
out how to begin the process of integrating it into Django, I get
stuck.  I've only used Django a few times and I'm really not familiar
with the internals but it didn't seem obvious to me how or where to
get started.

Is the consensus that further refactoring or rethinking of things like
QuerySet and Query are required to make this happen?  Is there an
"official" interface between Django and DB backends?

Mitch

On Aug 11, 1:49 pm, Waldemar Kornewald <[hidden email]> wrote:

> On Aug 11, 10:01 am, Malcolm Tredinnick <[hidden email]>
> wrote:
>
>
>
> > On Tue, 2009-08-11 at 00:03 -0700, Waldemar Kornewald wrote:
> > > And these are just the first few issues we've run into when analyzing
> > > the source.
>
> > Most of those are the kind of incremental changes that are part of
> > making the backend stuff generic. For example, pushing some of the logic
> > of the insert/update stuff down into the Query layer is a solution to
> > the last point. A lot of the delete stuff is already part of the query
> > layer, so it's not much extra to add a layer on the extra stuff.
>
> > This is definitely the sort of stuff that is good to know, but initially
> > I think it's good to keep phrasing as being specific problems, rather
> > than targeting particular solutions, since there is clearly some
> > difference in the way you want to go to the way we would like move and
> > it's primarily important to understand the problems we're solving,
> > rather than spinning around on prematurely debating micro-solutions.
>
> I thought the high-level consensus already was to move more stuff to
> the QuerySet/Query level and allow backends to override Query? What
> information do you need for refactoring the backend API?
>
> Bye,
> Waldemar Kornewald

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

Russell Keith-Magee-2
In reply to this post by Waldemar Kornewald-4

On Wed, Aug 12, 2009 at 1:49 AM, Waldemar Kornewald<[hidden email]> wrote:

>
> On Aug 11, 10:01 am, Malcolm Tredinnick <[hidden email]>
> wrote:
>> On Tue, 2009-08-11 at 00:03 -0700, Waldemar Kornewald wrote:
>> > And these are just the first few issues we've run into when analyzing
>> > the source.
>>
>> Most of those are the kind of incremental changes that are part of
>> making the backend stuff generic. For example, pushing some of the logic
>> of the insert/update stuff down into the Query layer is a solution to
>> the last point. A lot of the delete stuff is already part of the query
>> layer, so it's not much extra to add a layer on the extra stuff.
>>
>> This is definitely the sort of stuff that is good to know, but initially
>> I think it's good to keep phrasing as being specific problems, rather
>> than targeting particular solutions, since there is clearly some
>> difference in the way you want to go to the way we would like move and
>> it's primarily important to understand the problems we're solving,
>> rather than spinning around on prematurely debating micro-solutions.
>
> I thought the high-level consensus already was to move more stuff to
> the QuerySet/Query level and allow backends to override Query?

Yes. I don't see how what Malcolm said contradicts this. Malcolm was
referring to the fact that some of the model level operations - such
as delete - are already implemented as interfaces on Query. If you
need to move more model functions into the Query to satisfy AppEngine,
this is something we are open to discussing.

> What
> information do you need for refactoring the backend API?

A concrete proposal. "I think we should move save() to the Query
interface because.... here is a sample implementation explaining what
I mean."

As Malcolm said - it's very easy to get stuck in incredibly complex
debates of entirely hypothetical situations. The current
implementation is sufficient for SQL backends, and is relatively clean
in design. You are the expert on AppEngine. You are the person in the
position to tell us where you need flexibility that is not currently
afforded. Make a concrete proposal, and we'll discuss it.

Yours,
Russ Magee %-)

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

Russell Keith-Magee-2
In reply to this post by Mitchell Garnaat

On Wed, Aug 12, 2009 at 2:56 AM, mitch<[hidden email]> wrote:

>
> As a heavy SimpleDB user, I would love to see a way to use Django and
> SimpleDB together and I would certainly be willing to devote time to
> helping make that happen.  I think I can contribute on the SimpleDB
> side (boto has supported SimpleDB for over a year and I'm very
> familiar with the service) but every time I sit down to try to figure
> out how to begin the process of integrating it into Django, I get
> stuck.  I've only used Django a few times and I'm really not familiar
> with the internals but it didn't seem obvious to me how or where to
> get started.

Begin at the beginning :-) Pick a small task - e.g., converting a
simple Django ORM query into SimpleDB's query API, and see where that
takes you. Rinse and repeat until you have a working API.

> Is the consensus that further refactoring or rethinking of things like
> QuerySet and Query are required to make this happen?

Not really a consensus - more a general feeling that there are some
SQL-specifics that still need to be purged. The Query/QuerySet
interface was designed with support for non-SQL backends as a design
goal, but without an actual non-SQL backend implementation to prove
the design. Finding and purging these SQL-specific components is the
work in progress.

> Is there an
> "official" interface between Django and DB backends?

It's official in the sense that exists in the core codebase. However,
unlike most of Django's official interfaces, it is not extensively
documented. The two interfaces are sql.Query and db.backend. If you're
interested in this problem, you're going to need to go spelunking
through the code, trying to understand it from the SQL point of view,
and then adapting it for a non-SQL backend.

Yours,
Russ Magee %-)

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

Waldemar Kornewald-4

On Aug 12, 2:09 am, Russell Keith-Magee <[hidden email]>
wrote:
> > Is the consensus that further refactoring or rethinking of things like
> > QuerySet and Query are required to make this happen?
>
> Not really a consensus - more a general feeling that there are some
> SQL-specifics that still need to be purged. The Query/QuerySet
> interface was designed with support for non-SQL backends as a design
> goal, but without an actual non-SQL backend implementation to prove
> the design. Finding and purging these SQL-specific components is the
> work in progress.

Is there a way to override sql.subqueries? Or is there a need for it,
at all? Those are used by Model and QuerySet, but if I override Query
We could probably use something hacky like isinstance(self,
InsertQuery) to detect that we'm actually an insert query and act
accordingly. It would be nice to have a less hacky solutino for this,
though.

I've started summarizing all TODOs and required changes on a higher
level here:
http://code.djangoproject.com/wiki/NonSqlBackends
I hope this is what you guys need. The low-level App Engine stuff is
on a separate wiki page.

Could everyone please add his findings to that wiki page? I only have
basic knowledge about SimpleDB, so Mitch, we could need your help
here. (the wiki page even has a little TODO for you :).

Bye,
Waldemar Kornewald
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

Malcolm Tredinnick

On Wed, 2009-08-12 at 02:32 -0700, Waldemar Kornewald wrote:

> On Aug 12, 2:09 am, Russell Keith-Magee <[hidden email]>
> wrote:
> > > Is the consensus that further refactoring or rethinking of things like
> > > QuerySet and Query are required to make this happen?
> >
> > Not really a consensus - more a general feeling that there are some
> > SQL-specifics that still need to be purged. The Query/QuerySet
> > interface was designed with support for non-SQL backends as a design
> > goal, but without an actual non-SQL backend implementation to prove
> > the design. Finding and purging these SQL-specific components is the
> > work in progress.
>
> Is there a way to override sql.subqueries?

Not yet. As mentioned earlier in the thread, a large chunk of the
process of making non-SQL support is to allow wholesale overriding of
the django.db.models.sql namespace. Everything in there is SQL specific
and I would hope that that is the layer that is entirely replaced by
alternate storage systems. So overriding subqueries feels like too low
level, because it has plenty of builtin SQL assumptions.

So anything that imports and uses things from django.db.models.sql needs
to instead use a common API to fetch the appropriate module and we need
to thin down the interface. Nothing outside of django.db.models.query
should be importing from that namespace, ideally (I believe that should
be true now, unless we've introduced bugs) and then we need to introduce
something like a (*shudder*) set of factory methods or otherwise split
up the features. That's certainly where the design changes are initially
going to happen in this, we already know that.

Regards,
Malcolm


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply | Threaded
Open this post in threaded view
|

Re: App Engine support

Waldemar Kornewald-4

On Aug 12, 11:40 am, Malcolm Tredinnick <[hidden email]>
wrote:
> On Wed, 2009-08-12 at 02:32 -0700, Waldemar Kornewald wrote:
> > Is there a way to override sql.subqueries?
>
> Not yet. As mentioned earlier in the thread, a large chunk of the
> process of making non-SQL support is to allow wholesale overriding of
> the django.db.models.sql namespace. Everything in there is SQL specific
> and I would hope that that is the layer that is entirely replaced by
> alternate storage systems. So overriding subqueries feels like too low
> level, because it has plenty of builtin SQL assumptions.

OK, I think I misunderstood how far you want to go with the
refactoring. Non-SQL support could be achieved with a few smaller
changes (my impression was that the preferred solution is to override
sql.Query), but it looks like you want to do a much bigger
refactoring.

> So anything that imports and uses things from django.db.models.sql needs
> to instead use a common API to fetch the appropriate module and we need
> to thin down the interface. Nothing outside of django.db.models.query
> should be importing from that namespace, ideally (I believe that should
> be true now, unless we've introduced bugs) and then we need to introduce
> something like a (*shudder*) set of factory methods or otherwise split
> up the features. That's certainly where the design changes are initially
> going to happen in this, we already know that.

When do you plan to start with the changes?

I'm trying to get some more App Engine developers involved, so this we
can get this done faster. Mitch, do you have anyone else on the
SimpleDB side who could help, too?

Bye,
Waldemar Kornewald
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to [hidden email]
To unsubscribe from this group, send email to [hidden email]
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en
-~----------~----~----~----~------~----~------~--~---

123