Quantcast

Deep serialization of GAE models

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Deep serialization of GAE models

Fernando Correia
If I have two Google App Engine model objects like these:

class Project(db.Model):
    name = db.StringProperty()

class ProjectParticipant(db.Model):
    project = db.ReferenceProperty(Project, collection_name='participants')
    name = db.StringProperty()

When I return an instance of ProjectParticipant from the GAE service,
an instance of the Project is embedded. That may not be very
efficient. If I have 100 participants in a project, the project's data
will be serialized 100 times...

I'd rather have only the Project's key be serialized. Is there an
option for this in PyAMF? Or is the best strategy to use a Data
Transfer Object and copy the data from the model object to the DTO and
return the DTO?
_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deep serialization of GAE models

Nick Joyce
Hi Fernando,

Both AMF0 and AMF3 can contain references to objects so if the participants of a project were rendered individually, the project object would only be rendered once and then references would be made to it (2 bytes usually).

Does that help?

On 2 Aug 2008, at 00:35, Fernando Correia wrote:

If I have two Google App Engine model objects like these:

class Project(db.Model):
   name = db.StringProperty()

class ProjectParticipant(db.Model):
   project = db.ReferenceProperty(Project, collection_name='participants')
   name = db.StringProperty()

When I return an instance of ProjectParticipant from the GAE service,
an instance of the Project is embedded. That may not be very
efficient. If I have 100 participants in a project, the project's data
will be serialized 100 times...

I'd rather have only the Project's key be serialized. Is there an
option for this in PyAMF? Or is the best strategy to use a Data
Transfer Object and copy the data from the model object to the DTO and
return the DTO?
_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users


_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deep serialization of GAE models

Fernando Correia
2008/8/2 Nick Joyce <[hidden email]>:
> Hi Fernando,
> Both AMF0 and AMF3 can contain references to objects so if the participants
> of a project were rendered individually, the project object would only be
> rendered once and then references would be made to it (2 bytes usually).
> Does that help?

Thanks for your help! Sure, in the scenario I described that would
save a lot of bytes.

In other cases, I might actually only to return a key. I will try an
approach with DTO to see if I can have more control over what is
serialized.
_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deep serialization of GAE models

Fernando Correia
In reply to this post by Nick Joyce
2008/8/2 Nick Joyce <[hidden email]>:
> Both AMF0 and AMF3 can contain references to objects so if the participants
> of a project were rendered individually, the project object would only be
> rendered once and then references would be made to it (2 bytes usually).
> Does that help?

When I first read your answer, I thought PyAMF already had support for
this feature of AMF. Now I understand that it doesn't, but it could be
added. Is that what you mean?

I also realize now why the referenced class is being serialized. On
Google App Engine, when you use a reference property, it automatically
fetches the referenced object and returns it. So PyAMF's standard
serialization of GAE object will probably always perform a deep
serialization. I wonder if it will have problems with circular
references.

Anyway, I was able to implement a pattern for returning DTOs. I
explain my approach in this article:

http://fernandoacorreia.wordpress.com/2008/08/02/returning-dtos-from-pyamf/

Thanks for your assistance.
_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deep serialization of GAE models

Nick Joyce
On 2 Aug 2008, at 16:48, Fernando Correia wrote:

2008/8/2 Nick Joyce <[hidden email]>:
Both AMF0 and AMF3 can contain references to objects so if the participants
of a project were rendered individually, the project object would only be
rendered once and then references would be made to it (2 bytes usually).
Does that help?

When I first read your answer, I thought PyAMF already had support for
this feature of AMF. Now I understand that it doesn't, but it could be
added. Is that what you mean?

Sorry for not being clear, PyAMF already supports this feature.



I also realize now why the referenced class is being serialized. On
Google App Engine, when you use a reference property, it automatically
fetches the referenced object and returns it. So PyAMF's standard
serialization of GAE object will probably always perform a deep
serialization. I wonder if it will have problems with circular
references.

This is one of the features that the reference type is designed to support. An example:

import pyamf

x = [1, 2, 3]
x.append(x) 

y = pyamf.decode(pyamf.encode(x)).next()

assert id(y[3]) == id(y), 'not a circular reference'



Anyway, I was able to implement a pattern for returning DTOs. I
explain my approach in this article:

http://fernandoacorreia.wordpress.com/2008/08/02/returning-dtos-from-pyamf/

Thanks for your assistance.
_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users


_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deep serialization of GAE models

Fernando Correia
Thank you for the clarificatios, Nick.
_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deep serialization of GAE models

workie
Hi,

I am trying to write a basic proof of concept application and I keep falling short.  If I ever get it working, perhaps I'll write a tutorial for those who come after me.

BUT, I am trying to get Flex talking to a GAE app via PyAMF.  I wrote a VO (value object or data transfer object) class in Flex to mirror the Python class, but couldn't get my app working until I found some posts discussing the _key field.  Once I added this string field to the Flex client and used it as a string key on the Python server, I finally am now able to create objects and reload them on the Flex client.

The next step is to chain these objects together.  I created a SelfReferenceProperty property on the Python model class.  My class is called Block and I called this field parentBlock.  But, I cannot figure out what field(s) I am required to create on the client to manipulate this field.  I just need the parent key string.  I don't want to pass in a whole object.  Is there anyway to get this?

Am I missing something?  Perhaps I shouldn't be trying to write a simple VO class in Python to arbitrate on top of the model class?  Should I just write a dictionary instead like Fernando did in his example?

Augh!

Thanks!

Fernando Correia wrote
Thank you for the clarificatios, Nick.
_______________________________________________
PyAMF users mailing list - users@pyamf.org
http://lists.pyamf.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deep serialization of GAE models

Nick Joyce
On 22 Feb 2010, at 11:04, workie wrote:

>
> Hi,
>
> I am trying to write a basic proof of concept application and I keep falling
> short.  If I ever get it working, perhaps I'll write a tutorial for those
> who come after me.
>
> BUT, I am trying to get Flex talking to a GAE app via PyAMF.  I wrote a VO
> (value object or data transfer object) class in Flex to mirror the Python
> class, but couldn't get my app working until I found some posts discussing
> the _key field.  Once I added this string field to the Flex client and used
> it as a string key on the Python server, I finally am now able to create
> objects and reload them on the Flex client.
>
> The next step is to chain these objects together.  I created a
> SelfReferenceProperty property on the Python model class.  My class is
> called Block and I called this field parentBlock.  But, I cannot figure out
> what field(s) I am required to create on the client to manipulate this
> field.  I just need the parent key string.  I don't want to pass in a whole
> object.  Is there anyway to get this?
>
> Am I missing something?  Perhaps I shouldn't be trying to write a simple VO
> class in Python to arbitrate on top of the model class?  Should I just write
> a dictionary instead like Fernando did in his example?

There is no way to do this directly. You can, however use a python property:

class Block(db.Model):
    class __amf__:
        exclude = ('parentBlock',)

    parentBlock = db.SelfReferenceProperty()

    @property
    def parentKey(self):
        if hasattr(self.parent, 'key'):
            return str(self.parent.key())

        return None

This means that the `parentBlock` attribute will never be encoded but the `parentKey` will be.

Hope that helps,

Nick
_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deep serialization of GAE models

workie

Nick Joyce wrote
On 22 Feb 2010, at 11:04, workie wrote:

>
> Hi,
>
> I am trying to write a basic proof of concept application and I keep falling
> short.  If I ever get it working, perhaps I'll write a tutorial for those
> who come after me.
>
> BUT, I am trying to get Flex talking to a GAE app via PyAMF.  I wrote a VO
> (value object or data transfer object) class in Flex to mirror the Python
> class, but couldn't get my app working until I found some posts discussing
> the _key field.  Once I added this string field to the Flex client and used
> it as a string key on the Python server, I finally am now able to create
> objects and reload them on the Flex client.
>
> The next step is to chain these objects together.  I created a
> SelfReferenceProperty property on the Python model class.  My class is
> called Block and I called this field parentBlock.  But, I cannot figure out
> what field(s) I am required to create on the client to manipulate this
> field.  I just need the parent key string.  I don't want to pass in a whole
> object.  Is there anyway to get this?
>
> Am I missing something?  Perhaps I shouldn't be trying to write a simple VO
> class in Python to arbitrate on top of the model class?  Should I just write
> a dictionary instead like Fernando did in his example?

There is no way to do this directly. You can, however use a python property:

class Block(db.Model):
    class __amf__:
        exclude = ('parentBlock',)

    parentBlock = db.SelfReferenceProperty()

    @property
    def parentKey(self):
        if hasattr(self.parent, 'key'):
            return str(self.parent.key())

        return None

This means that the `parentBlock` attribute will never be encoded but the `parentKey` will be.

Hope that helps,

Nick
_______________________________________________
PyAMF users mailing list - users@pyamf.org
http://lists.pyamf.org/mailman/listinfo/users
Ahh, looks promising.  I am trying it right now.  Out of curiosity, why would you refer to self.parent and not self.parentBlock?  Is that a type or is there some way Python "knows" that a SelfReferenceProperty is a parent?

Thanks!!!
-Lisa
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deep serialization of GAE models

Nick Joyce
On 22 Feb 2010, at 11:35, workie wrote:

>>
>> class Block(db.Model):
>>    class __amf__:
>>        exclude = ('parentBlock',)
>>
>>    parentBlock = db.SelfReferenceProperty()
>>
>>    @property
>>    def parentKey(self):
>>        if hasattr(self.parent, 'key'):
>>            return str(self.parent.key())
>>
>>        return None
>>
>> This means that the `parentBlock` attribute will never be encoded but the
>> `parentKey` will be.
>>
>> Hope that helps,
>>
>> Nick
>> _______________________________________________
>> PyAMF users mailing list - [hidden email]
>> http://lists.pyamf.org/mailman/listinfo/users
>>
>>
>
> Ahh, looks promising.  I am trying it right now.  Out of curiosity, why
> would you refer to self.parent and not self.parentBlock?  Is that a type or
> is there some way Python "knows" that a SelfReferenceProperty is a parent?
>
> Thanks!!!
> -Lisa


Whoops, that was a typo on my part, self.parent should be self.parentBlock ..

_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deep serialization of GAE models

workie
In reply to this post by Nick Joyce

Nick Joyce wrote
Hi Fernando,

Both AMF0 and AMF3 can contain references to objects so if the  
participants of a project were rendered individually, the project  
object would only be rendered once and then references would be made  
to it (2 bytes usually).

Does that help?

On 2 Aug 2008, at 00:35, Fernando Correia wrote:

> If I have two Google App Engine model objects like these:
>
> class Project(db.Model):
>    name = db.StringProperty()
>
> class ProjectParticipant(db.Model):
>    project = db.ReferenceProperty(Project,  
> collection_name='participants')
>    name = db.StringProperty()
>
> When I return an instance of ProjectParticipant from the GAE service,
> an instance of the Project is embedded. That may not be very
> efficient. If I have 100 participants in a project, the project's data
> will be serialized 100 times...
>
> I'd rather have only the Project's key be serialized. Is there an
> option for this in PyAMF? Or is the best strategy to use a Data
> Transfer Object and copy the data from the model object to the DTO and
> return the DTO?
> _______________________________________________
> PyAMF users mailing list - users@pyamf.org
> http://lists.pyamf.org/mailman/listinfo/users


_______________________________________________
PyAMF users mailing list - users@pyamf.org
http://lists.pyamf.org/mailman/listinfo/users
Finally!  It works! It works!

FYI for those who come behind, I also wrote a setter:

    @parentKey.setter
    def parentKey(self, value):
        if value is None:
            self.parentBlock = None
        else:
            self.parentBlock = db.get(db.Key(str(value)))

Not sure if there is a better way, but this seemed to work for me.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deep serialization of GAE models

workie
oops - that didn't work when I uploaded it to appspot.com.  The logfile said that setter attrib does not exist.  Hmmm.

So far I am getting the impression that forcing a Flex class to map to a Python model class via PyAMF is a big huge pain.  Do you think it might be better to create a data dictionary and send that back and forth via PyAMF?


workie wrote
Nick Joyce wrote
Hi Fernando,

Both AMF0 and AMF3 can contain references to objects so if the  
participants of a project were rendered individually, the project  
object would only be rendered once and then references would be made  
to it (2 bytes usually).

Does that help?

On 2 Aug 2008, at 00:35, Fernando Correia wrote:

> If I have two Google App Engine model objects like these:
>
> class Project(db.Model):
>    name = db.StringProperty()
>
> class ProjectParticipant(db.Model):
>    project = db.ReferenceProperty(Project,  
> collection_name='participants')
>    name = db.StringProperty()
>
> When I return an instance of ProjectParticipant from the GAE service,
> an instance of the Project is embedded. That may not be very
> efficient. If I have 100 participants in a project, the project's data
> will be serialized 100 times...
>
> I'd rather have only the Project's key be serialized. Is there an
> option for this in PyAMF? Or is the best strategy to use a Data
> Transfer Object and copy the data from the model object to the DTO and
> return the DTO?
> _______________________________________________
> PyAMF users mailing list - users@pyamf.org
> http://lists.pyamf.org/mailman/listinfo/users


_______________________________________________
PyAMF users mailing list - users@pyamf.org
http://lists.pyamf.org/mailman/listinfo/users
Finally!  It works! It works!

FYI for those who come behind, I also wrote a setter:

    @parentKey.setter
    def parentKey(self, value):
        if value is None:
            self.parentBlock = None
        else:
            self.parentBlock = db.get(db.Key(str(value)))

Not sure if there is a better way, but this seemed to work for me.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deep serialization of GAE models

Nick Joyce
Hi,

On 1 Mar 2010, at 16:05, workie wrote:

>
> oops - that didn't work when I uploaded it to appspot.com.  The logfile said
> that setter attrib does not exist.  Hmmm.

x.setter is python2.6+ syntax only. Since AppEngine is only targetting 2.5.x specifically I would make sure that you run the dev server with the same version to ensure that kind of error doesn't occur. Saying that, I should have put 2 + 2 together and noticed that also .. *ahem*

>
> So far I am getting the impression that forcing a Flex class to map to a
> Python model class via PyAMF is a big huge pain.  Do you think it might be
> better to create a data dictionary and send that back and forth via PyAMF?

Thats a real shame! I have been using PyAMF with GAE for some time now and things generally seem to work well .. what do you think could be improved (integration, documentation, examples etc.)?

You certainly can use popos/dicts but then you have to handle the data massaging into db.Model instances yourself (assuming you want persistence of some kind). When you combine that with deeply nested object graphs, things can get pretty hairy pretty quickly..
_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deep serialization of GAE models

workie
I guess I didn't realize I was using Python 2.6.  I have both installed.  Grrr.  Configuring my Windows laptop for development was kind of a pain.  It took me a while to find an installable Windows msi for Python 2.5.

Well it's certainly encouraging that you've been doing it for some time, Nick, I really appreciate all your help.  I want to be able to get and set the key for a SelfReferenceProperty and later on I will want to do the same for a ReferenceProperty.  How do you do this?

I feel compelled to write a brief tutorial on this when I get it working.  It may seem obvious somehow to more experienced coders, but Python, PyAMF, GAE and Flex are all new for me.  

Thanks for all your help!


Nick Joyce wrote
Hi,

On 1 Mar 2010, at 16:05, workie wrote:

>
> oops - that didn't work when I uploaded it to appspot.com.  The logfile said
> that setter attrib does not exist.  Hmmm.

x.setter is python2.6+ syntax only. Since AppEngine is only targetting 2.5.x specifically I would make sure that you run the dev server with the same version to ensure that kind of error doesn't occur. Saying that, I should have put 2 + 2 together and noticed that also .. *ahem*

>
> So far I am getting the impression that forcing a Flex class to map to a
> Python model class via PyAMF is a big huge pain.  Do you think it might be
> better to create a data dictionary and send that back and forth via PyAMF?

Thats a real shame! I have been using PyAMF with GAE for some time now and things generally seem to work well .. what do you think could be improved (integration, documentation, examples etc.)?

You certainly can use popos/dicts but then you have to handle the data massaging into db.Model instances yourself (assuming you want persistence of some kind). When you combine that with deeply nested object graphs, things can get pretty hairy pretty quickly..
_______________________________________________
PyAMF users mailing list - users@pyamf.org
http://lists.pyamf.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deep serialization of GAE models

Nick Joyce
On 1 Mar 2010, at 19:20, workie wrote:

>
> I guess I didn't realize I was using Python 2.6.  I have both installed.
> Grrr.  Configuring my Windows laptop for development was kind of a pain.  It
> took me a while to find an installable Windows msi for Python 2.5.
>
> Well it's certainly encouraging that you've been doing it for some time,
> Nick, I really appreciate all your help.  I want to be able to get and set
> the key for a SelfReferenceProperty and later on I will want to do the same
> for a ReferenceProperty.  How do you do this?

An example implementation of this would be (in Python 2.5):

from google.appengine.ext import db

class Block(db.Model):
   class __amf__:
       exclude = ('parentBlock',)

   parentBlock = db.SelfReferenceProperty()

   def _get_parentKey(self):
       if hasattr(self.parentBlock, 'key'):
           return str(self.parentBlock.key())

       return None

   def _set_parentKey(self, value):
       self.parentBlock = value

   parentKey = property(_get_parentKey, _set_parentKey)

>
> I feel compelled to write a brief tutorial on this when I get it working.
> It may seem obvious somehow to more experienced coders, but Python, PyAMF,
> GAE and Flex are all new for me.  

That would be awesome, let me know if there is anything you need.

Nick
_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Deep serialization of GAE models

Alan Anderson
This post has NOT been accepted by the mailing list yet.
So maybe this is beginning to make some sense to me.  I'm going to restate what you wrote with some small changes--namely, I changed a property name from parentKey to parentBlockKeyStr to help convey that it is the string representation of a key object and not an instance of  db.Key, and I added a property named "name. "

    from google.appengine.ext import db

    class Block(db.Model):

        class __amf__:
            exclude = ('parentBlock',)

        parentBlock = db.SelfReferenceProperty()

        def _get_parentBlockKeyStr(self):
            if hasattr(self.parentBlock, 'key'):
                return str(self.parentBlock.key())
            return None

        def _set_parentBlockKeyStr(self, blockKeyStr):
            self.parentBlcokKeyStr = blockKeyStr
            self.parentBlock = db.Key(blockKeyStr) # keep parentBlock and parentBlockKeyStr consistent

        parentBlockKeyStrKey = property(_get_parentBlockKeyStr, _set_parentBlockKeyStr)

        name = db.StringProperty()


Please confirm or correct my conclusion that the corresponding Flex/ActionScript client-side class would be defined as follows.

        [RemoteClass(alias="gae.dbmodels.Block")]
        [Bindable]
        public class Block {
                public var _key: String;
                public var parentBlockKeyStr: String;
                public var name: String;

                public function Block() {
                        this._key = "";
                        }
                }

After confirming or correcting the above, please address the following.

What I really want is to simply have parentBlock return str(self.parentBlock.key())--that is, the string representation of block.parentBlock.key(), and when I change it in the Flex client code and send it back to the server to be saved to the datastore, I want the value translated back to db.Key(parentBlock). Can that be accomplished with the Iexternalize feature? If so, please provide the code for it.

Thanks.

Loading...