Looking for help with plain/binary serialization

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Looking for help with plain/binary serialization

Ben Sizer
Hello all,

I'm trying to write an application where I send raw data from
Actionscript to Python and back again. Now the Python to Actionscript
direction is documented in
\doc\tutorials\examples\actionscript\socket, and that seems to work ok
- I use writeElement in Python and readObject at the other end and
it's fine. But I can't find good docs for the reverse direction:
sending objects from Flash to Python.

I'm trying this on the AS side (abridged code):

    objectEncoding = ObjectEncoding.AMF0;
    private static const MSG_START: int = 1;
    this.writeObject(MSG_START);
    flush();

And this on the Python receiving end (using the Twisted server framework):

    def dataReceived(self, data):
        byte_stream = pyamf.util.BufferedByteStream(data)
        amf_object = pyamf.decode(byte_stream)
        message_type = amf_object.next() # should be an integer?
        print "Message type in: %s" % (message_type)

But rather than seeing the "Message type in: 1" as I would expect, I'm
getting "Message type in: pyamf.Undefined". Presumably I am doing
something wrong. And I have no idea how to even begin to debug this or
where to start looking at doing things differently.

On top of this, I know there is a potential bug waiting - if the
'data' doesn't contain a full object, presumably the decode() call
will fail, but I don't know what it will do instead. (Return None?
Raise an exception?) I presume I can just keep appending to a
BufferedByteStream each time instead, and attempting to read from it,
but if I do manage to decode an object from the stream, does that
modify the stream to remove that object so that it's ready for the
next object (which may already be partially in the buffer)? I can't
find docs on this.

Any advice or pointers would be gratefully welcomed!

--
Ben Sizer
_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Looking for help with plain/binary serialization

Nick Joyce
On 30 Sep 2010, at 10:19, Ben Sizer wrote:

> Hello all,
>
> I'm trying to write an application where I send raw data from
> Actionscript to Python and back again. Now the Python to Actionscript
> direction is documented in
> \doc\tutorials\examples\actionscript\socket, and that seems to work ok
> - I use writeElement in Python and readObject at the other end and
> it's fine. But I can't find good docs for the reverse direction:
> sending objects from Flash to Python.
>
> I'm trying this on the AS side (abridged code):
>
>    objectEncoding = ObjectEncoding.AMF0;
>    private static const MSG_START: int = 1;
>    this.writeObject(MSG_START);
>    flush();
>
> And this on the Python receiving end (using the Twisted server framework):
>
>    def dataReceived(self, data):
>        byte_stream = pyamf.util.BufferedByteStream(data)
>        amf_object = pyamf.decode(byte_stream)
>        message_type = amf_object.next() # should be an integer?
>        print "Message type in: %s" % (message_type)
>
> But rather than seeing the "Message type in: 1" as I would expect, I'm
> getting "Message type in: pyamf.Undefined". Presumably I am doing
> something wrong. And I have no idea how to even begin to debug this or
> where to start looking at doing things differently.

AMF3 is the default version for PyAMF to attempt any encode/decode. AMF0's int type is the same as AMF3's 'undefined' type which is why you're getting 'pyamf.Undefined'.

Switch the pyamf.decode(byte_stream, encoding=pyamf.AMF0) and you should get the results you expect.

The usual tools for working with AMF (like Charles/Fiddler/etc.) tend to not help in these situations (because they assume you're using HTTP) and so falling back to Wireshark (or some sort of packet inspection tool) helps a lot in these situations (indeed it has been my trusted friend whilst working on RTMPy recently).

>
> On top of this, I know there is a potential bug waiting - if the
> 'data' doesn't contain a full object, presumably the decode() call
> will fail, but I don't know what it will do instead. (Return None?
> Raise an exception?) I presume I can just keep appending to a
> BufferedByteStream each time instead, and attempting to read from it,
> but if I do manage to decode an object from the stream, does that
> modify the stream to remove that object so that it's ready for the
> next object (which may already be partially in the buffer)? I can't
> find docs on this.

As you rightly pointed out, dataReceived is not guaranteed to contain complete AMF objects so you need to buffer them.

Check out http://gist.github.com/603940 as a possible way to deal with this issue. The PyAMF En/Decoders will raise IOError if they cannot read enough data from the underlying stream otherwise will raise pyamf.DecodeError (in the case of decoding) if some malformed data was found and then any more basic errors (MemoryError, etc. etc.)

Hope that helps.

Nick
_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Looking for help with plain/binary serialization

Ben Sizer
On 30 September 2010 03:56, Nick Joyce <[hidden email]> wrote:
>
> AMF3 is the default version for PyAMF to attempt any encode/decode. AMF0's int type is the same as AMF3's 'undefined' type which is why you're getting 'pyamf.Undefined'.

Thanks Nick, that was very helpful. Is there any good reason for me to
use AMF0 rather than AMF3? The socket example uses AMF0 - is that just
for backwards compatibility?

> As you rightly pointed out, dataReceived is not guaranteed to contain complete AMF objects so you need to buffer them.
>
> Check out http://gist.github.com/603940 as a possible way to deal with this issue. The PyAMF En/Decoders will raise IOError if they cannot read enough data from the underlying stream otherwise will raise pyamf.DecodeError (in the case of decoding) if some malformed data was found and then any more basic errors (MemoryError, etc. etc.)

Ok, noted. A first draft of this approach seems to work just fine.

One more question, slightly related: I want to send a variety of
message types between the 2 applications, and bandwidth is a concern,
but so is code complexity. If I have several message types derived
from a common base, how efficient is it to send those as whole
objects, and is there any support in PyAMF for reconstructing them in
a form similar to the one they take in Actionscript? Am I better off
just explicitly sending a message type followed by bespoke data, and
reading it back in similar fashion? (And does PyAMF buy me much in
this situation? I'm not entirely sure what the encoding and decoding
does in this case, if I know I am just writing integers, bytes, etc.)

--
Ben Sizer
_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Looking for help with plain/binary serialization

Nick Joyce
On 5 Oct 2010, at 06:52, Ben Sizer wrote:

> On 30 September 2010 03:56, Nick Joyce <[hidden email]> wrote:
>>
>> AMF3 is the default version for PyAMF to attempt any encode/decode. AMF0's int type is the same as AMF3's 'undefined' type which is why you're getting 'pyamf.Undefined'.
>
> Thanks Nick, that was very helpful. Is there any good reason for me to
> use AMF0 rather than AMF3? The socket example uses AMF0 - is that just
> for backwards compatibility?

I suppose the only necessary reason for using AMF0 over AMF3 is that you need to support Flash <= 8. If I had a choice, using AMF3 make sense.

>
>> As you rightly pointed out, dataReceived is not guaranteed to contain complete AMF objects so you need to buffer them.
>>
>> Check out http://gist.github.com/603940 as a possible way to deal with this issue. The PyAMF En/Decoders will raise IOError if they cannot read enough data from the underlying stream otherwise will raise pyamf.DecodeError (in the case of decoding) if some malformed data was found and then any more basic errors (MemoryError, etc. etc.)
>
> Ok, noted. A first draft of this approach seems to work just fine.
>
> One more question, slightly related: I want to send a variety of
> message types between the 2 applications, and bandwidth is a concern,
> but so is code complexity. If I have several message types derived
> from a common base, how efficient is it to send those as whole
> objects, and is there any support in PyAMF for reconstructing them in
> a form similar to the one they take in Actionscript?

A large portion of the PyAMF code base is dedicated to this. If you haven't already, you should check out the docs [0]. We have spent a substantial amount of time ensuring that moving object graphs to/from PyAMF is as seamless/easy as possible.

> Am I better off just explicitly sending a message type followed by bespoke data, and
> reading it back in similar fashion? (And does PyAMF buy me much in
> this situation? I'm not entirely sure what the encoding and decoding
> does in this case, if I know I am just writing integers, bytes, etc.)

If you _really_ need to minimise the size of your message payloads, you can use IExternalizable [1] which allows you to have full control of the exact bytes that you write/read. How to use it in PyAMF [2].

We implement Flex Messaging's ISmallMessage in PyAMF (somewhat clumsily), which uses the alias 'DSK', 'DSC' etc. the source of which is available at [3].

Hth

[0] - http://pyamf.org/architecture/attributecontrol.html
[1] - http://livedocs.adobe.com/flex/3/langref/flash/utils/IExternalizable.html
[2] - http://stackoverflow.com/questions/2951494/how-can-i-map-field-names-between-django-pyamf-and-flex
[3] - http://github.com/hydralabs/pyamf/blob/master/pyamf/flex/messaging.py
_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Looking for help with plain/binary serialization

Ben Sizer
On 5 October 2010 08:49, Nick Joyce <[hidden email]> wrote:

> On 5 Oct 2010, at 06:52, Ben Sizer wrote:
>
>>
>> One more question, slightly related: I want to send a variety of
>> message types between the 2 applications, and bandwidth is a concern,
>> but so is code complexity. If I have several message types derived
>> from a common base, how efficient is it to send those as whole
>> objects, and is there any support in PyAMF for reconstructing them in
>> a form similar to the one they take in Actionscript?
>
> A large portion of the PyAMF code base is dedicated to this. If you haven't already, you should check out the docs [0]. We have spent a substantial amount of time ensuring that moving object graphs to/from PyAMF is as seamless/easy as possible.

Hey Nick,

I have looked over the docs but there are far too many to read them
all, and I'm at a disadvantage as I don't know what I'm looking for.
In particular I would never have known I needed a page titled
"Attribute Control"! And it's not clear exactly what I'm supposed to
be looking at: is it the concept of creating Python classes that map
to AS3 classes that is important, and calling pyamf.register_class
(with the AS3 class name?) so that Python can read them?

I expect the docs work well for someone who already has experience of
using all the remoting stuff with other platforms but from the view of
someone who's never done this before and doesn't want to set up remote
services or use Django/wsgi/whatever it's quite inpenetrable. It's
almost like the framework is too good at handling these awkward and
specific cases that it makes it hard to find out about the more
general case. I'm not using HTTP at all but the examples given seem to
be all web-based with the exception of the 'socket' example which
makes you pull things out piecemeal. If there's a specific example of
the middle ground, that would be helpful for people in my position.

> If you _really_ need to minimise the size of your message payloads, you can use IExternalizable [1] which allows you to have full control of the exact bytes that you write/read. How to use it in PyAMF [2].
>
> We implement Flex Messaging's ISmallMessage in PyAMF (somewhat clumsily), which uses the alias 'DSK', 'DSC' etc. the source of which is available at [3].

Yes, minimal message sizes are mandatory in my situation.
IExternalizable looks handy. Is there any more information about
ISmallMessage? The PyAMF site ranks more highly than the Adobe site
for this search term! Which isn't surprising when you see how lacking
the official docs on this interface are.

Thanks for your help,

Ben Sizer
_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Looking for help with plain/binary serialization

Thijs Triemstra
Hi Ben,

On 5 Oct 2010, at 11:20, Ben Sizer wrote:

I expect the docs work well for someone who already has experience of
using all the remoting stuff with other platforms but from the view of
someone who's never done this before and doesn't want to set up remote
services or use Django/wsgi/whatever it's quite inpenetrable. I

Any suggestions to improve http://pyamf.org/tutorials/index.html are welcome. From what you write that page sounds usable, there's the "Getting started with AMF for Django" reference etc..

Cheers,

Thijs

_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users

PGP.sig (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Looking for help with plain/binary serialization

Ben Sizer
On 13 October 2010 20:37, Thijs Triemstra | Collab <[hidden email]> wrote:

> On 5 Oct 2010, at 11:20, Ben Sizer wrote:
>
> > I expect the docs work well for someone who already has experience of
> > using all the remoting stuff with other platforms but from the view of
> > someone who's never done this before and doesn't want to set up remote
> > services or use Django/wsgi/whatever it's quite inpenetrable. I
>
> Any suggestions to improve¬†http://pyamf.org/tutorials/index.html are
> welcome. From what you write that page sounds usable, there's the "Getting
> started with AMF for Django" reference etc..

I've not had a chance to work on this since my last email but I can
see several problems with the docs generally:

 - THE SEARCH DOESN'T WORK! It hits an error, but then appears to
still be doing something, which is awful. (A sample Javascript error
in Firefox is "Error: Search.loadIndex is not a function Source File:
http://pyamf.org/search.html?q=pyamf   Line: 33") If it doesn't work,
take it off the site. It just makes people like me get frustrated,
waiting for a result that will never arrive.

 - The emphasis in the docs seems to be all on getting Python objects
to Actionscript, eg. serving up ORM results. But why is there no
example of it working the other way around? After all, if AMF is your
default data format then you'd think the data originated in
Actionscript. Nick gave me a link to
http://pyamf.org/architecture/attributecontrol.html but that doesn't
say anything about sending things from AS3->Python.

 - The 'architecture' page says nothing about why those links are
grouped together or what they are about. Yet each link is to something
very useful. Why is important information such as the Attribute
Control page hidden under 'architecture'?  It looks like the pages
linked here should be at the very front of the documentation, not
buried down after everything else.

 - The example also includes no details about what you will receive at
the ActionScript side. Maybe that is documented elsewhere, but there
should be a link to it. At one point there is a mention of
'flash.net.registerClassAlias' but that is not documented here either.

 - The attribute control example is associated with the Google App
Engine. This makes it harder to follow the example because you have 2
concepts introduced in one page: the GAE stuff and the object
serialisation stuff. Each example should really only introduce one new
concept to begin with, so this page should start off with a simple
Python object, and then move on to showing how it works with GAE or
Django model objects later.

 - The website insists on serving the example source code to me as
downloadable files rather than files I can view online. This makes it
awkward to browse through them.

 - The API reference contains little to no information about which
methods you are supposed to be able to use and which ones you aren't
(which is why I ended up creating http://dev.pyamf.org/ticket/808 ) -
this means you have to dig through tutorial after tutorial to try and
find all the allowed methods.

Hopefully that is some constructive criticism! At the moment I still
can't see from these docs how I can get some simple message objects
from AS3 to Python and back again via binary sockets, without writing
things out manually field by field.

--
Ben Sizer
_______________________________________________
PyAMF users mailing list - [hidden email]
http://lists.pyamf.org/mailman/listinfo/users