ctypes and byte order

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

ctypes and byte order

Jean-Michel Pichavant
Hi list,

I'm currently writing python code that writes a small binary file to be used by another device which code is written in C.
The python code runs on a little endian CPU, and unfortunately, the other device is using a big endian MIPS.

My problem is the following: I cannot make an array of n int work in big endian, I can workaround the problem by creating n intergers, which is nicely done thanks to list comprehension but that seems to me like a misuse of the ctypes modules.

ctypes is expecting a 'c_uint_be_Array_5' (note the _be_ for big endian), and I don't know how to construct such object.

I hope I am clear.

Thanks,

JM

here's a code sample (python 2.7):

--------------------
import ctypes, sys

iarray = ctypes.c_uint32*5  # array of 5 unsigned 32 bits

class Foo(ctypes.Structure):
        """Native byte order, won't work on my big endian device"""
        _fields_ = [('bar', iarray)]

class Foo_be(ctypes.BigEndianStructure):
        """big endian version, but python code fails"""
        _fields_ = [('bar', iarray)]

class Foo_be_working(ctypes.BigEndianStructure):
        """Working big endian version, looks more like a workaround."""
        _fields_ = [('bar%i'%i, ctypes.c_uint32) for i in range(5)]


print sys.version
f = Foo(iarray(0,1,2,3,4))
print "uint32 array: ", f.bar

f_be = Foo_be_working(0,1,2,3,4)
print "bar0 and bar5: ", f_be.bar0, f_be.bar5

f_be = Foo_be(iarray(0,1,2,3,4)) # will raise an exception

--------------------


The output

2.7.3 (default, Mar 13 2014, 11:03:55)
[GCC 4.7.2]
uint32 array:  <__main__.c_uint_Array_5 object at 0x1cf4560>
bar0 and bar4:  0 4


TypeError: incompatible types, c_uint_Array_5 instance instead of c_uint_be_Array_5 instance
     24
---> 25 f_be = Foo_be(iarray(0,1,2,3,4))
     26



-- IMPORTANT NOTICE:

The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Reply | Threaded
Open this post in threaded view
|

ctypes and byte order

Peter Otten
Jean-Michel Pichavant wrote:

> I'm currently writing python code that writes a small binary file to be
> used by another device which code is written in C. The python code runs on
> a little endian CPU, and unfortunately, the other device is using a big
> endian MIPS.
>
> My problem is the following: I cannot make an array of n int work in big
> endian, I can workaround the problem by creating n intergers, which is
> nicely done thanks to list comprehension but that seems to me like a
> misuse of the ctypes modules.
>
> ctypes is expecting a 'c_uint_be_Array_5' (note the _be_ for big endian),
> and I don't know how to construct such object.
>
> I hope I am clear.

Yes, very clear, thanks for the effort, although the nitpicker in me has to
mention that an off-by-one bug slipped into your code ;)

> class Foo_be_working(ctypes.BigEndianStructure):
>     """Working big endian version, looks more like a workaround."""
>     _fields_ = [('bar%i'%i, ctypes.c_uint32) for i in range(5)]
[...]
> f_be = Foo_be_working(0,1,2,3,4)
> print "bar0 and bar5: ", f_be.bar0, f_be.bar5

I'm not very familiar with ctypes, but a random walk through the source
found the _endian module and the __ctypes_be__ attribute. So

>
> iarray = ctypes.c_uint32*5  # array of 5 unsigned 32 bits
>
> class Foo(ctypes.Structure):
>     """Native byte order, won't work on my big endian device"""
>     _fields_ = [('bar', iarray)]
>
> class Foo_be(ctypes.BigEndianStructure):
>     """big endian version, but python code fails"""
>     _fields_ = [('bar', iarray)]

becomes

$ cat be2.py
import ctypes, sys

iarray_be = ctypes.c_uint32.__ctype_be__*5

class Foo_be(ctypes.BigEndianStructure):
        _fields_ = [('bar', iarray_be)]

print sys.version
f_be = Foo_be((0,1,2,3,0x11223344))
print hex(f_be.bar[4])

$ python be2.py
2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2]
0x11223344L

which might do what you want.


Reply | Threaded
Open this post in threaded view
|

ctypes and byte order

Jean-Michel Pichavant
----- Original Message -----

> From: "Peter Otten" <__peter__ at web.de>
> becomes
>
> $ cat be2.py
> import ctypes, sys
>
> iarray_be = ctypes.c_uint32.__ctype_be__*5
>
> class Foo_be(ctypes.BigEndianStructure):
>         _fields_ = [('bar', iarray_be)]
>
> print sys.version
> f_be = Foo_be((0,1,2,3,0x11223344))
> print hex(f_be.bar[4])
>
> $ python be2.py
> 2.7.6 (default, Mar 22 2014, 22:59:56)
> [GCC 4.8.2]
> 0x11223344L
>
> which might do what you want.

Brilliant !

I've tested it and it yields the exact same results (binary file content wise) than my "workaround" structure.
But that's way better since my actual structure is more complex and arrays will be required.

Though I'm slightly puzzled by the ctypes author(s) choice, this is not documented and requires to peek into the source code. Dunder attributes are rarely part of an interface.

Anyway, thanks for your help !

JM



-- IMPORTANT NOTICE:

The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Reply | Threaded
Open this post in threaded view
|

ctypes and byte order

Peter Otten
Jean-Michel Pichavant wrote:

> ----- Original Message -----
>> From: "Peter Otten" <__peter__ at web.de>
>> becomes
>>
>> $ cat be2.py
>> import ctypes, sys
>>
>> iarray_be = ctypes.c_uint32.__ctype_be__*5
>>
>> class Foo_be(ctypes.BigEndianStructure):
>>         _fields_ = [('bar', iarray_be)]
>>
>> print sys.version
>> f_be = Foo_be((0,1,2,3,0x11223344))
>> print hex(f_be.bar[4])
>>
>> $ python be2.py
>> 2.7.6 (default, Mar 22 2014, 22:59:56)
>> [GCC 4.8.2]
>> 0x11223344L
>>
>> which might do what you want.
>
> Brilliant !
>
> I've tested it and it yields the exact same results (binary file content
> wise) than my "workaround" structure. But that's way better since my
> actual structure is more complex and arrays will be required.
>
> Though I'm slightly puzzled by the ctypes author(s) choice, this is not
> documented and requires to peek into the source code. Dunder attributes
> are rarely part of an interface.

On further reflection it dawned on me that what may have made this work is
passing in the tuple instead of an array:

>>> import ctypes
>>> array = ctypes.c_uint32 * 3
>>> class B(ctypes.BigEndianStructure):
...     _fields_ = [("bar", array)]
...
>>> b = B((1,2,3))
>>> b.bar
<ctypes._endian.c_uint_be_Array_3 object at 0x7f87700798c0>
>>> b.bar[2]
3L
>>> b = B(array(1,2,3))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: incompatible types, c_uint_Array_3 instance instead of
c_uint_be_Array_3 instance

Oops...

So the only thing that doesn't work transparently is initialising a data
structure (or at least array) from its "other-endian" twin.




Reply | Threaded
Open this post in threaded view
|

ctypes and byte order

Terry Reedy
In reply to this post by Jean-Michel Pichavant
On 6/18/2015 5:39 AM, Jean-Michel Pichavant wrote:

> I'm currently writing python code that writes a small binary file to
> be used by another device which code is written in C. The python code
> runs on a little endian CPU, and unfortunately, the other device is
> using a big endian MIPS.

The struct module is designed for this.  It reads and writes packed
binary data of various types and sizes in either big or little endian
order.  It should be easier than ctypes.

--
Terry Jan Reedy


Reply | Threaded
Open this post in threaded view
|

ctypes and byte order

Jean-Michel Pichavant
----- Original Message -----

> From: "Terry Reedy" <tjreedy at udel.edu>
> To: python-list at python.org
> Sent: Thursday, 18 June, 2015 7:02:16 PM
> Subject: Re: ctypes and byte order
>
> On 6/18/2015 5:39 AM, Jean-Michel Pichavant wrote:
>
> > I'm currently writing python code that writes a small binary file
> > to
> > be used by another device which code is written in C. The python
> > code
> > runs on a little endian CPU, and unfortunately, the other device is
> > using a big endian MIPS.
>
> The struct module is designed for this.  It reads and writes packed
> binary data of various types and sizes in either big or little endian
> order.  It should be easier than ctypes.
>
> --
> Terry Jan Reedy

Yep, I knew about struct but I did go for ctypes because... I wanted to try it and thought it would be a suitable solution (with more capabilities).
Looks like I fooled myself (not in the way ctypes is badly designed, it's just not what I needed).

Now I'm left with a dilemma:

My current ctypes code is working (thanks Peter !), but the code is rather cumbersome.

1/ don't change it, and wish good luck to my next co-worker looking at the code
2/ rewrite everything with struct
3/ rewrite everything with cffi (Laura's suggestion) which has the awsomeness to suppport C code copy/paste. It's still a "C Foreign Function Interface" like ctypes and could be overkill


We all know how this ends up...

JM


-- IMPORTANT NOTICE:

The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.