Writing dual python2/python3 code and the use of PyQt5.uic

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Writing dual python2/python3 code and the use of PyQt5.uic

Eli Schwartz
I've been devoting some time recently to incrementally porting a PyQt5
application "calibre" (https://github.com/kovidgoyal/calibre/) from
python2 to polyglot code, hoping to get it running with python3 at some
point. One recent stumbling block I hit was that it uses the uic module
to compile forms into a cStringIO.StringIO.

What I wanted to do was switch it to using the modern io interface, but
as it turns out, PyQt5 uses unqualified "str" instances to write out the
form. As a result, I cannot use io.StringIO on python2, and I cannot use
io.BytesIO on python3.

The uic module quite specifically seems to expect that it will
manipulate unicode strings. I was able to get io.StringIO buffers to
work correctly with both python2 and python3 after I modified two files
to contain a "from __future__ import unicode_literals":

/usr/lib/python2.7/site-packages/PyQt5/uic/__init__.py
/usr/lib/python2.7/site-packages/PyQt5/uic/Compiler/indenter.py

Can this be added to at least these two files in the official
distribution? It seems to be a mistake that it ever assumed the str type.

--
Eli Schwartz
Arch Linux Bug Wrangler and Trusted User


_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt

signature.asc (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Writing dual python2/python3 code and the use of PyQt5.uic

Eli Schwartz
On 3/24/19 6:07 AM, Phil Thompson wrote:

> On 24 Mar 2019, at 1:48 am, Eli Schwartz <[hidden email]> wrote:
>>
>> I've been devoting some time recently to incrementally porting a PyQt5
>> application "calibre" (https://github.com/kovidgoyal/calibre/) from
>> python2 to polyglot code, hoping to get it running with python3 at some
>> point. One recent stumbling block I hit was that it uses the uic module
>> to compile forms into a cStringIO.StringIO.
>>
>> What I wanted to do was switch it to using the modern io interface, but
>> as it turns out, PyQt5 uses unqualified "str" instances to write out the
>> form. As a result, I cannot use io.StringIO on python2, and I cannot use
>> io.BytesIO on python3.
>>
>> The uic module quite specifically seems to expect that it will
>> manipulate unicode strings. I was able to get io.StringIO buffers to
>> work correctly with both python2 and python3 after I modified two files
>> to contain a "from __future__ import unicode_literals":
>>
>> /usr/lib/python2.7/site-packages/PyQt5/uic/__init__.py
>> /usr/lib/python2.7/site-packages/PyQt5/uic/Compiler/indenter.py
>>
>> Can this be added to at least these two files in the official
>> distribution? It seems to be a mistake that it ever assumed the str type.
>
> This...
>
> https://python-future.org/unicode_literals.html
>
> ...makes me nervous. I would prefer a more specific fix.
The alternative is to add u'' decorations to the specific strings that
cannot be python2's dual purpose str/bytes.

I *think* this should be sufficient:
https://github.com/eli-schwartz/pyqt5/commit/1a014130024530eb0241cec7ef7ffb82e2c69fc6

patch:
https://github.com/eli-schwartz/pyqt5/commit/1a014130024530eb0241cec7ef7ffb82e2c69fc6.patch

(Applied on top of a tarball import as I could not find any version
control sources for PyQt5.)

--
Eli Schwartz
Arch Linux Bug Wrangler and Trusted User


_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt

signature.asc (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Writing dual python2/python3 code and the use of PyQt5.uic

Phil Thompson-5
On 24 Mar 2019, at 1:59 pm, Eli Schwartz <[hidden email]> wrote:

>
> On 3/24/19 6:07 AM, Phil Thompson wrote:
>> On 24 Mar 2019, at 1:48 am, Eli Schwartz <[hidden email]> wrote:
>>>
>>> I've been devoting some time recently to incrementally porting a PyQt5
>>> application "calibre" (https://github.com/kovidgoyal/calibre/) from
>>> python2 to polyglot code, hoping to get it running with python3 at some
>>> point. One recent stumbling block I hit was that it uses the uic module
>>> to compile forms into a cStringIO.StringIO.
>>>
>>> What I wanted to do was switch it to using the modern io interface, but
>>> as it turns out, PyQt5 uses unqualified "str" instances to write out the
>>> form. As a result, I cannot use io.StringIO on python2, and I cannot use
>>> io.BytesIO on python3.
>>>
>>> The uic module quite specifically seems to expect that it will
>>> manipulate unicode strings. I was able to get io.StringIO buffers to
>>> work correctly with both python2 and python3 after I modified two files
>>> to contain a "from __future__ import unicode_literals":
>>>
>>> /usr/lib/python2.7/site-packages/PyQt5/uic/__init__.py
>>> /usr/lib/python2.7/site-packages/PyQt5/uic/Compiler/indenter.py
>>>
>>> Can this be added to at least these two files in the official
>>> distribution? It seems to be a mistake that it ever assumed the str type.
>>
>> This...
>>
>> https://python-future.org/unicode_literals.html
>>
>> ...makes me nervous. I would prefer a more specific fix.
>
> The alternative is to add u'' decorations to the specific strings that
> cannot be python2's dual purpose str/bytes.
>
> I *think* this should be sufficient:
> https://github.com/eli-schwartz/pyqt5/commit/1a014130024530eb0241cec7ef7ffb82e2c69fc6
>
> patch:
> https://github.com/eli-schwartz/pyqt5/commit/1a014130024530eb0241cec7ef7ffb82e2c69fc6.patch
>
> (Applied on top of a tarball import as I could not find any version
> control sources for PyQt5.)

What about Python v3.3 and earlier?

Phil
_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Writing dual python2/python3 code and the use of PyQt5.uic

Eli Schwartz
On 3/24/19 10:18 AM, Phil Thompson wrote:

> On 24 Mar 2019, at 1:59 pm, Eli Schwartz <[hidden email]> wrote:
>>
>> On 3/24/19 6:07 AM, Phil Thompson wrote:
>>> On 24 Mar 2019, at 1:48 am, Eli Schwartz <[hidden email]> wrote:
>>>>
>>>> I've been devoting some time recently to incrementally porting a PyQt5
>>>> application "calibre" (https://github.com/kovidgoyal/calibre/) from
>>>> python2 to polyglot code, hoping to get it running with python3 at some
>>>> point. One recent stumbling block I hit was that it uses the uic module
>>>> to compile forms into a cStringIO.StringIO.
>>>>
>>>> What I wanted to do was switch it to using the modern io interface, but
>>>> as it turns out, PyQt5 uses unqualified "str" instances to write out the
>>>> form. As a result, I cannot use io.StringIO on python2, and I cannot use
>>>> io.BytesIO on python3.
>>>>
>>>> The uic module quite specifically seems to expect that it will
>>>> manipulate unicode strings. I was able to get io.StringIO buffers to
>>>> work correctly with both python2 and python3 after I modified two files
>>>> to contain a "from __future__ import unicode_literals":
>>>>
>>>> /usr/lib/python2.7/site-packages/PyQt5/uic/__init__.py
>>>> /usr/lib/python2.7/site-packages/PyQt5/uic/Compiler/indenter.py
>>>>
>>>> Can this be added to at least these two files in the official
>>>> distribution? It seems to be a mistake that it ever assumed the str type.
>>>
>>> This...
>>>
>>> https://python-future.org/unicode_literals.html
>>>
>>> ...makes me nervous. I would prefer a more specific fix.
>>
>> The alternative is to add u'' decorations to the specific strings that
>> cannot be python2's dual purpose str/bytes.
>>
>> I *think* this should be sufficient:
>> https://github.com/eli-schwartz/pyqt5/commit/1a014130024530eb0241cec7ef7ffb82e2c69fc6
>>
>> patch:
>> https://github.com/eli-schwartz/pyqt5/commit/1a014130024530eb0241cec7ef7ffb82e2c69fc6.patch
>>
>> (Applied on top of a tarball import as I could not find any version
>> control sources for PyQt5.)
>
> What about Python v3.3 and earlier?
python 3.3 was in security-fixes-only mode since March 2014, and it
finally reached End Of Life a year and a half ago, in September 2017.

IMO there is no reason to support EOL versions of python 3; everyone,
without exception, should be using newer versions of python 3. Even
python 3.4 is a big improvement on python 3.3, and that was EOL a few
days ago.

Either way, however, PEP 414 which added u'' strings back to python 3.x
was accepted for python 3.3, not 3.4, so it should work there too.

If a downstream project explicitly needs support for 3.2, which saw its
last source-only release in 2014 and was fully EOL in February 2016...
well, that is a very unfortunate development choice and I would strongly
urge such people to upgrade to a somewhat modern version of python (or
downgrade to 2.7 as that is still supported for another year).

Does PyQt5 have a policy for the minimum supported version of python 3?
Is it sufficient to only support versions of python that are still
supported by the Python Software Foundation?

The README specifies "2.6 or later", but it seems possible that the
README wording is simply unintended. If that means development must
remain compatible with 3.0 then that locks out a lot of options. And I'm
not even sure how much adoption python 3 actually had at that stage. o_O

--
Eli Schwartz
Arch Linux Bug Wrangler and Trusted User


_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt

signature.asc (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Writing dual python2/python3 code and the use of PyQt5.uic

Phil Thompson-5
On 24 Mar 2019, at 3:44 pm, Eli Schwartz <[hidden email]> wrote:

>
> On 3/24/19 10:18 AM, Phil Thompson wrote:
>> On 24 Mar 2019, at 1:59 pm, Eli Schwartz <[hidden email]> wrote:
>>>
>>> On 3/24/19 6:07 AM, Phil Thompson wrote:
>>>> On 24 Mar 2019, at 1:48 am, Eli Schwartz <[hidden email]> wrote:
>>>>>
>>>>> I've been devoting some time recently to incrementally porting a PyQt5
>>>>> application "calibre" (https://github.com/kovidgoyal/calibre/) from
>>>>> python2 to polyglot code, hoping to get it running with python3 at some
>>>>> point. One recent stumbling block I hit was that it uses the uic module
>>>>> to compile forms into a cStringIO.StringIO.
>>>>>
>>>>> What I wanted to do was switch it to using the modern io interface, but
>>>>> as it turns out, PyQt5 uses unqualified "str" instances to write out the
>>>>> form. As a result, I cannot use io.StringIO on python2, and I cannot use
>>>>> io.BytesIO on python3.
>>>>>
>>>>> The uic module quite specifically seems to expect that it will
>>>>> manipulate unicode strings. I was able to get io.StringIO buffers to
>>>>> work correctly with both python2 and python3 after I modified two files
>>>>> to contain a "from __future__ import unicode_literals":
>>>>>
>>>>> /usr/lib/python2.7/site-packages/PyQt5/uic/__init__.py
>>>>> /usr/lib/python2.7/site-packages/PyQt5/uic/Compiler/indenter.py
>>>>>
>>>>> Can this be added to at least these two files in the official
>>>>> distribution? It seems to be a mistake that it ever assumed the str type.
>>>>
>>>> This...
>>>>
>>>> https://python-future.org/unicode_literals.html
>>>>
>>>> ...makes me nervous. I would prefer a more specific fix.
>>>
>>> The alternative is to add u'' decorations to the specific strings that
>>> cannot be python2's dual purpose str/bytes.
>>>
>>> I *think* this should be sufficient:
>>> https://github.com/eli-schwartz/pyqt5/commit/1a014130024530eb0241cec7ef7ffb82e2c69fc6
>>>
>>> patch:
>>> https://github.com/eli-schwartz/pyqt5/commit/1a014130024530eb0241cec7ef7ffb82e2c69fc6.patch
>>>
>>> (Applied on top of a tarball import as I could not find any version
>>> control sources for PyQt5.)
>>
>> What about Python v3.3 and earlier?
>
> python 3.3 was in security-fixes-only mode since March 2014, and it
> finally reached End Of Life a year and a half ago, in September 2017.
>
> IMO there is no reason to support EOL versions of python 3; everyone,
> without exception, should be using newer versions of python 3. Even
> python 3.4 is a big improvement on python 3.3, and that was EOL a few
> days ago.
>
> Either way, however, PEP 414 which added u'' strings back to python 3.x
> was accepted for python 3.3, not 3.4, so it should work there too.
>
> If a downstream project explicitly needs support for 3.2, which saw its
> last source-only release in 2014 and was fully EOL in February 2016...
> well, that is a very unfortunate development choice and I would strongly
> urge such people to upgrade to a somewhat modern version of python (or
> downgrade to 2.7 as that is still supported for another year).
>
> Does PyQt5 have a policy for the minimum supported version of python 3?
> Is it sufficient to only support versions of python that are still
> supported by the Python Software Foundation?
>
> The README specifies "2.6 or later", but it seems possible that the
> README wording is simply unintended. If that means development must
> remain compatible with 3.0 then that locks out a lot of options. And I'm
> not even sure how much adoption python 3 actually had at that stage. o_O

I completely agree with the points you make. However the current PyQt policy is to support all versions of Python v3. This sort of committment is particularly important to commercial customers.

Actually this is a good time to review this - given the imminent EOL of Python v2.

I'll put out a draft devised policy (based on EOL dates) for comment in the next day or so.

Phil
_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Writing dual python2/python3 code and the use of PyQt5.uic

Eli Schwartz
On 3/24/19 3:06 PM, Phil Thompson wrote:
> I completely agree with the points you make. However the current PyQt
> policy is to support all versions of Python v3. This sort of
> committment is particularly important to commercial customers.

Yes, if there is a clear current commitment to support these versions
then indeed you cannot just cut off support without warning. Thanks for
clarifying.

> Actually this is a good time to review this - given the imminent EOL
> of Python v2.
>
> I'll put out a draft devised policy (based on EOL dates) for comment
> in the next day or so.

That seems like a useful policy to establish. Especially with regard to
older versions of a major release cycle. Of course I do have a biased
(python2.7+, python3.5+ strictly open-source) perspective. :)

...

As far as the original topic of uic using str(), in order to solve this
while not using unicode_literals everywhere, I can see three solutions:

- have separate python2 and python3 versions of these files which use
u'' on python2, but retain 3.2 compatibility by having plain '' strings
in the python3 version

- add a conditional check for sys.version_info and determine whether to
call unicode() on the strings (slower than using u'', but the speed is
probably negligible...)

- moving some code to something like PyQt5/uic/unicode_constants.py, and
using 'from __future__ import unicode_literals' to change only those
files to use native unicode types, which should work on both python2 and
python3 without affecting extra things like docstrings as it will only
affect that file.

And option four:
- make a note to fix this, only after an EOL policy is devised and
deployed which allows you to stop supporting 3.0-3.2 that do not support
u'' strings.

Do any of these options seem like a good idea to you? I'd probably go
with the if sys.version_info[:1] == (2,): var = unicode(var)


--
Eli Schwartz
Bug Wrangler and Trusted User


_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt

signature.asc (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Writing dual python2/python3 code and the use of PyQt5.uic

Phil Thompson-5
On 25 Mar 2019, at 3:55 am, Eli Schwartz <[hidden email]> wrote:

>
> On 3/24/19 3:06 PM, Phil Thompson wrote:
>> I completely agree with the points you make. However the current PyQt
>> policy is to support all versions of Python v3. This sort of
>> committment is particularly important to commercial customers.
>
> Yes, if there is a clear current commitment to support these versions
> then indeed you cannot just cut off support without warning. Thanks for
> clarifying.
>
>> Actually this is a good time to review this - given the imminent EOL
>> of Python v2.
>>
>> I'll put out a draft devised policy (based on EOL dates) for comment
>> in the next day or so.
>
> That seems like a useful policy to establish. Especially with regard to
> older versions of a major release cycle. Of course I do have a biased
> (python2.7+, python3.5+ strictly open-source) perspective. :)
>
> ...
>
> As far as the original topic of uic using str(), in order to solve this
> while not using unicode_literals everywhere, I can see three solutions:
>
> - have separate python2 and python3 versions of these files which use
> u'' on python2, but retain 3.2 compatibility by having plain '' strings
> in the python3 version

This approach is taken for other things, see the port_v2 and port_v3 directories.

> - add a conditional check for sys.version_info and determine whether to
> call unicode() on the strings (slower than using u'', but the speed is
> probably negligible...)
>
> - moving some code to something like PyQt5/uic/unicode_constants.py, and
> using 'from __future__ import unicode_literals' to change only those
> files to use native unicode types, which should work on both python2 and
> python3 without affecting extra things like docstrings as it will only
> affect that file.
>
> And option four:
> - make a note to fix this, only after an EOL policy is devised and
> deployed which allows you to stop supporting 3.0-3.2 that do not support
> u'' strings.

That will certainly happen longer term.

> Do any of these options seem like a good idea to you? I'd probably go
> with the if sys.version_info[:1] == (2,): var = unicode(var)

So 1) now and 4) later.

Phil
_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Writing dual python2/python3 code and the use of PyQt5.uic

Florian Bruhin
In reply to this post by Phil Thompson-5
On Sun, Mar 24, 2019 at 02:18:14PM +0000, Phil Thompson wrote:

> On 24 Mar 2019, at 1:59 pm, Eli Schwartz <[hidden email]> wrote:
> >
> > On 3/24/19 6:07 AM, Phil Thompson wrote:
> >> On 24 Mar 2019, at 1:48 am, Eli Schwartz <[hidden email]> wrote:
> >>>
> >>> I've been devoting some time recently to incrementally porting a PyQt5
> >>> application "calibre" (https://github.com/kovidgoyal/calibre/) from
> >>> python2 to polyglot code, hoping to get it running with python3 at some
> >>> point. One recent stumbling block I hit was that it uses the uic module
> >>> to compile forms into a cStringIO.StringIO.
> >>>
> >>> What I wanted to do was switch it to using the modern io interface, but
> >>> as it turns out, PyQt5 uses unqualified "str" instances to write out the
> >>> form. As a result, I cannot use io.StringIO on python2, and I cannot use
> >>> io.BytesIO on python3.
> >>>
> >>> The uic module quite specifically seems to expect that it will
> >>> manipulate unicode strings. I was able to get io.StringIO buffers to
> >>> work correctly with both python2 and python3 after I modified two files
> >>> to contain a "from __future__ import unicode_literals":
> >>>
> >>> /usr/lib/python2.7/site-packages/PyQt5/uic/__init__.py
> >>> /usr/lib/python2.7/site-packages/PyQt5/uic/Compiler/indenter.py
> >>>
> >>> Can this be added to at least these two files in the official
> >>> distribution? It seems to be a mistake that it ever assumed the str type.
> >>
> >> This...
> >>
> >> https://python-future.org/unicode_literals.html
> >>
> >> ...makes me nervous. I would prefer a more specific fix.
> >
> > The alternative is to add u'' decorations to the specific strings that
> > cannot be python2's dual purpose str/bytes.
> >
> > I *think* this should be sufficient:
> > https://github.com/eli-schwartz/pyqt5/commit/1a014130024530eb0241cec7ef7ffb82e2c69fc6
> >
> > patch:
> > https://github.com/eli-schwartz/pyqt5/commit/1a014130024530eb0241cec7ef7ffb82e2c69fc6.patch
> >
> > (Applied on top of a tarball import as I could not find any version
> > control sources for PyQt5.)
>
> What about Python v3.3 and earlier?
To get the same effect with compatibility for all versions, you could do
something like this (untested):

  def u(s):
      return s if sys.version_info[0] == 3 else unicode(s)

And then use u("...") instead of u"..."

Florian

--
https://www.qutebrowser.org | [hidden email] (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | https://the-compiler.org/pubkey.asc
         I love long mails! | https://email.is-not-s.ms/

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Writing dual python2/python3 code and the use of PyQt5.uic

Florian Bruhin
On Mon, Mar 25, 2019 at 01:42:42PM +0100, Florian Bruhin wrote:

> On Sun, Mar 24, 2019 at 02:18:14PM +0000, Phil Thompson wrote:
> > On 24 Mar 2019, at 1:59 pm, Eli Schwartz <[hidden email]> wrote:
> > > The alternative is to add u'' decorations to the specific strings that
> > > cannot be python2's dual purpose str/bytes.
> > >
> > > I *think* this should be sufficient:
> > > https://github.com/eli-schwartz/pyqt5/commit/1a014130024530eb0241cec7ef7ffb82e2c69fc6
> > >
> > > patch:
> > > https://github.com/eli-schwartz/pyqt5/commit/1a014130024530eb0241cec7ef7ffb82e2c69fc6.patch
> > >
> > > (Applied on top of a tarball import as I could not find any version
> > > control sources for PyQt5.)
> >
> > What about Python v3.3 and earlier?
>
> To get the same effect with compatibility for all versions, you could do
> something like this (untested):
>
>   def u(s):
>       return s if sys.version_info[0] == 3 else unicode(s)
>
> And then use u("...") instead of u"..."
Whoops - my bad, Eli mentioned something similar already, should've read the
rest of the thread *before* replying :)

Sorry for the noise!

Florian

--
https://www.qutebrowser.org | [hidden email] (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | https://the-compiler.org/pubkey.asc
         I love long mails! | https://email.is-not-s.ms/

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt

signature.asc (849 bytes) Download Attachment