How to build pyqt sources to provide the same content than official wheel?

classic Classic list List threaded Threaded
12 messages Options
BPL
Reply | Threaded
Open this post in threaded view
|

How to build pyqt sources to provide the same content than official wheel?

BPL
Hello everyone (specially Phil :))

I've recently subscribed to the pyqt mailing list so it seems I can't reply to existing threads when using gmail. I'm the user from #pyqt who was having issues when building pyqt from sources who's mentioned here https://www.riverbankcomputing.com/pipermail/pyqt/2019-July/041953.html

Anyway, I'm really interested to get a qt bug fix who's already living in upstream https://bugreports.qt.io/browse/QTBUG-74492 , as you can see that fix is living in both 5.12.5, 5.13.1 and dev branches.

The current pypi wheel https://pypi.org/project/PyQt5/5.13.0/ at this moment still contains that annoying bug so I've tried to build https://www.riverbankcomputing.com/static/Downloads/PyQt5/PyQt5_gpl-5.13.1.dev1907191722.zip but the result has been not good. Here'a the steps I've followed:

1) I've built https://github.com/qt/qtbase 5.13.0-394-g69ef6e8212 (you can check the commit hash by doing git rev-parse). Why did I choose this one? Mainly because this specific commit had the bug fix I was really interested with and also because the MODULE_VERSION = 5.13.1.

2) To build it I've had to add openssl support otherwise pyqt would crash when building (for more info you can check the other Florian's thread where he also proposes a pyqt fix in one of the sip files), here's the configuration I've used:

configure.bat -prefix "d:/qt/5.13.0-394-g69ef6e8212" -opensource -confirm-license -nomake examples -nomake tests -opengl dynamic -ssl -L D:/software/vcpkg/installed/x86-windows/lib -I D:/software/vcpkg/installed/x86-windows/include -v

3) Then, I've just run this little batch file:

%echo off
setlocal
set SIP_PATH=D:\virtual_envs\packages_sources\3rdparty\sip-4.19.18
set QMAKE_PATH=D:\qt\5.13.0-394-g69ef6e8212\bin
rem set QMAKE_PATH=D:\qt\5.13.0-941-g56fb2266f2\bin
set PYQT_PATH=PyQt5_gpl-5.13.1.dev1907171931
set PATH=%QMAKE_PATH%;%SIP_PATH%\sipgen;%PATH%
where qmake
if ERRORLEVEL 1 (
    echo qmake.exe not found in %QMAKE_PATH%
    exit /b
)
where sip
if ERRORLEVEL 1 (
    echo sip.exe not found in %SIP_PATH%
    exit /b
)
pushd %PYQT_PATH%
python configure.py --sip-incdir=%SIP_PATH%\siplib --confirm-license
nmake
popd

4) Finally, I've just installed all the generated content doing `nmake install`

Anyway, the result has been really bad and the experience quite frustrating... I've ended up with an installation that doesn't look at all like the nice installation from the official pypi wheels, a lot of missing content and also a lot of additional sip files copied to the virtualenv... the installation at this this point is basically unusable :'(

So my questions are:

- What'd be the right procedure to get a nice wheel that looks like the official pypi ones? 
- I've been told eventually you'll be providing a proper setup.py in the source packages that will allow to build nice wheels instead this non-standard bizarre way to build pyqt, is that true? That'd be extremely awesome!
- Is there any release date where a new pyqt wheel will be landing at pypi that contains the bug fix of https://bugreports.qt.io/browse/QTBUG-74492 ?

Thanks in advance.

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: How to build pyqt sources to provide the same content than official wheel?

Phil Thompson-5
On 20/07/2019 11:58, Scener Spanish wrote:

> Hello everyone (specially Phil :))
>
> I've recently subscribed to the pyqt mailing list so it seems I can't
> reply
> to existing threads when using gmail. I'm the user from #pyqt who was
> having issues when building pyqt from sources who's mentioned here
> https://www.riverbankcomputing.com/pipermail/pyqt/2019-July/041953.html
>
> Anyway, I'm really interested to get a qt bug fix who's already living
> in
> upstream https://bugreports.qt.io/browse/QTBUG-74492 , as you can see
> that
> fix is living in both 5.12.5, 5.13.1 and dev branches.
>
> The current pypi wheel https://pypi.org/project/PyQt5/5.13.0/ at this
> moment still contains that annoying bug so I've tried to build
> https://www.riverbankcomputing.com/static/Downloads/PyQt5/PyQt5_gpl-5.13.1.dev1907191722.zip
> but
> the result has been not good. Here'a the steps I've followed:
>
> 1) I've built https://github.com/qt/qtbase 5.13.0-394-g69ef6e8212 (you
> can
> check the commit hash by doing git rev-parse). Why did I choose this
> one?
> Mainly because this specific commit had the bug fix I was really
> interested
> with and also because the MODULE_VERSION = 5.13.1.
>
> 2) To build it I've had to add openssl support otherwise pyqt would
> crash
> when building (for more info you can check the other Florian's thread
> where
> he also proposes a pyqt fix in one of the sip files), here's the
> configuration I've used:
>
> configure.bat -prefix "d:/qt/5.13.0-394-g69ef6e8212" -opensource
> -confirm-license -nomake examples -nomake tests -opengl dynamic -ssl -L
> D:/software/vcpkg/installed/x86-windows/lib -I
> D:/software/vcpkg/installed/x86-windows/include -v
>
>
> 3) Then, I've just run this little batch file:
>
> %echo off
> setlocal
> set SIP_PATH=D:\virtual_envs\packages_sources\3rdparty\sip-4.19.18
> set QMAKE_PATH=D:\qt\5.13.0-394-g69ef6e8212\bin
> rem set QMAKE_PATH=D:\qt\5.13.0-941-g56fb2266f2\bin
> set PYQT_PATH=PyQt5_gpl-5.13.1.dev1907171931
> set PATH=%QMAKE_PATH%;%SIP_PATH%\sipgen;%PATH%
> where qmake
> if ERRORLEVEL 1 (
>     echo qmake.exe not found in %QMAKE_PATH%
>     exit /b
> )
> where sip
> if ERRORLEVEL 1 (
>     echo sip.exe not found in %SIP_PATH%
>     exit /b
> )
> pushd %PYQT_PATH%
> python configure.py --sip-incdir=%SIP_PATH%\siplib --confirm-license
> nmake
> popd
>
>
> 4) Finally, I've just installed all the generated content doing `nmake
> install`
>
> Anyway, the result has been really bad and the experience quite
> frustrating... I've ended up with an installation that doesn't look at
> all
> like the nice installation from the official pypi wheels, a lot of
> missing
> content and also a lot of additional sip files copied to the
> virtualenv...
> the installation at this this point is basically unusable :'(

I don't see why it should be unusable. It will be the full installation
(a wheel is a subset), use your existing Qt installation and can still
be uninstalled with pip.

> So my questions are:
>
> - What'd be the right procedure to get a nice wheel that looks like the
> official pypi ones?

At the moment you can't do this because it relies on internal tools.

> - I've been told eventually you'll be providing a proper setup.py in
> the
> source packages that will allow to build nice wheels instead this
> non-standard bizarre way to build pyqt, is that true? That'd be
> extremely
> awesome!

Yes, see below.

> - Is there any release date where a new pyqt wheel will be landing at
> pypi
> that contains the bug fix of
> https://bugreports.qt.io/browse/QTBUG-74492 ?

That depends on when the fixed version of Qt is released.

sip5 implements a new build system that supports PEP517. This is
finished (apart from the docs) and is just being tweaked to support
PyQt5's new build system (which is currently being written). This might
sound a lot of work but is mostly a refactoring of those internal tools
I mentioned earlier that I've been using for years.

PyQt5's new build system will also include a tool that will allow you to
update a wheel with a new version of Qt.

ETA for sip5 is between 1 and 2 months.

Phil
_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
BPL
Reply | Threaded
Open this post in threaded view
|

Re: How to build pyqt sources to provide the same content than official wheel?

BPL
I don't see why it should be unusable. It will be the full installation
(a wheel is a subset), use your existing Qt installation and can still
be uninstalled with pip.

Ok, maybe unusable hasn't been a good word (sorry about my English btw), what I meant is
the artifacts produced by nmake install are a very small subset from the artifacts produced
by the official pypi wheel, compare both:


You can see there are a lot of missing submodules (pyd), also sip.pyd is missing and basically the Qt folder also missing...
in addition, there are a lot of non-required sip files installed in the virtual env.

So, is this because the way I've compiled pyqt from sources is not complete? or is it because after installing pyqt from sources I'm also
required to do some extra manual steps...? What do you mean by "use your existing Qt installation"? copying it manually to the virtualenv?
 
That depends on when the fixed version of Qt is released.

Actually, that's another question I'd like to ask you, what do you mean by "fixed version of Qt is released"?
When developing pyqt, are you just using official packages from here https://download.qt.io/archive/qt/ 
instead building yourself the Qt repos at some specific commit hash?
 
sip5 implements a new build system that supports PEP517. This is
finished (apart from the docs) and is just being tweaked to support
PyQt5's new build system (which is currently being written). This might
sound a lot of work but is mostly a refactoring of those internal tools
I mentioned earlier that I've been using for years.

PyQt5's new build system will also include a tool that will allow you to
update a wheel with a new version of Qt.

ETA for sip5 is between 1 and 2 months.

Wow, this sounds extremely awesome, really looking forward to it! 

I must to say in the past years I've avoided building pyqt from sources because I've always found it a 
really complex non-standard python process, it's always looked to me like the type of deployment 
that works really way for the maintainer but not so well for the users/customers. Said otherwise, 
pyqt & sip are both amazing piece of technologies but the deployment process looks to me like a bit obscure.
Which on the other hand, makes sense as the existing tools to create c++ wrappers packages have never been ideal.
I've got myself a bunch of custom tools to create swig wrapper that don't follow the best python
packaging practices neither but they work "for me" :/

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: How to build pyqt sources to provide the same content than official wheel?

Phil Thompson-5
On 20/07/2019 13:29, BPL wrote:

>>
>> I don't see why it should be unusable. It will be the full
>> installation
>> (a wheel is a subset), use your existing Qt installation and can still
>> be uninstalled with pip.
>>
>
> Ok, maybe unusable hasn't been a good word (sorry about my English
> btw),
> what I meant is
> the artifacts produced by nmake install are a very small subset from
> the
> artifacts produced
> by the official pypi wheel, compare both:
>
> * Content produced by pyqt sources:
> https://dl.dropboxusercontent.com/s/nxhzst7f9ajtd3n/2019-07-20_14-10-42.txt
> * Content produced by pypi wheel:
> https://dl.dropboxusercontent.com/s/kqbbeu6wsnx0s4z/2019-07-20_14-10-48.txt
>
> You can see there are a lot of missing submodules (pyd), also sip.pyd
> is
> missing and basically the Qt folder also missing...

configure.py will introspect your Qt installation to see what can be
built. If there are missing .pyd files then your Qt installation doesn't
contain the corresponding libraries.

You have to build the sip.pyd module separately as described in the
documentation.

> in addition, there are a lot of non-required sip files installed in the
> virtual env.

They are required if you are building other PyQt5 related packages from
source.

> So, is this because the way I've compiled pyqt from sources is not
> complete? or is it because after installing pyqt from sources I'm also
> required to do some extra manual steps...? What do you mean by "use
> your
> existing Qt installation"? copying it manually to the virtualenv?

You don't copy your Qt installation, PyQt (when build from sources) will
use it where it is.

>> That depends on when the fixed version of Qt is released.
>>
>
> Actually, that's another question I'd like to ask you, what do you mean
> by
> "fixed version of Qt is released"?
> When developing pyqt, are you just using official packages from here
> https://download.qt.io/archive/qt/
> instead building yourself the Qt repos at some specific commit hash?

I never build Qt.

Phil
_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
BPL
Reply | Threaded
Open this post in threaded view
|

Re: How to build pyqt sources to provide the same content than official wheel?

BPL
configure.py will introspect your Qt installation to see what can be
built. If there are missing .pyd files then your Qt installation doesn't
contain the corresponding libraries.

I see, ty to clarify, this makes a lot of sense... so basically one of my mistakes is the way
I've configured qt base before building is probably missing some parameters
 
You have to build the sip.pyd module separately as described in the
documentation.

If something it's already docummented then of course... I'm the only to be blamed, ty to point out :)

You don't copy your Qt installation, PyQt (when build from sources) will
use it where it is.

Oh, I didn't know this neither... that's one of the main reasons why I'd said "unusable" in the first place :P
 
I never build Qt.

Good to know, so could you please confirm if pyqt use the packages uploaded here https://download.qt.io/archive/qt/ ?

Anyway... I'll try again to see if I can make it work but hopefully a new pyqt wheel with this annoying bug fix https://bugreports.qt.io/browse/QTBUG-74492
will land soon at pypi, I've got some opengl tools in place where this glitch makes really annoying work with them, it's a really disturbing bug :/

Thank you very much.

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: How to build pyqt sources to provide the same content than official wheel?

Phil Thompson-5
On 20/07/2019 13:53, BPL wrote:
>> I never build Qt.
>
> Good to know, so could you please confirm if pyqt use the packages
> uploaded
> here https://download.qt.io/archive/qt/ ?

I use the packages downloaded using the online installer/maintenance
tool.

Phil
_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: How to build pyqt sources to provide the same content than official wheel?

Florian Bruhin
In reply to this post by BPL
On Sat, Jul 20, 2019 at 02:53:29PM +0200, BPL wrote:
> Anyway... I'll try again to see if I can make it work but hopefully a new
> pyqt wheel with this annoying bug fix
> https://bugreports.qt.io/browse/QTBUG-74492
> will land soon at pypi, I've got some opengl tools in place where this
> glitch makes really annoying work with them, it's a really disturbing bug :/

As you can see in the bug report, the fix lands in Qt 5.12.5, 5.13.1 and 5.14.0.
Via the Qt wiki, you can see when those are planned to be released:

https://wiki.qt.io/Qt_5.12_Release (5.12.5: 27.08.2019)
https://wiki.qt.io/Qt_5.13_Release (5.13.1: 15.08.2019)
https://wiki.qt.io/Qt_5.14_Release (5.14.0: 26.11.2019)

After a new Qt is released, usually a couple of days afterwards, PyQt gets
updated (might be a bit longer for new feature releases like 5.14 will be).
So if you're okay with upgrading to Qt 5.13, you'll hopefully see an updated
PyQt in a month or so.

I'd also recommend subscribing to the Qt Releasing mailinglist - it's quite
low-traffic, and you get announcement mails when Qt updates are released, or
when there was a release team meeting (with the meeting minutes):

https://lists.qt-project.org/listinfo/releasing

Florian

--
https://www.qutebrowser.org | [hidden email] (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | https://the-compiler.org/pubkey.asc
         I love long mails! | https://email.is-not-s.ms/

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt

signature.asc (849 bytes) Download Attachment
BPL
Reply | Threaded
Open this post in threaded view
|

Re: How to build pyqt sources to provide the same content than official wheel?

BPL
Florian,

Thanks to provide those wiki docs as well as the qt releasing mailing list, they're definitely
helpful indeed to make proper planning.

If I look in retrospective is funny though, that QTBUG-74492 is the first one I've ever been involved with, 
this bug was opened in 21.03.2019 and after few months passed nobody was working on it or they didn't
know what the problem was. I decided to spend some days bisecting the bug and finding the root cause 
of the problem, thanks to that the author who introduced the bug in the first place could fix it properly in few days. In any case, 
finding the issue was really tricky and the fix ended up to be quite "trivial" one. The moral of the story here is, 
for something as trivial as this one you'll need to wait quite a lot of time to get
a proper release... and that's assuming you're interested enough to help Qt team to narrow it down. The 
workload of Qt devs isn't small neither... which indicates there is room to optimize development in that company.

Bug opened in 21.03.2019 and official release with the fix delivered in 15.08.2019... That's quite a lot
of time to wait for...

Anyway, at this point a global view of how things work in both Qt & pyqt has become +/- clear, which is great so you
know what you can or can't expect.

On Sat, Jul 20, 2019 at 3:10 PM Florian Bruhin <[hidden email]> wrote:
On Sat, Jul 20, 2019 at 02:53:29PM +0200, BPL wrote:
> Anyway... I'll try again to see if I can make it work but hopefully a new
> pyqt wheel with this annoying bug fix
> https://bugreports.qt.io/browse/QTBUG-74492
> will land soon at pypi, I've got some opengl tools in place where this
> glitch makes really annoying work with them, it's a really disturbing bug :/

As you can see in the bug report, the fix lands in Qt 5.12.5, 5.13.1 and 5.14.0.
Via the Qt wiki, you can see when those are planned to be released:

https://wiki.qt.io/Qt_5.12_Release (5.12.5: 27.08.2019)
https://wiki.qt.io/Qt_5.13_Release (5.13.1: 15.08.2019)
https://wiki.qt.io/Qt_5.14_Release (5.14.0: 26.11.2019)

After a new Qt is released, usually a couple of days afterwards, PyQt gets
updated (might be a bit longer for new feature releases like 5.14 will be).
So if you're okay with upgrading to Qt 5.13, you'll hopefully see an updated
PyQt in a month or so.

I'd also recommend subscribing to the Qt Releasing mailinglist - it's quite
low-traffic, and you get announcement mails when Qt updates are released, or
when there was a release team meeting (with the meeting minutes):

https://lists.qt-project.org/listinfo/releasing

Florian

--
https://www.qutebrowser.org | [hidden email] (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | https://the-compiler.org/pubkey.asc
         I love long mails! | https://email.is-not-s.ms/

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: How to build pyqt sources to provide the same content than official wheel?

Kyle Altendorf
On 2019-07-20 10:34, BPL wrote:

> If I look in retrospective is funny though, that QTBUG-74492 is the
> first one I've ever been involved with,
> this bug was opened in 21.03.2019 and after few months passed nobody
> was working on it or they didn't
> know what the problem was. I decided to spend some days bisecting the
> bug and finding the root cause
> of the problem, thanks to that the author who introduced the bug in the
> first place could fix it properly in few days. In any case,
> finding the issue was really tricky and the fix ended up to be quite
> "trivial" one. The moral of the story here is,
> for something as trivial as this one you'll need to wait quite a lot of
> time to get
> a proper release... and that's assuming you're interested enough to
> help Qt team to narrow it down. The
> workload of Qt devs isn't small neither... which indicates there is
> room to optimize development in that company.
>
> Bug opened in 21.03.2019 and official release with the fix delivered in
> 15.08.2019... That's quite a lot
> of time to wait for...

The indication here is that you are highly judgmental of a company that
is giving you decades (times many developers) worth of code for free.  
If my searching is correct, there are 878 issues with more votes than
the four that yours had.  It's really not unreasonable that to get it
resolved you had to put in some time and then you have to wait for a  
normal next release cycle.  It's not like this is a security bug or a
regression causing significant problems for the majority of users.  
Start paying Qt for individual priority development support and then
this can start being considered 'a long time'.  Though even then I
wouldn't expect it to cause a release.

-kyle
_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
BPL
Reply | Threaded
Open this post in threaded view
|

Re: How to build pyqt sources to provide the same content than official wheel?

BPL
In reply to this post by BPL
Phil,

Usually I don't have any of my available qt distributions in path, when you said

You don't copy your Qt installation, PyQt (when build from sources) will 
use it where it is.  

I guess you meant PyQt (when built from sources) will use the Qt installation from either path
or from the Qt folder in the target dir where PyQt has been installed. I can see pyqt 
__init__.py looks like this:

def find_qt():
    import os

    path = os.environ['PATH']

    dll_dir = os.path.dirname(__file__) + '\\Qt\\bin'
    if os.path.isfile(dll_dir + '\\Qt5Core.dll'):
        path = dll_dir + ';' + path
        os.environ['PATH'] = path
    else:
        for dll_dir in path.split(';'):
            if os.path.isfile(dll_dir + '\\Qt5Core.dll'):
                break
        else:
            raise ImportError("unable to find Qt5Core.dll on PATH")

    try:
        os.add_dll_directory(dll_dir)
    except AttributeError:
        pass


find_qt()
del find_qt

As a workaround I've decided to create a junction pointing out 
to the Qt distribution used to build pyqt from sources, ie:

(py364_32) d:\virtual_envs\py364_32\Lib\site-packages\PyQt5>mklink /j Qt d:\qt\5.13.0-394-g69ef6e8212
Junction created for Qt <<===>> d:\qt\5.13.0-394-g69ef6e8212

But for some reason when I do `from PyQt5.Qt import *` I'll just get this content where you can see there isn't any available
QApplication :(

And when doing `from PyQt5.QtWidgets import *` I'll get:

    from PyQt5.QtWidgets import *
ImportError: DLL load failed: The specified module could not be found.

So I'm stuck at this point, could you advice?

On Sat, Jul 20, 2019 at 4:34 PM BPL <[hidden email]> wrote:
Florian,

Thanks to provide those wiki docs as well as the qt releasing mailing list, they're definitely
helpful indeed to make proper planning.

If I look in retrospective is funny though, that QTBUG-74492 is the first one I've ever been involved with, 
this bug was opened in 21.03.2019 and after few months passed nobody was working on it or they didn't
know what the problem was. I decided to spend some days bisecting the bug and finding the root cause 
of the problem, thanks to that the author who introduced the bug in the first place could fix it properly in few days. In any case, 
finding the issue was really tricky and the fix ended up to be quite "trivial" one. The moral of the story here is, 
for something as trivial as this one you'll need to wait quite a lot of time to get
a proper release... and that's assuming you're interested enough to help Qt team to narrow it down. The 
workload of Qt devs isn't small neither... which indicates there is room to optimize development in that company.

Bug opened in 21.03.2019 and official release with the fix delivered in 15.08.2019... That's quite a lot
of time to wait for...

Anyway, at this point a global view of how things work in both Qt & pyqt has become +/- clear, which is great so you
know what you can or can't expect.

On Sat, Jul 20, 2019 at 3:10 PM Florian Bruhin <[hidden email]> wrote:
On Sat, Jul 20, 2019 at 02:53:29PM +0200, BPL wrote:
> Anyway... I'll try again to see if I can make it work but hopefully a new
> pyqt wheel with this annoying bug fix
> https://bugreports.qt.io/browse/QTBUG-74492
> will land soon at pypi, I've got some opengl tools in place where this
> glitch makes really annoying work with them, it's a really disturbing bug :/

As you can see in the bug report, the fix lands in Qt 5.12.5, 5.13.1 and 5.14.0.
Via the Qt wiki, you can see when those are planned to be released:

https://wiki.qt.io/Qt_5.12_Release (5.12.5: 27.08.2019)
https://wiki.qt.io/Qt_5.13_Release (5.13.1: 15.08.2019)
https://wiki.qt.io/Qt_5.14_Release (5.14.0: 26.11.2019)

After a new Qt is released, usually a couple of days afterwards, PyQt gets
updated (might be a bit longer for new feature releases like 5.14 will be).
So if you're okay with upgrading to Qt 5.13, you'll hopefully see an updated
PyQt in a month or so.

I'd also recommend subscribing to the Qt Releasing mailinglist - it's quite
low-traffic, and you get announcement mails when Qt updates are released, or
when there was a release team meeting (with the meeting minutes):

https://lists.qt-project.org/listinfo/releasing

Florian

--
https://www.qutebrowser.org | [hidden email] (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | https://the-compiler.org/pubkey.asc
         I love long mails! | https://email.is-not-s.ms/

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
BPL
Reply | Threaded
Open this post in threaded view
|

Re: How to build pyqt sources to provide the same content than official wheel?

BPL
Phil,

don't mind my previous mail, I think I've found the issue here... problem some Pyqt5 pyd not loading is because they were missing
some external dlls such as libpng16.dll... right now I'm trying to figure out why, maybe the way I've compiled qt:

configure.bat -prefix "d:/qt/5.13.0-394-g69ef6e8212" -opensource -confirm-license -nomake examples -nomake tests -opengl dynamic -ssl -L D:/software/vcpkg/installed/x86-windows/lib -I D:/software/vcpkg/installed/x86-windows/include -v 

wasn't a good idea afterall... I suspect by doing this I've been forced to add d:/software/vcpkg/installed/x86-windows/bin to path when importing
PyQt as well so that libpng16.dll will be found... so probably the best way to link openssl is by downloading & compiling openssl in isolation (ie: https://www.openssl.org/source/)
and not relying in vcpkg at all :/

On Sat, Jul 20, 2019 at 5:54 PM BPL <[hidden email]> wrote:
Phil,

Usually I don't have any of my available qt distributions in path, when you said

You don't copy your Qt installation, PyQt (when build from sources) will 
use it where it is.  

I guess you meant PyQt (when built from sources) will use the Qt installation from either path
or from the Qt folder in the target dir where PyQt has been installed. I can see pyqt 
__init__.py looks like this:

def find_qt():
    import os

    path = os.environ['PATH']

    dll_dir = os.path.dirname(__file__) + '\\Qt\\bin'
    if os.path.isfile(dll_dir + '\\Qt5Core.dll'):
        path = dll_dir + ';' + path
        os.environ['PATH'] = path
    else:
        for dll_dir in path.split(';'):
            if os.path.isfile(dll_dir + '\\Qt5Core.dll'):
                break
        else:
            raise ImportError("unable to find Qt5Core.dll on PATH")

    try:
        os.add_dll_directory(dll_dir)
    except AttributeError:
        pass


find_qt()
del find_qt

As a workaround I've decided to create a junction pointing out 
to the Qt distribution used to build pyqt from sources, ie:

(py364_32) d:\virtual_envs\py364_32\Lib\site-packages\PyQt5>mklink /j Qt d:\qt\5.13.0-394-g69ef6e8212
Junction created for Qt <<===>> d:\qt\5.13.0-394-g69ef6e8212

But for some reason when I do `from PyQt5.Qt import *` I'll just get this content where you can see there isn't any available
QApplication :(

And when doing `from PyQt5.QtWidgets import *` I'll get:

    from PyQt5.QtWidgets import *
ImportError: DLL load failed: The specified module could not be found.

So I'm stuck at this point, could you advice?

On Sat, Jul 20, 2019 at 4:34 PM BPL <[hidden email]> wrote:
Florian,

Thanks to provide those wiki docs as well as the qt releasing mailing list, they're definitely
helpful indeed to make proper planning.

If I look in retrospective is funny though, that QTBUG-74492 is the first one I've ever been involved with, 
this bug was opened in 21.03.2019 and after few months passed nobody was working on it or they didn't
know what the problem was. I decided to spend some days bisecting the bug and finding the root cause 
of the problem, thanks to that the author who introduced the bug in the first place could fix it properly in few days. In any case, 
finding the issue was really tricky and the fix ended up to be quite "trivial" one. The moral of the story here is, 
for something as trivial as this one you'll need to wait quite a lot of time to get
a proper release... and that's assuming you're interested enough to help Qt team to narrow it down. The 
workload of Qt devs isn't small neither... which indicates there is room to optimize development in that company.

Bug opened in 21.03.2019 and official release with the fix delivered in 15.08.2019... That's quite a lot
of time to wait for...

Anyway, at this point a global view of how things work in both Qt & pyqt has become +/- clear, which is great so you
know what you can or can't expect.

On Sat, Jul 20, 2019 at 3:10 PM Florian Bruhin <[hidden email]> wrote:
On Sat, Jul 20, 2019 at 02:53:29PM +0200, BPL wrote:
> Anyway... I'll try again to see if I can make it work but hopefully a new
> pyqt wheel with this annoying bug fix
> https://bugreports.qt.io/browse/QTBUG-74492
> will land soon at pypi, I've got some opengl tools in place where this
> glitch makes really annoying work with them, it's a really disturbing bug :/

As you can see in the bug report, the fix lands in Qt 5.12.5, 5.13.1 and 5.14.0.
Via the Qt wiki, you can see when those are planned to be released:

https://wiki.qt.io/Qt_5.12_Release (5.12.5: 27.08.2019)
https://wiki.qt.io/Qt_5.13_Release (5.13.1: 15.08.2019)
https://wiki.qt.io/Qt_5.14_Release (5.14.0: 26.11.2019)

After a new Qt is released, usually a couple of days afterwards, PyQt gets
updated (might be a bit longer for new feature releases like 5.14 will be).
So if you're okay with upgrading to Qt 5.13, you'll hopefully see an updated
PyQt in a month or so.

I'd also recommend subscribing to the Qt Releasing mailinglist - it's quite
low-traffic, and you get announcement mails when Qt updates are released, or
when there was a release team meeting (with the meeting minutes):

https://lists.qt-project.org/listinfo/releasing

Florian

--
https://www.qutebrowser.org | [hidden email] (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | https://the-compiler.org/pubkey.asc
         I love long mails! | https://email.is-not-s.ms/

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
BPL
Reply | Threaded
Open this post in threaded view
|

Re: How to build pyqt sources to provide the same content than official wheel?

BPL
Just to let you know all the problems I've faced were mainly because I'd compiled qt+ssl using vcpkg include/lib directories and by doing so
I'd be forced to add vcpkg bin directory at runtime, otherwise the pyqt dlls will have missing dependencies.

As a simple solution I've compiled qt with no ssl support (auto) and just modified qocspresponse.sip generated by metasip, which it's
missing a code guard, :

%If (Qt_5_13_0 -)
%If (PyQt_SSL)

%ModuleCode
#include <qocspresponse.h>
%End
%End
%End

That way I've been able to use latest qt from github repo without waiting weeks/months to be released \:O/

Phil, would it be possible to change qocspresponse.sip with this code guard so you won't be forced to use qt with ssl feature or it's this a limitation of metasip?

Thanks.

On Sat, Jul 20, 2019 at 6:38 PM BPL <[hidden email]> wrote:
Phil,

don't mind my previous mail, I think I've found the issue here... problem some Pyqt5 pyd not loading is because they were missing
some external dlls such as libpng16.dll... right now I'm trying to figure out why, maybe the way I've compiled qt:

configure.bat -prefix "d:/qt/5.13.0-394-g69ef6e8212" -opensource -confirm-license -nomake examples -nomake tests -opengl dynamic -ssl -L D:/software/vcpkg/installed/x86-windows/lib -I D:/software/vcpkg/installed/x86-windows/include -v 

wasn't a good idea afterall... I suspect by doing this I've been forced to add d:/software/vcpkg/installed/x86-windows/bin to path when importing
PyQt as well so that libpng16.dll will be found... so probably the best way to link openssl is by downloading & compiling openssl in isolation (ie: https://www.openssl.org/source/)
and not relying in vcpkg at all :/

On Sat, Jul 20, 2019 at 5:54 PM BPL <[hidden email]> wrote:
Phil,

Usually I don't have any of my available qt distributions in path, when you said

You don't copy your Qt installation, PyQt (when build from sources) will 
use it where it is.  

I guess you meant PyQt (when built from sources) will use the Qt installation from either path
or from the Qt folder in the target dir where PyQt has been installed. I can see pyqt 
__init__.py looks like this:

def find_qt():
    import os

    path = os.environ['PATH']

    dll_dir = os.path.dirname(__file__) + '\\Qt\\bin'
    if os.path.isfile(dll_dir + '\\Qt5Core.dll'):
        path = dll_dir + ';' + path
        os.environ['PATH'] = path
    else:
        for dll_dir in path.split(';'):
            if os.path.isfile(dll_dir + '\\Qt5Core.dll'):
                break
        else:
            raise ImportError("unable to find Qt5Core.dll on PATH")

    try:
        os.add_dll_directory(dll_dir)
    except AttributeError:
        pass


find_qt()
del find_qt

As a workaround I've decided to create a junction pointing out 
to the Qt distribution used to build pyqt from sources, ie:

(py364_32) d:\virtual_envs\py364_32\Lib\site-packages\PyQt5>mklink /j Qt d:\qt\5.13.0-394-g69ef6e8212
Junction created for Qt <<===>> d:\qt\5.13.0-394-g69ef6e8212

But for some reason when I do `from PyQt5.Qt import *` I'll just get this content where you can see there isn't any available
QApplication :(

And when doing `from PyQt5.QtWidgets import *` I'll get:

    from PyQt5.QtWidgets import *
ImportError: DLL load failed: The specified module could not be found.

So I'm stuck at this point, could you advice?

On Sat, Jul 20, 2019 at 4:34 PM BPL <[hidden email]> wrote:
Florian,

Thanks to provide those wiki docs as well as the qt releasing mailing list, they're definitely
helpful indeed to make proper planning.

If I look in retrospective is funny though, that QTBUG-74492 is the first one I've ever been involved with, 
this bug was opened in 21.03.2019 and after few months passed nobody was working on it or they didn't
know what the problem was. I decided to spend some days bisecting the bug and finding the root cause 
of the problem, thanks to that the author who introduced the bug in the first place could fix it properly in few days. In any case, 
finding the issue was really tricky and the fix ended up to be quite "trivial" one. The moral of the story here is, 
for something as trivial as this one you'll need to wait quite a lot of time to get
a proper release... and that's assuming you're interested enough to help Qt team to narrow it down. The 
workload of Qt devs isn't small neither... which indicates there is room to optimize development in that company.

Bug opened in 21.03.2019 and official release with the fix delivered in 15.08.2019... That's quite a lot
of time to wait for...

Anyway, at this point a global view of how things work in both Qt & pyqt has become +/- clear, which is great so you
know what you can or can't expect.

On Sat, Jul 20, 2019 at 3:10 PM Florian Bruhin <[hidden email]> wrote:
On Sat, Jul 20, 2019 at 02:53:29PM +0200, BPL wrote:
> Anyway... I'll try again to see if I can make it work but hopefully a new
> pyqt wheel with this annoying bug fix
> https://bugreports.qt.io/browse/QTBUG-74492
> will land soon at pypi, I've got some opengl tools in place where this
> glitch makes really annoying work with them, it's a really disturbing bug :/

As you can see in the bug report, the fix lands in Qt 5.12.5, 5.13.1 and 5.14.0.
Via the Qt wiki, you can see when those are planned to be released:

https://wiki.qt.io/Qt_5.12_Release (5.12.5: 27.08.2019)
https://wiki.qt.io/Qt_5.13_Release (5.13.1: 15.08.2019)
https://wiki.qt.io/Qt_5.14_Release (5.14.0: 26.11.2019)

After a new Qt is released, usually a couple of days afterwards, PyQt gets
updated (might be a bit longer for new feature releases like 5.14 will be).
So if you're okay with upgrading to Qt 5.13, you'll hopefully see an updated
PyQt in a month or so.

I'd also recommend subscribing to the Qt Releasing mailinglist - it's quite
low-traffic, and you get announcement mails when Qt updates are released, or
when there was a release team meeting (with the meeting minutes):

https://lists.qt-project.org/listinfo/releasing

Florian

--
https://www.qutebrowser.org | [hidden email] (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | https://the-compiler.org/pubkey.asc
         I love long mails! | https://email.is-not-s.ms/

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt