Attempting to reduce the memory footprint of my PyQt5 application

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Attempting to reduce the memory footprint of my PyQt5 application

Xavion
Hi All,

I'm trying to cut down on the memory usage of my PyQt5 application.  I'm assuming that one way to do this is to only import the Qt submodules that are actually used.

The three attached files illustrate my point.  The Qt5 Designer only imports the necessary submodules (e.g. "QtCore/QVariant").  Whereas, the PyQt5 UIC and RCC generators import the parent modules in full (e.g. "QtCore").

Assuming my memory reduction theory is right, is there any way that I can get the UIC and RCC generators to behave like the Qt5 Designer?  There's probably no point in using this strategy throughout the rest of my code if not.


--
Regards, Xavion.


_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt

Qt5-Designer.txt (1K) Download Attachment
PyQt5-UIC.txt (340 bytes) Download Attachment
PyQt5-RCC.txt (270 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

David Boddie
On Tue Sep 6 05:22:29 BST 2016, Xavion wrote:

> I'm trying to cut down on the memory usage of my PyQt5 application.  I'm
> assuming that one way to do this is to only import the Qt submodules that
> are actually used.
>
> The three attached files illustrate my point.  The Qt5 Designer only
> imports the necessary submodules (e.g. "QtCore/QVariant").  Whereas, the
> PyQt5 UIC and RCC generators import the parent modules in full (e.g.
> "QtCore").

In PyQt, unless I'm mistaken, you get the whole module when you import a
class from it so there's no way to just import the classes you use. There
may be some tricks done to save memory at run-time but you will already be
benefiting from those.

> Assuming my memory reduction theory is right, is there any way that I can
> get the UIC and RCC generators to behave like the Qt5 Designer?  There's
> probably no point in using this strategy throughout the rest of my code if
> not.

I don't think there's a strategy to use.

You could check to see that you're not importing modules that you don't use.
Otherwise, I'm not sure if I can give you any general memory-saving tips.
Perhaps others can offer some advice here.

David
_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

Florian Bruhin
* David Boddie <[hidden email]> [2016-09-07 01:03:30 +0200]:

> On Tue Sep 6 05:22:29 BST 2016, Xavion wrote:
>
> > I'm trying to cut down on the memory usage of my PyQt5 application.  I'm
> > assuming that one way to do this is to only import the Qt submodules that
> > are actually used.
> >
> > The three attached files illustrate my point.  The Qt5 Designer only
> > imports the necessary submodules (e.g. "QtCore/QVariant").  Whereas, the
> > PyQt5 UIC and RCC generators import the parent modules in full (e.g.
> > "QtCore").
>
> In PyQt, unless I'm mistaken, you get the whole module when you import a
> class from it so there's no way to just import the classes you use. There
> may be some tricks done to save memory at run-time but you will already be
> benefiting from those.
Correct - not only in PyQt, but in Python in general. Basically,
"from foo import bar" is equivalent to:

    import foo
    bar = foo.bar
    del foo

Florian

--
http://www.the-compiler.org | [hidden email] (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc
         I love long mails! | http://email.is-not-s.ms/

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt

signature.asc (817 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

Xavion
So, you're both saying that the following two lines are equivalent (from a memory footprint standpoint):
  • from PyQt5.QtCore import QVariant
  • from PyQt5 import QtCore
In other words, in both cases, the whole of 'QtCore' will be imported (rather than just 'QVariant' in the first case).

Florian: I think you're saying that I can reduce the memory usage by doing the following.  If so, can we get the UIC and RCC generators to behave this way?
  • from PyQt5 import QtCore
  • mVariant = QtCore.QVariant
  • del QtCore


_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

Andreas Pakulat
Hi,

On Wed, Sep 7, 2016 at 11:57 PM, Xavion <[hidden email]> wrote:

> So, you're both saying that the following two lines are equivalent (from a
> memory footprint standpoint):
>
> from PyQt5.QtCore import QVariant
> from PyQt5 import QtCore
>
> In other words, in both cases, the whole of 'QtCore' will be imported
> (rather than just 'QVariant' in the first case).
>
> Florian: I think you're saying that I can reduce the memory usage by doing
> the following.  If so, can we get the UIC and RCC generators to behave this
> way?
>
> from PyQt5 import QtCore
> mVariant = QtCore.QVariant
> del QtCore

Lets clear up a misconception you seem to have: Neither Qt nor PyQt
have 'submodules' you can 'load'. Both consist of a set of shared
libraries, QtCore, QtXml, QtGui etc., where the PyQt modules just
contain the 'glue code' between C++ and Python. If you load 'part' of
one of those modules the whole of the shared library is loaded into
memory (at least with the common linkers used on the 3 major desktop
OS).

So wether you import QtCore or just QtCore.QVariant makes no
difference in terms of memory footprint of your application. AFAIK the
only option to reduce the memory needed to load QtCore (or other Qt
modules) is to strip stuff out of them when building. Qt allows to
disable certain parts and features that you don't need and I'd assume
PyQt adapts to that also only including stuff that Qt has support
built in for.

Andreas
_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

oliver-56


On Wed, Sep 7, 2016, 19:14 Andreas Pakulat <[hidden email]> wrote:
Hi,

On Wed, Sep 7, 2016 at 11:57 PM, Xavion <[hidden email]> wrote:
> So, you're both saying that the following two lines are equivalent (from a
> memory footprint standpoint):
>
> from PyQt5.QtCore import QVariant
> from PyQt5 import QtCore
>
> In other words, in both cases, the whole of 'QtCore' will be imported
> (rather than just 'QVariant' in the first case).

That is correct and same for all python packages/modules not just pyqt 

> Florian: I think you're saying that I can reduce the memory usage by doing
> the following...
>
> from PyQt5 import QtCore
> mVariant = QtCore.QVariant
> del QtCore

Python modules can be unloaded this way but I don't think a DLL gets unloaded from memory once it is no longer in use by an app. But even if there was code in the Python interpreter to unload a DLL it would not get used because the above code will still be holding a reference to a symbol that is defined in the DLL so the DLL will not be considered unlovable. 

Why are you so concerned with memory occupied by modules? 

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

oliver-56
In reply to this post by Xavion
Xavion, 
The better way to save memory is to use following techniques: 
  1. don't import more than necessary; for example, import from PyQt5.QtCore, not from PyQt5.Qt
  2. consider using Python slots (unrelated to Qt slots) for large numbers of objects stored in lists
  3. consider delaying loading of data (from disk or network), and only load what the app can show/process/edit
  4. use generators instead of lists by using range() and itertools module when possible, and consider creating your own generators using the yield statement
  5. *might* save some mem by using struct module where appropriate
  6. only create circular references when necessary (see http://engineering.hearsaysocial.com/2013/06/16/circular-references-in-python/), although it seems that python 3.4 is now quite good at deleting objects that were stuck in cyclical refs even if they have a del method (but the docs say that there can still be uncollectable garbage once in a while, without being specific, so if your app happens to generate a lot of objects with the particular kind of circular reference that is not collectable by interpreter, you would see steady increase in memory by your app as you create and delete those objects). Looks like objgraph might be a good utility for identifying cycles but I have not used it yet (just found out while researching this topic)

All that being said, you should probably dig deeper into into all of the above only when you determine you have a memory leak or are using gigs more memory than necessary. 
Oliver


Oliver
Open Source contributions: PyPubSub, nose2pytest, Lua-iCxxiof
StackOverflow contributions


On Tue, Sep 6, 2016 at 12:22 AM, Xavion <[hidden email]> wrote:
Hi All,

I'm trying to cut down on the memory usage of my PyQt5 application.  I'm assuming that one way to do this is to only import the Qt submodules that are actually used.

The three attached files illustrate my point.  The Qt5 Designer only imports the necessary submodules (e.g. "QtCore/QVariant").  Whereas, the PyQt5 UIC and RCC generators import the parent modules in full (e.g. "QtCore").

Assuming my memory reduction theory is right, is there any way that I can get the UIC and RCC generators to behave like the Qt5 Designer?  There's probably no point in using this strategy throughout the rest of my code if not.


--
Regards, Xavion.


_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt


_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

oliver-56
In reply to this post by oliver-56
Correction: even Python modules cannot be unloaded, not even in Python 3.5. Multiple discussions on the web indicate this, and a simple Python script with a utility that shows memory consumption (like process explorer on Windows) shows it. Here I tried with a pure-Python package called pytest:

import time, sys, gc

time.sleep(20)

# save modules catalog and import a big module
old_modules = sys.modules.copy()
import pytest # will cause 16 megs increase in process mem
print(sys.modules)
time.sleep(10)

# cleanup and restore original catalog to ensure that neither pytest or any of the many mofulrd it 
# imported are still referenced anywhere, and get gc to cleanup
del pytest
sys.modules = old_modules
del old_modules
gc.collect()   # mem stays at 16 megs more than original
print(sys.modules)
time.sleep(10)

Looks like there might be another attempt at supporting pure-Python module unloading by developers of CPython but it is not clear when, and to what extent it would work (you can imagine that modules would have to be marked for how many times their symbols are referenced because two modules A and B might import a common module C -- unloading A should not unload C if B is present, but then unloading B after A unloaded should also unload C). 

One post on python dev bug tracker site confirms that unloading a C/C++ extension (DLL) is near impossible. 
Oliver


Oliver
Open Source contributions: PyPubSub, nose2pytest, Lua-iCxxiof
StackOverflow contributions


On Wed, Sep 7, 2016 at 9:48 PM, oliver <[hidden email]> wrote:


On Wed, Sep 7, 2016, 19:14 Andreas Pakulat <[hidden email]> wrote:
Hi,

On Wed, Sep 7, 2016 at 11:57 PM, Xavion <[hidden email]> wrote:
> So, you're both saying that the following two lines are equivalent (from a
> memory footprint standpoint):
>
> from PyQt5.QtCore import QVariant
> from PyQt5 import QtCore
>
> In other words, in both cases, the whole of 'QtCore' will be imported
> (rather than just 'QVariant' in the first case).

That is correct and same for all python packages/modules not just pyqt 

> Florian: I think you're saying that I can reduce the memory usage by doing
> the following...
>
> from PyQt5 import QtCore
> mVariant = QtCore.QVariant
> del QtCore

Python modules can be unloaded this way but I don't think a DLL gets unloaded from memory once it is no longer in use by an app. But even if there was code in the Python interpreter to unload a DLL it would not get used because the above code will still be holding a reference to a symbol that is defined in the DLL so the DLL will not be considered unlovable. 

Why are you so concerned with memory occupied by modules? 


_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

Florian Bruhin
In reply to this post by Xavion
* Xavion <[hidden email]> [2016-09-08 07:57:22 +1000]:
> So, you're both saying that the following two lines are equivalent (from a
> memory footprint standpoint):
>
>    - from PyQt5.QtCore import QVariant
>    - from PyQt5 import QtCore
>
> In other words, in both cases, the whole of 'QtCore' will be imported
> (rather than just 'QVariant' in the first case).

Right.

> Florian: I think you're saying that I can reduce the memory usage by doing
> the following.  If so, can we get the UIC and RCC generators to behave this
> way?
>
>    - from PyQt5 import QtCore
>    - mVariant = QtCore.QVariant
>    - del QtCore

No, that's not what I was saying - I was just saying this is roughly
what happens behind the scenes when you do "from PyQt5.QtCore import
QVariant". The "del QtCore" just makes that name unavailable again, it
doesn't change anything from a memory point of view.

Florian

--
http://www.the-compiler.org | [hidden email] (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc
         I love long mails! | http://email.is-not-s.ms/

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt

signature.asc (817 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

Florian Bruhin
In reply to this post by oliver-56
* oliver <[hidden email]> [2016-09-08 00:26:03 -0400]:
>    2. consider using Python slots (unrelated to Qt slots) for large numbers
>    of objects stored in lists

Note this only helps when you actually have *a lot* of equal,
immutable objects, like here:
http://tech.oyster.com/save-ram-with-python-slots/

I wouldn't recommend it as a general thing, as it definitely makes
code less maintainable.

> All that being said, you should probably dig deeper into into all of the
> above only when you determine you have a memory leak or are using gigs more
> memory than necessary.

Right - I think the most important thing - before blindly trying to
fix things - is to measure, not guess.

There are various memory profilers for Python, I usually use Pympler:
http://pythonhosted.org/Pympler/

Maybe also try playing with a C++ memory profiler (like valgrind):
http://valgrind.org/

Though I had mixed success with that, and it's *really* slow.

Florian

--
http://www.the-compiler.org | [hidden email] (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc
         I love long mails! | http://email.is-not-s.ms/

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt

signature.asc (817 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

Xavion
Thanks for all of the help and insight, guys.  I wasn't expecting to get such informative responses.  I will look into the methods and tools you've suggested, and go from there.  I will let you know if I have any follow-up questions along the way.

Some Asian guy had reported a nasty memory leak in my application a while ago.  It was still in PyQt4 form at the time.  I wasn't able to reproduce the leak here.  The language barrier, and lengthy delays between responses, made it difficult to request more information from him.

When I revisited the problem recently, the first logical step was to avoid loading unnecessary PyQt modules.  I wanted to eliminate that as a potential cause before spending more time interrogating my code.  I didn't want to be sending myself on a wild goose chase in the event that PyQt was responsible.


_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

Xavion
Okay, I've found the time to do some more testing on this.  Pympler wasn't able to find any memory leaks, but that could've just meant that I wasn't using it properly.  Either way, the good news is that I've managed to find the source of the leak manually.

Basically, I've got a QTimer which triggers the (re)starting of a QThread instance every 15 seconds.  Said thread creates a QProcess, which runs a shell command and collects its output.  The QProcess is then destroyed and the memory is (supposedly) freed.

However, despite this arrangement, the memory usage of my application still goes up by about 4 KiB each time the QThread is run.  While this may sound insignificant, bear in mind that said application is intended to run for days on end (in the system tray).  Over the period of a week, the memory increase would be roughly 160 MiB.  One of my users has reported seeing exactly this (and more, the longer it goes on).

No matter what I try, I can't seem to stop it from happening.  Even with the following code, the memory footprint still increases by the same amount each time the QThread is run.
pStatus = QtCore.QProcess()
pStatus.start(sCommand, sArgs)
pStatus.waitForStarted(500)
while not pStatus.waitForFinished(500) :
pass
pStatus.deleteLater()
gc.collect()
self.exit()

However, note that merely creating the QProcess (i.e. running the first line of code by itself) doesn't result in any ongoing increase in memory.  This leads me to believe that the leak has something to do with the execution of the shell command.  Does anyone know how to make it go away?


_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

Kovid Goyal
Use subprocess? Simpler code and no leaks.

subprocess.check_output([command] + args)

And also use python threads instead of QThread. In general, minimize the
use of Qt to only the parts it is actually needed for. Object lifetime
issues (leaks/segfaults) are very easy to trigger when using python+Qt.
So I long ago settled on the best practice of avoiding Qt as much
possible.

Kovid.

On Thu, Sep 15, 2016 at 10:14:13AM +1000, Xavion wrote:

> Okay, I've found the time to do some more testing on this.  Pympler wasn't
> able to find any memory leaks, but that could've just meant that I wasn't
> using it properly.  Either way, the good news is that I've managed to find
> the source of the leak manually.
>
> Basically, I've got a QTimer which triggers the (re)starting of a QThread
> instance every 15 seconds.  Said thread creates a QProcess, which runs a
> shell command and collects its output.  The QProcess is then destroyed and
> the memory is (supposedly) freed.
>
> However, despite this arrangement, the memory usage of my application still
> goes up by about 4 KiB each time the QThread is run.  While this may sound
> insignificant, bear in mind that said application is intended to run for
> days on end (in the system tray).  Over the period of a week, the memory
> increase would be roughly 160 MiB.  One of my users has reported seeing
> exactly this (and more, the longer it goes on).
>
> No matter what I try, I can't seem to stop it from happening.  Even with
> the following code, the memory footprint still increases by the same amount
> each time the QThread is run.
>
> pStatus = QtCore.QProcess()
> pStatus.start(sCommand, sArgs)
> pStatus.waitForStarted(500)
> while not pStatus.waitForFinished(500) :
>
> pass
>
> pStatus.deleteLater()
> gc.collect()
> self.exit()
>
>
> However, note that merely creating the QProcess (i.e. running the first
> line of code by itself) doesn't result in any ongoing increase in memory.
> This leads me to believe that the leak has something to do with the
> execution of the shell command.  Does anyone know how to make it go away?

> _______________________________________________
> PyQt mailing list    [hidden email]
> https://www.riverbankcomputing.com/mailman/listinfo/pyqt


--
_____________________________________

Dr. Kovid Goyal
http://www.kovidgoyal.net
http://calibre-ebook.com
_____________________________________
_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

Xavion
Use subprocess? Simpler code and no leaks.

subprocess.check_output([command] + args)

I had a feeling someone would tell me to do that :-).

Believe it or not, the same problem happens with the 'subprocess' module.  As such, I have just opened a ticket on the Python bug tracker.

I guess this news pretty much gets Qt and PyQt off the hook.  I'm sure you guys are pleased about that :-).


_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

Kovid Goyal
That seems highly unlikely, and I cannot reproduce it on my Arch Linux
system, with the following script:

import psutil
import subprocess
import time
import gc
import os

p = psutil.Process(os.getpid())
print(p.memory_info())

while True:
    subprocess.check_output(['true'])
    time.sleep(1)
    gc.collect(), gc.collect(), gc.collect()
    print(p.memory_info())

Output memory usage stays constant after the first two iterations.

Kovid.

On Thu, Sep 15, 2016 at 02:48:23PM +1000, Xavion wrote:

> >
> > Use subprocess? Simpler code and no leaks.
> >
> > subprocess.check_output([command] + args)
> >
>
> I had a feeling someone would tell me to do that :-).
>
> Believe it or not, the same problem happens with the 'subprocess' module.
> As such, I have just opened a ticket <http://bugs.python.org/issue28165> on
> the Python bug tracker.
>
> I guess this news pretty much gets Qt and PyQt off the hook.  I'm sure you
> guys are pleased about that :-).

> _______________________________________________
> PyQt mailing list    [hidden email]
> https://www.riverbankcomputing.com/mailman/listinfo/pyqt


--
_____________________________________

Dr. Kovid Goyal
http://www.kovidgoyal.net
http://calibre-ebook.com
_____________________________________
_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

Elvis Stansvik
In reply to this post by Xavion

Den 15 sep. 2016 6:49 fm skrev "Xavion" <[hidden email]>:
>>
>> Use subprocess? Simpler code and no leaks.
>>
>> subprocess.check_output([command] + args)
>
>
> I had a feeling someone would tell me to do that :-).
>
> Believe it or not, the same problem happens with the 'subprocess' module.  As such, I have just opened a ticket on the Python bug tracker.
>
> I guess this news pretty much gets Qt and PyQt off the hook.  I'm sure you guys are pleased about that :-).

You mentioned you're spawning a thread, which in turn spawn your process (may I ask why the threading?). QProcess is designed to be used async on top of the Qt event loop.

Are you sure that everything related to the thread is destroyed properly?

Another tip is to run the app for 15 seconds under valgrind to see where your leak is.

Elvis

>
>
> _______________________________________________
> PyQt mailing list    [hidden email]
> https://www.riverbankcomputing.com/mailman/listinfo/pyqt


_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

Xavion
I've just updated the ticket I created on the Python bug tracker earlier.  My most recent post lists the code fragment I'm using (i.e. via a timer, not a loop).  The one before that details how I'm going about monitoring the memory (i.e. from a Bash script).

I just incorporated your three "gc.collect()" calls and it made no difference.  Starting the processes from the main event loop (rather than in a QThread) didn't help either.  Everything was fine with the QThread anyway: disabling the process call kept the memory constant.

I'll have a go at using Valgrind tomorrow.

Thanks again for all of your help, guys.


_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

Kovid Goyal
I cannot reproduce via a timer either, script is:

import psutil
import subprocess
import time
import gc
import os
import threading

p = psutil.Process(os.getpid())
print(p.memory_info())

def f():
    subprocess.check_output('true')
    gc.collect(), gc.collect(), gc.collect()
    print(p.memory_info())
    threading.Timer(1, f, ()).start()

f()
input()

Output is:

pmem(rss=11374592, vms=58744832, shared=4669440, text=4096, lib=0, data=7122944, dirty=0)
pmem(rss=11931648, vms=58892288, shared=4993024, text=4096, lib=0, data=7270400, dirty=0)
pmem(rss=11960320, vms=134393856, shared=4993024, text=4096, lib=0, data=15794176, dirty=0)
pmem(rss=14303232, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)
pmem(rss=12242944, vms=212086784, shared=5210112, text=4096, lib=0, data=24322048, dirty=0)

And before you ask, I verified that memory usage was constant using top
as well.

Kovid.

On Thu, Sep 15, 2016 at 06:11:02PM +1000, Xavion wrote:

> I've just updated the ticket <http://bugs.python.org/issue28165> I created
> on the Python bug tracker earlier.  My most recent post lists the code
> fragment I'm using (i.e. via a timer, not a loop).  The one before that
> details how I'm going about monitoring the memory (i.e. from a Bash script).
>
> I just incorporated your three "gc.collect()" calls and it made no
> difference.  Starting the processes from the main event loop (rather than
> in a QThread) didn't help either.  Everything was fine with the QThread
> anyway: disabling the process call kept the memory constant.
>
> I'll have a go at using Valgrind tomorrow.
>
> Thanks again for all of your help, guys.

> _______________________________________________
> PyQt mailing list    [hidden email]
> https://www.riverbankcomputing.com/mailman/listinfo/pyqt


--
_____________________________________

Dr. Kovid Goyal
http://www.kovidgoyal.net
http://calibre-ebook.com
_____________________________________
_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

Xavion
Yes, I can confirm that there's no leak when it's called in that fashion.  However, there is one when the timer is (re)called from within a loop.  I've just posted a new comment to the ticket and attached code/output samples (there) to illustrate my point.

While we're there, I found another leak in my program (manually).  It's to do with overwriting of icons that are attached to widgets in memory.  Specifically, I'm talking about the QMainWindow and QSystemTrayIcon classes.  I'm sure there are other cases; I just haven't looked for them.

Each time I give either one of those a new icon (at runtime), my memory footprint increases proportionately.  Trying to suck the old icon out first and deleting it manually doesn't help.  Is Qt or PyQt likely to blame for this lack of automatic cleanup, and how easy would it be to fix?


_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Attempting to reduce the memory footprint of my PyQt5 application

Xavion
For the sake of completeness, I thought I should mention that I finally got around to filing a bug report about the icon-related memory leak on the Qt website.  It will be interesting to see how long it takes them to fix it, assuming that they don't consider it to be "in the hands of the OS/WM".


_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
12