Possible memory leak with signal connections

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Possible memory leak with signal connections

Kevin Keating
I've found what appears to be a memory leak caused by connecting signals to slots that lack a QtCore.pyqtSlot decorator.  I know that connections without the pyqtSlot decorator are expected to use more memory than connections with the decorator based on https://www.codeproject.com/Articles/1123088/PyQt-signal-slot-connection-performance.  In the script I've pasted below, though, connections without the pyqtSlot decorator continue to consume memory even after the signal has been disconnected and the QObjects have been discarded.  The script does the following:
  - Instantiates a bunch of SignalObjects, which are QObjects with a signal, and stores the SingalObjects in a list.
  - Instantiates a bunch of SlotObjects, which are QObjects with slots.  Each SlotObject slot is connected to the signals from all the SignalObjects.  The SlotObjects are then immediately discarded.
  - Discards all SignalObjects.
  - Runs gc.collect()
  - Runs the QApplication's event loop for a second in case there are any pending DeferredDelete events.

If the SlotObject class uses the QtCore.pyqtSlot decorator, then memory usage at the end of the script is the same as what it was at the beginning of the script, which makes sense since all the objects that get created should be completely destroyed before the script finishes.  Here's the output that I get with pyqtSlot decorators:
        Memory before signal_objects creation: 17.1MiB
        Memory before slot objects creation: 17.1MiB
                Difference: 0.0B
        Memory after slot objects creation: 17.1MiB
                Difference: 0.0B
        Memory after event loop runs: 17.1MiB
                Difference: 0.0B
If the SlotObject class doesn't use the QtCore.pyqtSlot decorator, though, then memory usage at the end of the script is substantially higher.  Here's the output that I get without pyqtSlot decorators:
        Memory before signal_objects creation: 17.0MiB
        Memory before slot objects creation: 17.0MiB
                Difference: 0.0B
        Memory after slot objects creation: 210.6MiB
                Difference: 193.5MiB
        Memory after event loop runs: 210.6MiB
                Difference: 193.5MiB
Is this expected behavior?  Without pyqtSlot decorators, is there anything I can do to recover the 193 MiB of memory other than terminating the process?  Thanks!

I've tested the script with Python 3.6.2, PyQt 5.12.2, and Qt 5.12.3 on Windows 10, Linux, and Mac OS.  I've also tested with Python 3.6.5, PyQt 5.13.1, and Qt 5.13.1 on Windows 10.  All of them give similar results.  The script requires the psutil package (https://pypi.org/project/psutil/) to monitor memory usage.

- Kevin


import gc
import os

from PyQt5 import QtCore, QtWidgets
import psutil


memory_start = None


class SignalObject(QtCore.QObject):

    mySignal = QtCore.pyqtSignal()


class SlotObject(QtCore.QObject):

    def __init__(self, signal_objects):
        super(SlotObject, self).__init__()

        for cur_signal_obj in signal_objects:
            cur_signal_obj.mySignal.connect(self.my_slot)
            cur_signal_obj.mySignal.connect(self.my_slot2)
            # Immediately disconnecting the signals allows some of the memory to
            # be recovered after the event loop runs, but not much
            # cur_signal_obj.mySignal.disconnect(self.my_slot)
            # cur_signal_obj.mySignal.disconnect(self.my_slot2)

    # @QtCore.pyqtSlot()
    def my_slot(self):
        print("Slot called!")

    # @QtCore.pyqtSlot()
    def my_slot2(self):
        print("Slot2 called!")


def memory_usage():
    process = psutil.Process(os.getpid())
    # with psutils 5.6, uss doesn't work on Windows without elevated permissions
    # return process.memory_full_info().uss
    return process.memory_info().vms


def format_bytes(x):
    for unit in ['', 'Ki', 'Mi', 'Gi', 'Ti', 'Pi', 'Ei', 'Zi']:
        if abs(x) < 1024.0:
            return "%2.1f%sB" % (x, unit)
        x /= 1024.0
    return "%.1fYiB" % x


def create_slot_objects(signal_objects):
    for _ in range(1000):
        SlotObject(signal_objects)


def report_memory(msg):
    mem_now = memory_usage()
    print(f"Memory {msg}: {format_bytes(mem_now)}")
    print(f"\tDifference: {format_bytes(mem_now - memory_start)}")


def report_memory_after_event_loop():
    report_memory("after event loop runs")


def main():
    global memory_start
    app = QtWidgets.QApplication([])
    QtCore.QTimer.singleShot(1000, report_memory_after_event_loop)
    QtCore.QTimer.singleShot(1100, app.quit)
    memory_start = memory_usage()
    print(f"Memory before signal_objects creation: {format_bytes(memory_start)}")
    signal_objects = [SignalObject() for _ in range(100)]
    report_memory("before slot objects creation")
    create_slot_objects(signal_objects)
    del signal_objects
    gc.collect()
    report_memory("after slot objects creation")
    app.exec_()


if __name__ == "__main__":
    main()


_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

QOpenGLWidget

kristof.mulier
Hi,

I'm building an application in PyQt5 for which I need to draw 2D live graphs. I'm looking for a way to render them with the GPU.

1. First approach
---------------------
My first approach was embedding matplotlib into PyQt5. Matplotlib is not so fast, because it relies on CPU-rendering only.

2. Second approach
------------------------
I build my own PyQt5 widget and override the paintEvent() method to draw a few thousand lines with the QPainter() to display the live graph. The graph points are stored in numpy arrays. I simply draw lines between the points. Unfortunately,  QPainter() relies on the CPU.

Right now, it looks like I have several options for GPU-based rendering:

 - QOpenGLWidget
(see https://doc.qt.io/qt-5/qopenglwidget.html). From what I read on the Qt docs, it looks too good to be true. I would simply need to subclass QOpenGLWidget instead of QWidget. Apparently, I eveb don't have to change anything in the paintEvent() method. Is this real? Where are the pitfalls?

 - PyOpenGL
Python bindings for OpenGL. I have no OpenGL experience. I found the following tutorial: https://www.labri.fr/perso/nrougier/python-opengl/ . Great, but I'm afraid to spend a lot of time on this. When I scroll through the tutorial, it appears to me that the focus is on 3D shapes. I don't need that.

 - QGraphicsView
I remember reading something about QGraphicsView() in a StackOverflow answer. It would be a Qt class to draw lots of shapes. But can it handle thousands of tiny lines that form the live graph? Does it rely on the CPU or the GPU for rendering?


Could you point me in the right direction please?





_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: QOpenGLWidget

V. Armando Solé
Hello,

Perhaps you want to give a try at the silx library
(http://www.silx.org/doc/silx/latest/). I provides high-level PyQt
visualization widgets with a dual backend (matplotlib and PyOpenGL) so
that one does not need to know any OpenGL to get the performance.

Best regards,

Armando

On 22.09.2019 17:07, [hidden email] wrote:

> Hi,
>
> I'm building an application in PyQt5 for which I need to draw 2D live
> graphs. I'm looking for a way to render them with the GPU.
>
> 1. FIRST APPROACH
>
> ---------------------
>
> My first approach was embedding matplotlib into PyQt5. Matplotlib is
> not so fast, because _it relies on CPU-rendering only_.
>
> 2. SECOND APPROACH
>
> ------------------------
>
> I build my own PyQt5 widget and override the paintEvent() method to
> draw a few thousand lines with the QPainter() to display the live
> graph. The graph points are stored in numpy arrays. I simply draw
> lines between the points. Unfortunately,  QPainter() relies on the
> CPU.
>
> Right now, it looks like I have several options for GPU-based
> rendering:
>
>  - QOPENGLWIDGET
>  (see https://doc.qt.io/qt-5/qopenglwidget.html). From what I read on
> the Qt docs, it looks too good to be true. I would simply need to
> subclass QOpenGLWidget instead of QWidget. Apparently, I eveb don't
> have to change anything in the paintEvent() method. Is this real?
> Where are the pitfalls?
>
>  - PYOPENGL
> Python bindings for OpenGL. I have no OpenGL experience. I found the
> following tutorial: https://www.labri.fr/perso/nrougier/python-opengl/
> . Great, but I'm afraid to spend a lot of time on this. When I scroll
> through the tutorial, it appears to me that the focus is on 3D shapes.
> I don't need that.
>
>  - QGRAPHICSVIEW
>
> I remember reading something about QGraphicsView() in a StackOverflow
> answer. It would be a Qt class to draw lots of shapes. But can it
> handle thousands of tiny lines that form the live graph? Does it rely
> on the CPU or the GPU for rendering?
>
> Could you point me in the right direction please?
> _______________________________________________
> PyQt mailing list    [hidden email]
> https://www.riverbankcomputing.com/mailman/listinfo/pyqt
_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: QOpenGLWidget

Thomas Caswell
You should also look at pyqtgraph (http://www.pyqtgraph.org) which places a premium on speed.

Depending on exactly how much data you have and exactly how fast you need to go, using blitting it Matplotlib it may be possible to eek out a bit more performance.

Tom

On Sun, Sep 22, 2019 at 8:03 PM V. Armando Sole <[hidden email]> wrote:
Hello,

Perhaps you want to give a try at the silx library
(http://www.silx.org/doc/silx/latest/). I provides high-level PyQt
visualization widgets with a dual backend (matplotlib and PyOpenGL) so
that one does not need to know any OpenGL to get the performance.

Best regards,

Armando

On 22.09.2019 17:07, [hidden email] wrote:
> Hi,
>
> I'm building an application in PyQt5 for which I need to draw 2D live
> graphs. I'm looking for a way to render them with the GPU.
>
> 1. FIRST APPROACH
>
> ---------------------
>
> My first approach was embedding matplotlib into PyQt5. Matplotlib is
> not so fast, because _it relies on CPU-rendering only_.
>
> 2. SECOND APPROACH
>
> ------------------------
>
> I build my own PyQt5 widget and override the paintEvent() method to
> draw a few thousand lines with the QPainter() to display the live
> graph. The graph points are stored in numpy arrays. I simply draw
> lines between the points. Unfortunately,  QPainter() relies on the
> CPU.
>
> Right now, it looks like I have several options for GPU-based
> rendering:
>
>  - QOPENGLWIDGET
>  (see https://doc.qt.io/qt-5/qopenglwidget.html). From what I read on
> the Qt docs, it looks too good to be true. I would simply need to
> subclass QOpenGLWidget instead of QWidget. Apparently, I eveb don't
> have to change anything in the paintEvent() method. Is this real?
> Where are the pitfalls?
>
>  - PYOPENGL
> Python bindings for OpenGL. I have no OpenGL experience. I found the
> following tutorial: https://www.labri.fr/perso/nrougier/python-opengl/
> . Great, but I'm afraid to spend a lot of time on this. When I scroll
> through the tutorial, it appears to me that the focus is on 3D shapes.
> I don't need that.
>
>  - QGRAPHICSVIEW
>
> I remember reading something about QGraphicsView() in a StackOverflow
> answer. It would be a Qt class to draw lots of shapes. But can it
> handle thousands of tiny lines that form the live graph? Does it rely
> on the CPU or the GPU for rendering?
>
> Could you point me in the right direction please?
> _______________________________________________
> PyQt mailing list    [hidden email]
> https://www.riverbankcomputing.com/mailman/listinfo/pyqt
_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt


--
Thomas Caswell
[hidden email]

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak with signal connections

Phil Thompson-5
In reply to this post by Kevin Keating
On 19/09/2019 19:53, Kevin Keating wrote:

> I've found what appears to be a memory leak caused by connecting
> signals to slots that lack a QtCore.pyqtSlot decorator.  I know that
> connections without the pyqtSlot decorator are expected to use more
> memory than connections with the decorator based on
> https://www.codeproject.com/Articles/1123088/PyQt-signal-slot-connection-performance
> In the script I've pasted below, though, connections without the
> pyqtSlot decorator continue to consume memory even after the signal
> has been disconnected and the QObjects have been discarded.  The
> script does the following:
>   - Instantiates a bunch of SignalObjects, which are QObjects with a
> signal, and stores the SingalObjects in a list.
>   - Instantiates a bunch of SlotObjects, which are QObjects with
> slots.  Each SlotObject slot is connected to the signals from all the
> SignalObjects.  The SlotObjects are then immediately discarded.
>   - Discards all SignalObjects.
>   - Runs gc.collect()
>   - Runs the QApplication's event loop for a second in case there are
> any pending DeferredDelete events.
>
> If the SlotObject class uses the QtCore.pyqtSlot decorator, then
> memory usage at the end of the script is the same as what it was at
> the beginning of the script, which makes sense since all the objects
> that get created should be completely destroyed before the script
> finishes.  Here's the output that I get with pyqtSlot decorators:
>         Memory before signal_objects creation: 17.1MiB
>         Memory before slot objects creation: 17.1MiB
>                 Difference: 0.0B
>         Memory after slot objects creation: 17.1MiB
>                 Difference: 0.0B
>         Memory after event loop runs: 17.1MiB
>                 Difference: 0.0B
> If the SlotObject class doesn't use the QtCore.pyqtSlot decorator,
> though, then memory usage at the end of the script is substantially
> higher.  Here's the output that I get without pyqtSlot decorators:
>         Memory before signal_objects creation: 17.0MiB
>         Memory before slot objects creation: 17.0MiB
>                 Difference: 0.0B
>         Memory after slot objects creation: 210.6MiB
>                 Difference: 193.5MiB
>         Memory after event loop runs: 210.6MiB
>                 Difference: 193.5MiB
> Is this expected behavior?  Without pyqtSlot decorators, is there
> anything I can do to recover the 193 MiB of memory other than
> terminating the process?  Thanks!
>
> I've tested the script with Python 3.6.2, PyQt 5.12.2, and Qt 5.12.3
> on Windows 10, Linux, and Mac OS.  I've also tested with Python 3.6.5,
> PyQt 5.13.1, and Qt 5.13.1 on Windows 10.  All of them give similar
> results.  The script requires the psutil package
> (https://pypi.org/project/psutil/) to monitor memory usage.

Are you expecting that the Python interpreter will return unused memory
to the operating system?

Phil
_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak with signal connections

Kevin Keating

On 9/25/2019 9:17:24 AM, Phil Thompson <[hidden email]> wrote:

On 19/09/2019 19:53, Kevin Keating wrote:

> I've found what appears to be a memory leak caused by connecting
> signals to slots that lack a QtCore.pyqtSlot decorator.  I know that
> connections without the pyqtSlot decorator are expected to use more
> memory than connections with the decorator based on
> https://www.codeproject.com/Articles/1123088/PyQt-signal-slot-connection-performance. 
> In the script I've pasted below, though, connections without the
> pyqtSlot decorator continue to consume memory even after the signal
> has been disconnected and the QObjects have been discarded.  The
> script does the following:
>   - Instantiates a bunch of SignalObjects, which are QObjects with a
> signal, and stores the SingalObjects in a list.
>   - Instantiates a bunch of SlotObjects, which are QObjects with
> slots.  Each SlotObject slot is connected to the signals from all the
> SignalObjects.  The SlotObjects are then immediately discarded.
>   - Discards all SignalObjects.
>   - Runs gc.collect()
>   - Runs the QApplication's event loop for a second in case there are
> any pending DeferredDelete events.
>
> If the SlotObject class uses the QtCore.pyqtSlot decorator, then
> memory usage at the end of the script is the same as what it was at
> the beginning of the script, which makes sense since all the objects
> that get created should be completely destroyed before the script
> finishes.  Here's the output that I get with pyqtSlot decorators:
>         Memory before signal_objects creation: 17.1MiB
>         Memory before slot objects creation: 17.1MiB
>                 Difference: 0.0B
>         Memory after slot objects creation: 17.1MiB
>                 Difference: 0.0B
>         Memory after event loop runs: 17.1MiB
>                 Difference: 0.0B
> If the SlotObject class doesn't use the QtCore.pyqtSlot decorator,
> though, then memory usage at the end of the script is substantially
> higher.  Here's the output that I get without pyqtSlot decorators:
>         Memory before signal_objects creation: 17.0MiB
>         Memory before slot objects creation: 17.0MiB
>                 Difference: 0.0B
>         Memory after slot objects creation: 210.6MiB
>                 Difference: 193.5MiB
>         Memory after event loop runs: 210.6MiB
>                 Difference: 193.5MiB
> Is this expected behavior?  Without pyqtSlot decorators, is there
> anything I can do to recover the 193 MiB of memory other than
> terminating the process?  Thanks!
>
> I've tested the script with Python 3.6.2, PyQt 5.12.2, and Qt 5.12.3
> on Windows 10, Linux, and Mac OS.  I've also tested with Python 3.6.5,
> PyQt 5.13.1, and Qt 5.13.1 on Windows 10.  All of them give similar
> results.  The script requires the psutil package
> (https://pypi.org/project/psutil/) to monitor memory usage.

Are you expecting that the Python interpreter will return unused memory
to the operating system?

Phil

My assumption was that the memory usage from the signal connections wouldn't "build up" (i.e. that memory usage would be independent of the number of SignalObjects instantiated), which seems to be what happens with the pyqtSlot decorated version.  In create_slot_objects, each SlotObject is instantiated, which connects the signals, and then should be immediately garbage collected, which should disconnect the signals.  In the version of the script without the pyqtSlot decorators, the number reported for "Memory after slot objects creation" is proportional to the number of SlotObject instances created, which is why I'd assumed memory had leaked, but I could certainly be thinking through things wrong.

- Kevin

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt