QSortFilterProxyModel::sort() not sorting on column types

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

QSortFilterProxyModel::sort() not sorting on column types

J Barchan
I use Qt/PyQt 5.7, the standard distribution supplied for Ubuntu 18.04.

I have a QSortFilterProxyModel wrapped around a QStandardItemModel as its source model.  I try to use QSortFilterProxyModel::sort() to sort by column values.

I have a column of Python type datetime.date.  When I call the sort(), there is no "warning" but nothing happens.  The sort does not rearrange the items at all.  They are initially "randomly" ordered in rows, and the rows simply retain their current order.  It does not even reorder by string value.  It is as though it simply decides they are "unorderable".

When I sort instead by a Python str column they do get reordered.  FWIW, I have another column of Python type decimal.Decimal and sorting by that too does nothing, even though I think that should be convertible to float/double.  However, that may complicate matters so let's stick to the datetime.date case, I am just mentioning it in case it's relevant.

It took me a long time to realise where the problem lies.  I have a workaround: I explicitly change all my values in the model from datetime.date to QDate(value) and now it does sort.

But my understanding/experience from other PyQt methods is that it does this kind of Python type -> QVariant conversion for you behind the scenes itself.  I shouldn't have to change my types or write special code.

Is this a bug?  Is this in my PyQt 5.7 only?

--
Kindest,
Jonathan

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: QSortFilterProxyModel::sort() not sorting on column types

Maurizio Berti
To correctly support sorting for non string values you need to provide your own method.
Also, for numeric values ensure that the data is stored as a number and not as string. For QStandardItemModel you can achieve that by creating QStandardItems without any data value, and then set the numeric data using QStandardItem.setData(value, QtCore.Qt.DisplayValue); remember that the role must be specified, as the setData method of QStandardItem uses the UserRole + 1 role by default. This will solve the sorting for your float column.

To sort by date, it's better to set the QDateTime value as a custom role (>=UserRole) and store the object as it is, while you can choose to set the displayed string once you create the QStandardItems or by using an item delegate, if you need a visual representation.
After that just implement the lessThan() method of QSortFilterProxyModel, which allows to customize how the items are sorted whenever the sort() method is called.

Supposing that you're using the second column for the QDateTime items and you're using a role DateRole = QtCore.Qt.UserRole + 1:

class SortModel(QtCore.QSortFilterProxyModel):
    def lessThan(self, left, right):
        if left.column() == right.column() and left.column() == 1:
            return left.data(DateRole) < right.data(DateRole)
        return QtCore.QSortFilterProxyModel.lessThan(self, left, right)

class MyWidget(QtWidgets.QWidget):
    def __init__(self, *args, **kwargs):
        QtWidgets.QWidget.__init__(self, *args, **kwargs)
        [...]
        self.sourceModel = QtGui.QStandardItemModel()
        self.sortModel = SortModel()
        self.someTable.setModel(self.sortModel)
        self.sortModel.setSourceModel(self.sourceModel)

    def appendFields(self):
        floatItem = QtGui.QStandardItem()
        floatItem.setData(self.getSomeFloatNumber(), QtCore.Qt.DisplayRole)
        someDate = QtCore.QDateTime.currentDateTime()
        item = QtGui.QStandardItem(someDate.toString())
        item.setData(someDate, DateRole)
        self.sourceModel.appendRow([floatItem, dateItem])

I've tested it on an old 5.7.1, but from what you are experiencing it is not a bug, but the expected behavior, since you didn't correctly implement the sorting method and/or didn't provide a correct way for Qt to interpret data for its default sorting rules.

Regards,
Maurizio



Il giorno ven 1 feb 2019 alle ore 10:45 J Barchan <[hidden email]> ha scritto:
I use Qt/PyQt 5.7, the standard distribution supplied for Ubuntu 18.04.

I have a QSortFilterProxyModel wrapped around a QStandardItemModel as its source model.  I try to use QSortFilterProxyModel::sort() to sort by column values.

I have a column of Python type datetime.date.  When I call the sort(), there is no "warning" but nothing happens.  The sort does not rearrange the items at all.  They are initially "randomly" ordered in rows, and the rows simply retain their current order.  It does not even reorder by string value.  It is as though it simply decides they are "unorderable".

When I sort instead by a Python str column they do get reordered.  FWIW, I have another column of Python type decimal.Decimal and sorting by that too does nothing, even though I think that should be convertible to float/double.  However, that may complicate matters so let's stick to the datetime.date case, I am just mentioning it in case it's relevant.

It took me a long time to realise where the problem lies.  I have a workaround: I explicitly change all my values in the model from datetime.date to QDate(value) and now it does sort.

But my understanding/experience from other PyQt methods is that it does this kind of Python type -> QVariant conversion for you behind the scenes itself.  I shouldn't have to change my types or write special code.

Is this a bug?  Is this in my PyQt 5.7 only?

--
Kindest,
Jonathan
_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt


--
È difficile avere una convinzione precisa quando si parla delle ragioni del cuore. - "Sostiene Pereira", Antonio Tabucchi
http://www.jidesk.net

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: QSortFilterProxyModel::sort() not sorting on column types

J Barchan


On Fri, 1 Feb 2019 at 20:50, Maurizio Berti <[hidden email]> wrote:
To correctly support sorting for non string values you need to provide your own method.
Also, for numeric values ensure that the data is stored as a number and not as string. For QStandardItemModel you can achieve that by creating QStandardItems without any data value, and then set the numeric data using QStandardItem.setData(value, QtCore.Qt.DisplayValue); remember that the role must be specified, as the setData method of QStandardItem uses the UserRole + 1 role by default. This will solve the sorting for your float column.

To sort by date, it's better to set the QDateTime value as a custom role (>=UserRole) and store the object as it is, while you can choose to set the displayed string once you create the QStandardItems or by using an item delegate, if you need a visual representation.
After that just implement the lessThan() method of QSortFilterProxyModel, which allows to customize how the items are sorted whenever the sort() method is called.

Supposing that you're using the second column for the QDateTime items and you're using a role DateRole = QtCore.Qt.UserRole + 1:

class SortModel(QtCore.QSortFilterProxyModel):
    def lessThan(self, left, right):
        if left.column() == right.column() and left.column() == 1:
            return left.data(DateRole) < right.data(DateRole)
        return QtCore.QSortFilterProxyModel.lessThan(self, left, right)

class MyWidget(QtWidgets.QWidget):
    def __init__(self, *args, **kwargs):
        QtWidgets.QWidget.__init__(self, *args, **kwargs)
        [...]
        self.sourceModel = QtGui.QStandardItemModel()
        self.sortModel = SortModel()
        self.someTable.setModel(self.sortModel)
        self.sortModel.setSourceModel(self.sourceModel)

    def appendFields(self):
        floatItem = QtGui.QStandardItem()
        floatItem.setData(self.getSomeFloatNumber(), QtCore.Qt.DisplayRole)
        someDate = QtCore.QDateTime.currentDateTime()
        item = QtGui.QStandardItem(someDate.toString())
        item.setData(someDate, DateRole)
        self.sourceModel.appendRow([floatItem, dateItem])

I've tested it on an old 5.7.1, but from what you are experiencing it is not a bug, but the expected behavior, since you didn't correctly implement the sorting method and/or didn't provide a correct way for Qt to interpret data for its default sorting rules.

Regards,
Maurizio



Il giorno ven 1 feb 2019 alle ore 10:45 J Barchan <[hidden email]> ha scritto:
I use Qt/PyQt 5.7, the standard distribution supplied for Ubuntu 18.04.

I have a QSortFilterProxyModel wrapped around a QStandardItemModel as its source model.  I try to use QSortFilterProxyModel::sort() to sort by column values.

I have a column of Python type datetime.date.  When I call the sort(), there is no "warning" but nothing happens.  The sort does not rearrange the items at all.  They are initially "randomly" ordered in rows, and the rows simply retain their current order.  It does not even reorder by string value.  It is as though it simply decides they are "unorderable".

When I sort instead by a Python str column they do get reordered.  FWIW, I have another column of Python type decimal.Decimal and sorting by that too does nothing, even though I think that should be convertible to float/double.  However, that may complicate matters so let's stick to the datetime.date case, I am just mentioning it in case it's relevant.

It took me a long time to realise where the problem lies.  I have a workaround: I explicitly change all my values in the model from datetime.date to QDate(value) and now it does sort.

But my understanding/experience from other PyQt methods is that it does this kind of Python type -> QVariant conversion for you behind the scenes itself.  I shouldn't have to change my types or write special code.

Is this a bug?  Is this in my PyQt 5.7 only?

--
Kindest,
Jonathan
_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt


--
È difficile avere una convinzione precisa quando si parla delle ragioni del cuore. - "Sostiene Pereira", Antonio Tabucchi
http://www.jidesk.net

Thank you for replying, Maurizio.  It has taken me a few days to get back to this.

While I have no argument with your detailed reply (I'm quite sure you know much than I do), I do not understand why any of this necessary, and would appreciate any clarification.

To correctly support sorting for non string values you need to provide your own method.

If I did not make it clear I would remind you of one of my findings.  I start with Python datetime.date variables.  I populate the model via setData(index, dateValue) (no role specified, so EditRole).  At this point no sorting happens.  However, all I have to do is change that to setData(index, QDate(dateValue)) and it does then work, so no "need to provide your own method".

Why is that?  From http://doc.qt.io/qt-5/qsortfilterproxymodel.html#lessThan I am told that the sorting deals in QVariant types, and QMetaType::QDate is among those it handles.  Now, you know much better than I, but everywhere else I use PyQt it seems to convert between Python types and necessary QVariant types invisibly and all is well, but not here?  Why not?

I'm also a little lost about QStandardItem vs QStandardItemModel, if that's significant.  My code creates the (source) model as a QStandardItemModel.  That claims to use http://doc.qt.io/qt-5/qstandarditemmodel.html#setData,whose default role is EditRole.  Nowhere do I create a QStandardItem explicitly, whose http://doc.qt.io/qt-5/qstandarditem.html#setData uses default role is UserRole + 1.  So now I'm not sure which I am using, though I would think only QStandardItemModel?  Is this of relevance to my issue?

Thank you for any explanation.

--
Kindest,
Jonathan

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: QSortFilterProxyModel::sort() not sorting on column types

Maurizio Berti
If I did not make it clear I would remind you of one of my findings.  I start with Python datetime.date variables.  I populate the model via setData(index, dateValue) (no role specified, so EditRole).  At this point no sorting happens.  However, all I have to do is change that to setData(index, QDate(dateValue)) and it does then work, so no "need to provide your own method".

Why is that?  From http://doc.qt.io/qt-5/qsortfilterproxymodel.html#lessThan I am told that the sorting deals in QVariant types, and QMetaType::QDate is among those it handles.  Now, you know much better than I, but everywhere else I use PyQt it seems to convert between Python types and necessary QVariant types invisibly and all is well, but not here?  Why not?


Well, that's interesting: I've always assumed that the DisplayRole wouldn't accept data other than strings and numbers, but you made me realize that it can actually store any data that "supports" printable form (which QDate/Time do). Of course, this could be a small issue if you want a different string format for dates, but if the default one suits your needs, that's fine.
Since lessThan supports a small set of QVariant types (including QDate/Time) and we can use that QVariant type as a Display role, it's good enough.
Besides that, I tried to use setData with random QDateTimes converted QDate using their date() method, and it seems to work fine, without using QDate(dateValue) you mentioned.
To be clear, I tested with this code:

sourceModel.setData(
    sourceModel.index(row, 0), 
    QtCore.QDateTime().currentDateTime().addDays(-randrange(30)).addSecs(-randrange(86400)).date())
 
I'm using an old Qt 5.7.1, it's possible that it's a regression, have you checked up on QtBugs?

 
I'm also a little lost about QStandardItem vs QStandardItemModel, if that's significant.  My code creates the (source) model as a QStandardItemModel.  That claims to use http://doc.qt.io/qt-5/qstandarditemmodel.html#setData,whose default role is EditRole.  Nowhere do I create a QStandardItem explicitly, whose http://doc.qt.io/qt-5/qstandarditem.html#setData uses default role is UserRole + 1.  So now I'm not sure which I am using, though I would think only QStandardItemModel?  Is this of relevance to my issue?

Well, I've always used empty models at start and fill them as needed, since I mostly use dynamic models or prefer to create data in this way. But that's one way.
It doesn't change much from creating a model (even specifying the size, if you can) and then create new rows/columns and setting the data directly from the model.
In this case, the QStandardItems do not exist until the internal indexes are created (trying to access any index is enough, for example by setting or reading data): if you try to access a item(0, 0) on a new QStandardItemModel with a specified size, it will return None. If you try a itemFromIndex(model.index(0,0)) it will return an item.
That said, I don't think it would change much in your case, as soon as you always use the right role when setting data. I think that the default UserRole + 1 of item.setData is just for convenience, but since you always use the model.setData methods that wouldn't be an issue anyway.

Maurizio

--
È difficile avere una convinzione precisa quando si parla delle ragioni del cuore. - "Sostiene Pereira", Antonio Tabucchi
http://www.jidesk.net

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: QSortFilterProxyModel::sort() not sorting on column types

J Barchan


On Wed, 6 Feb 2019 at 13:33, Maurizio Berti <[hidden email]> wrote:
If I did not make it clear I would remind you of one of my findings.  I start with Python datetime.date variables.  I populate the model via setData(index, dateValue) (no role specified, so EditRole).  At this point no sorting happens.  However, all I have to do is change that to setData(index, QDate(dateValue)) and it does then work, so no "need to provide your own method".

Why is that?  From http://doc.qt.io/qt-5/qsortfilterproxymodel.html#lessThan I am told that the sorting deals in QVariant types, and QMetaType::QDate is among those it handles.  Now, you know much better than I, but everywhere else I use PyQt it seems to convert between Python types and necessary QVariant types invisibly and all is well, but not here?  Why not?


Well, that's interesting: I've always assumed that the DisplayRole wouldn't accept data other than strings and numbers, but you made me realize that it can actually store any data that "supports" printable form (which QDate/Time do). Of course, this could be a small issue if you want a different string format for dates, but if the default one suits your needs, that's fine.
Since lessThan supports a small set of QVariant types (including QDate/Time) and we can use that QVariant type as a Display role, it's good enough.
Besides that, I tried to use setData with random QDateTimes converted QDate using their date() method, and it seems to work fine, without using QDate(dateValue) you mentioned.
To be clear, I tested with this code:

sourceModel.setData(
    sourceModel.index(row, 0), 
    QtCore.QDateTime().currentDateTime().addDays(-randrange(30)).addSecs(-randrange(86400)).date())
 
I'm using an old Qt 5.7.1, it's possible that it's a regression, have you checked up on QtBugs?

 
I'm also a little lost about QStandardItem vs QStandardItemModel, if that's significant.  My code creates the (source) model as a QStandardItemModel.  That claims to use http://doc.qt.io/qt-5/qstandarditemmodel.html#setData,whose default role is EditRole.  Nowhere do I create a QStandardItem explicitly, whose http://doc.qt.io/qt-5/qstandarditem.html#setData uses default role is UserRole + 1.  So now I'm not sure which I am using, though I would think only QStandardItemModel?  Is this of relevance to my issue?

Well, I've always used empty models at start and fill them as needed, since I mostly use dynamic models or prefer to create data in this way. But that's one way.
It doesn't change much from creating a model (even specifying the size, if you can) and then create new rows/columns and setting the data directly from the model.
In this case, the QStandardItems do not exist until the internal indexes are created (trying to access any index is enough, for example by setting or reading data): if you try to access a item(0, 0) on a new QStandardItemModel with a specified size, it will return None. If you try a itemFromIndex(model.index(0,0)) it will return an item.
That said, I don't think it would change much in your case, as soon as you always use the right role when setting data. I think that the default UserRole + 1 of item.setData is just for convenience, but since you always use the model.setData methods that wouldn't be an issue anyway.

Maurizio

--
È difficile avere una convinzione precisa quando si parla delle ragioni del cuore. - "Sostiene Pereira", Antonio Tabucchi
http://www.jidesk.net

Hi Maurizio,

Thank you again for your interest in replying.

Besides that, I tried to use setData with random QDateTimes converted QDate using their date() method, and it seems to work fine, without using QDate(dateValue) you mentioned.

This does not matter now, in that I have a workaround, but I'll ask you one more time.  The example you give talks about converting QDateTimes to QDates.  Please forget about QDateTime, that is not the issue I am asking about.  As long as your original code only uses the Qt native types like QDate or QDateTime there will be no problem sorting.  I understand that.

My code uses the Python types from the datetime module.  datetime.date is a Python date type.  In all other places I have made PyQt calls it always correctly converts between, say, Python type datetime.date and Qt type QDate, in either direction, as necessary.  I do not have to do anything special in my code.  The PyQt wrappers for arguments etc. are doing it for me.  This is a huge (good) point about PyQt.

In the sole case of QSortFilterProxyModel.sort(), however, if I fill my model via:

model.setData(row, col, something_of_type_datetime_date)

everything about it works fine except for attempting to call the sort, which seems to do nothing to alter the order.  If I change my model just so that I used:

model.setData(row, col, QDate(something_of_type_datetime_date))

then the sorting also works.

The only question I do not understand/would like answered is: why do I have to use QDate and not datetime.date in the model to make just QSortFilterProxyModel::sort() work, when everything other than that works fine with datetime.date?  Why does PyQt auto-conversion between Python datetime.date and Qt QDate work everywhere other than during QSortFilterProxyModel.sort()?


--
Kindest,
Jonathan

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: QSortFilterProxyModel::sort() not sorting on column types

Maurizio Berti
This does not matter now, in that I have a workaround, but I'll ask you one more time.  The example you give talks about converting QDateTimes to QDates.  Please forget about QDateTime, that is not the issue I am asking about.  As long as your original code only uses the Qt native types like QDate or QDateTime there will be no problem sorting.  I understand that.

My code uses the Python types from the datetime module.  datetime.date is a Python date type.  In all other places I have made PyQt calls it always correctly converts between, say, Python type datetime.date and Qt type QDate, in either direction, as necessary.  I do not have to do anything special in my code.  The PyQt wrappers for arguments etc. are doing it for me.  This is a huge (good) point about PyQt.

I'm sorry, Jonathan: I think I didn't read your first message carefully enough, and then I didn't take the "Python type datetime" into account.

The only question I do not understand/would like answered is: why do I have to use QDate and not datetime.date in the model to make just QSortFilterProxyModel::sort() work, when everything other than that works fine with datetime.date?  Why does PyQt auto-conversion between Python datetime.date and Qt QDate work everywhere other than during QSortFilterProxyModel.sort()?

Well, I'm just guessing an answer here; it's possible that in every other situation the conversion is done automatically since you're doing operations involving unambiguous treatment of the date object, for example when setting a QDateTimeEdit date, or, in this case, display the date in the table field.
The sort() method instead attempts to sort by "guessing" the preferred sorting criteria according to the content of the field, which is not unique. You could have strings, floats and datetimes in the same column and *maybe* that's why the automatic conversion doesn't work properly in this case: while QDateTime is directly recognized by lessThan() along with other QVariant types, Python's datetime is not "native" to Qt and I suppose that it just doesn't know what to do with it (do you want to compare it as a string? as a number? as a QDate? as a QDateTime?), then I think it just "leaves" the object as it is, without doing any comparison.

Again, this is just a guess, tonight I'll check the sources and maybe I'll find out a better answer.

Maurizio

--
È difficile avere una convinzione precisa quando si parla delle ragioni del cuore. - "Sostiene Pereira", Antonio Tabucchi
http://www.jidesk.net

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: QSortFilterProxyModel::sort() not sorting on column types

J Barchan


On Wed, 6 Feb 2019 at 14:47, Maurizio Berti <[hidden email]> wrote:
This does not matter now, in that I have a workaround, but I'll ask you one more time.  The example you give talks about converting QDateTimes to QDates.  Please forget about QDateTime, that is not the issue I am asking about.  As long as your original code only uses the Qt native types like QDate or QDateTime there will be no problem sorting.  I understand that.

My code uses the Python types from the datetime module.  datetime.date is a Python date type.  In all other places I have made PyQt calls it always correctly converts between, say, Python type datetime.date and Qt type QDate, in either direction, as necessary.  I do not have to do anything special in my code.  The PyQt wrappers for arguments etc. are doing it for me.  This is a huge (good) point about PyQt.

I'm sorry, Jonathan: I think I didn't read your first message carefully enough, and then I didn't take the "Python type datetime" into account.

The only question I do not understand/would like answered is: why do I have to use QDate and not datetime.date in the model to make just QSortFilterProxyModel::sort() work, when everything other than that works fine with datetime.date?  Why does PyQt auto-conversion between Python datetime.date and Qt QDate work everywhere other than during QSortFilterProxyModel.sort()?

Well, I'm just guessing an answer here; it's possible that in every other situation the conversion is done automatically since you're doing operations involving unambiguous treatment of the date object, for example when setting a QDateTimeEdit date, or, in this case, display the date in the table field.
The sort() method instead attempts to sort by "guessing" the preferred sorting criteria according to the content of the field, which is not unique. You could have strings, floats and datetimes in the same column and *maybe* that's why the automatic conversion doesn't work properly in this case: while QDateTime is directly recognized by lessThan() along with other QVariant types, Python's datetime is not "native" to Qt and I suppose that it just doesn't know what to do with it (do you want to compare it as a string? as a number? as a QDate? as a QDateTime?), then I think it just "leaves" the object as it is, without doing any comparison.

Again, this is just a guess, tonight I'll check the sources and maybe I'll find out a better answer.

Maurizio

--
È difficile avere una convinzione precisa quando si parla delle ragioni del cuore. - "Sostiene Pereira", Antonio Tabucchi
http://www.jidesk.net

Hi Maurizio,

tonight I'll check the sources and maybe I'll find out a better answer.

That would be most interesting to me, if you feel like doing it.

Think of this.  When I go model.setData(row, col, python_date), the C++ definition for setData() takes a QVariant for the value.  Somewhere along the line, PyQt auto-converts Python datetime.date to a QVariant of type QMetaType::QDate containing a QDate, right?  (And similarly unwraps if I call model.data(row, col).)  At least that's my understanding.

When I go SortFilterProxyModel::sort(), I read the documentation at http://doc.qt.io/qt-5/qsortfilterproxymodel.html#lessThan.  I see QMetaType::QDate among the QVariant types it says it can handle.  When it comes to compare the values in my model, I'm expecting it to similarly come across my Python datetime.date values and auto-convert them to that QVariant which it knows it can handle.  But it does not appear to do that.

Meanwhile, that doc states it can also handle QMetaType::QStringQString is also a Qt type, not a Python one.  But I don't bother to put QStrings into my model, I just put in plain Python strs, and yet I find that those do sort.  Why is that OK but the date ones not?

(Hmm, I'm beginning to wonder/think: the example I gave with setData()/data() it must do whatever converting on the argument/return result there & then, from/to Python.  With the SortFilterProxyModel::sort(), encountering values is going on during the sort, which is written in C++, so that code will not know about Python type conversions.  Is this what it's all about, the reason why no conversion takes place during sort() only?)

To make it even more confusing.  The above doc link for SortFilterProxyModel::lessThan() states:

Any other type will be converted to a QString using QVariant::toString().

I therefore thought if my native Python datetime.date was "not recognised", it would at minimum do a string sort.  Might not be what I intended for my dates, but at least I'd see something happening in the way of reordering.  Instead, I see nothing gets moved, the original order is simply preserved, just as though for whatever reason the sort has done nothing.  No complaints, just nothing happens.

Thanks for your time!



--
Kindest,
Jonathan

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: QSortFilterProxyModel::sort() not sorting on column types

Maurizio Berti
Il giorno mer 6 feb 2019 alle ore 16:27 J Barchan <[hidden email]> ha scritto:
Think of this.  When I go model.setData(row, col, python_date), the C++ definition for setData() takes a QVariant for the value.  Somewhere along the line, PyQt auto-converts Python datetime.date to a QVariant of type QMetaType::QDate containing a QDate, right?  (And similarly unwraps if I call model.data(row, col).)  At least that's my understanding.

Nope :-)
Any kind of object can be assigned to a (any) DataRole to a valid QModelIndex. I don't know how it works on C++ specifically (I think it doesn't change that much), but in the PyQt world whenever you setData() with a Python object as argument, the object remains intact and is stored in the model as it is, despite the type of the object itself (or, probably, a "PyQtObject" as it was called for PyQt4 and SIP version 1, which probably is a special PyQt "variant" of QVariant, but I'm just guessing here - anyway, it's returned in its original form when accessed).
The conversion to a Qt type is only done whenever it's needed, and is usually automatic and transparent to the Python interface.


Meanwhile, that doc states it can also handle QMetaType::QStringQString is also a Qt type, not a Python one.  But I don't bother to put QStrings into my model, I just put in plain Python strs, and yet I find that those do sort.  Why is that OK but the date ones not?

If you use SIP v2 (which is default on PyQt5, but can be set for PyQt4 as well as with other compatible types) the QString class is not available. You cannot even import it, as it doesn't exist at all: PyQt automatically treats all strings as QStrings and viceversa as it's done with numeric values - this means that Python strings "are" QStrings. Other and more "complex" object types are automatically converted whenever it's needed, but they original state is maintained, so a Python datetime is NOT a QDateTime, but can be used as it would.
While it can be a small issue (as some useful QString processing methods are obviously not available), it's a huge advantage since you don't have to convert everytime between Qt and Python objects. The standard SIPv1 mode of python objects in PyQt4 was a continuous headache, as you'd have always needed to use the toPyObject() conversion, also creating issues with the annoying differences and issues between Python 2's str and unicode types and conversions.
So, long story short, the sort() of strings works just because Python strings are "virtually" QStrings.


To make it even more confusing.  The above doc link for SortFilterProxyModel::lessThan() states:

Any other type will be converted to a QString using QVariant::toString().
 
I think that, from the PyQt point of view, this means that it converts the object to a string only if the object is a known QVariant type that can be converted to a "QString". I'm not really sure about this, but I'm assuming this from the fact that other non-Qt objects are not converted to strings in models even when required (for example, displaying data in an item view), even if they do have a __str__() magic method implemented. Interestingly enough, the public toString() method of QVariant is not available on PyQt5 and cannot implemented therefore to create custom QVariant classes.


I therefore thought if my native Python datetime.date was "not recognised", it would at minimum do a string sort.  Might not be what I intended for my dates, but at least I'd see something happening in the way of reordering.  Instead, I see nothing gets moved, the original order is simply preserved, just as though for whatever reason the sort has done nothing.  No complaints, just nothing happens.

Considering what explained before, you might agree that lessThan() cannot do any kind of sorting with "unknown" PyQt objects. It gets an unknown object, doesn't "know" how to sort it against another (even similar) object, then it leaves the order unsorted.


Thanks for your time!

Thank you! Actively thinking about these problems helps having a better consciousness about how (PyQt) objects actually work. :-)

Maurizio

--
È difficile avere una convinzione precisa quando si parla delle ragioni del cuore. - "Sostiene Pereira", Antonio Tabucchi
http://www.jidesk.net

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: QSortFilterProxyModel::sort() not sorting on column types

J Barchan


On Wed, 6 Feb 2019 at 22:27, Maurizio Berti <[hidden email]> wrote:
Il giorno mer 6 feb 2019 alle ore 16:27 J Barchan <[hidden email]> ha scritto:
Think of this.  When I go model.setData(row, col, python_date), the C++ definition for setData() takes a QVariant for the value.  Somewhere along the line, PyQt auto-converts Python datetime.date to a QVariant of type QMetaType::QDate containing a QDate, right?  (And similarly unwraps if I call model.data(row, col).)  At least that's my understanding.

Nope :-)
Any kind of object can be assigned to a (any) DataRole to a valid QModelIndex. I don't know how it works on C++ specifically (I think it doesn't change that much), but in the PyQt world whenever you setData() with a Python object as argument, the object remains intact and is stored in the model as it is, despite the type of the object itself (or, probably, a "PyQtObject" as it was called for PyQt4 and SIP version 1, which probably is a special PyQt "variant" of QVariant, but I'm just guessing here - anyway, it's returned in its original form when accessed).
The conversion to a Qt type is only done whenever it's needed, and is usually automatic and transparent to the Python interface.


Meanwhile, that doc states it can also handle QMetaType::QStringQString is also a Qt type, not a Python one.  But I don't bother to put QStrings into my model, I just put in plain Python strs, and yet I find that those do sort.  Why is that OK but the date ones not?

If you use SIP v2 (which is default on PyQt5, but can be set for PyQt4 as well as with other compatible types) the QString class is not available. You cannot even import it, as it doesn't exist at all: PyQt automatically treats all strings as QStrings and viceversa as it's done with numeric values - this means that Python strings "are" QStrings. Other and more "complex" object types are automatically converted whenever it's needed, but they original state is maintained, so a Python datetime is NOT a QDateTime, but can be used as it would.
While it can be a small issue (as some useful QString processing methods are obviously not available), it's a huge advantage since you don't have to convert everytime between Qt and Python objects. The standard SIPv1 mode of python objects in PyQt4 was a continuous headache, as you'd have always needed to use the toPyObject() conversion, also creating issues with the annoying differences and issues between Python 2's str and unicode types and conversions.
So, long story short, the sort() of strings works just because Python strings are "virtually" QStrings.


To make it even more confusing.  The above doc link for SortFilterProxyModel::lessThan() states:

Any other type will be converted to a QString using QVariant::toString().
 
I think that, from the PyQt point of view, this means that it converts the object to a string only if the object is a known QVariant type that can be converted to a "QString". I'm not really sure about this, but I'm assuming this from the fact that other non-Qt objects are not converted to strings in models even when required (for example, displaying data in an item view), even if they do have a __str__() magic method implemented. Interestingly enough, the public toString() method of QVariant is not available on PyQt5 and cannot implemented therefore to create custom QVariant classes.


I therefore thought if my native Python datetime.date was "not recognised", it would at minimum do a string sort.  Might not be what I intended for my dates, but at least I'd see something happening in the way of reordering.  Instead, I see nothing gets moved, the original order is simply preserved, just as though for whatever reason the sort has done nothing.  No complaints, just nothing happens.

Considering what explained before, you might agree that lessThan() cannot do any kind of sorting with "unknown" PyQt objects. It gets an unknown object, doesn't "know" how to sort it against another (even similar) object, then it leaves the order unsorted.


Thanks for your time!

Thank you! Actively thinking about these problems helps having a better consciousness about how (PyQt) objects actually work. :-)

Maurizio

--
È difficile avere una convinzione precisa quando si parla delle ragioni del cuore. - "Sostiene Pereira", Antonio Tabucchi
http://www.jidesk.net

Hi Maurizio,

Thank you for all the time you have spent on this!  We'd better wrap it up, as at the end of the day although I got confused & stuck when I first tried it I do have an acceptable workaround for sorting of storing QDate(python_date) instead of the normal plain python_date in the model, which will do me in this instance.

Nope :-)
Any kind of object can be assigned to a (any) DataRole to a valid QModelIndex.

I suspect this is the key to my misunderstanding.  When I pass a Python datetime to some PyQt function which expects a QDate (I can't think of one), or a Python decimal.Decimal where PyQt/Qt function expects a float, they get auto-converted, right?  But when I pass those to, say, QStandardItemModel.setData(), you are saying there is no conversion and the Python object is stored as-is, right?  And when QStandardItemModel.sort()/lessThan() comes along and looks at the data which is a Python datetime or Decimal type, this does not count as the same situation (e.g. explicit argument to function) where PyQt would auto-convert as necessary --- quite possibly because we are down in C++ Qt code by then which knows nothing about Python --- so it fails.  Is that about right?

Many thanks.


--
Kindest,
Jonathan

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: QSortFilterProxyModel::sort() not sorting on column types

Maurizio Berti
Any kind of object can be assigned to a (any) DataRole to a valid QModelIndex.

I suspect this is the key to my misunderstanding.  When I pass a Python datetime to some PyQt function which expects a QDate (I can't think of one), or a Python decimal.Decimal where PyQt/Qt function expects a float, they get auto-converted, right?

I think that "basic" standard data types are used transparently between Python and Qt. I don't know how this exactly works behind the scenes (expecially for "advanced" numeric types, like qint8 or qlonglong), I think that standard types like int, float and strings are just used as they are represented internally by the Python interpreter, while PyQt automatically offers automatic conversion for more advanced classes whenever it makes sense.

In the old PyQt4 docs, it explicitly says:
A Python date object may be used whenever a QDate is expected.
The same appears for QDateTime and QTime, but for other "compatible" types also, like when using string objects for QByteArrays.

This means that you can use a Python date object as an argument for any method that expects a QDate (for example, the QDateTimeEdit.setDate()), including the QDate __init__ itself (that's your case), since new QDate objects can be also created using the QDate(QDate) initialization method.
So, a Python date/time is "accepted" as an argument and automatically converted to a Qt date/time object for the meanings of the method called. The original object is unchanged and there's no reference to it from the Qt side of things.
 
But when I pass those to, say, QStandardItemModel.setData(), you are saying there is no conversion and the Python object is stored as-is, right?  And when QStandardItemModel.sort()/lessThan() comes along and looks at the data which is a Python datetime or Decimal type, this does not count as the same situation (e.g. explicit argument to function) where PyQt would auto-convert as necessary --- quite possibly because we are down in C++ Qt code by then which knows nothing about Python --- so it fails.  Is that about right?

Quite correct. Remember that an item model doesn't care about the type of data stored (and it shouldn't), it's only an abstract categorization of a collection of objects.
Think of it as an advanced Python list, where you could put any kind of object in it. It wouldn't make sense to "change" the object type when you add it to the list, right?
The only scenarios in which the data type becomes important is when the data is represented (in an item view) or some level of interaction is required, such as finding data or comparing. For visual representation it behaves like printing the Python list: if the object has __str__ implemented it will show the returned string, even if the object is not a string, so it will display the content if the object is "known" as printable for Qt; but if you try to sort a non builtin Python type, you will likely get back the original sorting, since it doesn't know how to handle the data.

I wanted to do a couple of test to better demonstrate all this, but it seems that I'm not able to automatically display a Python date object in a QTableView (I thought I did it while testing yesterday, but maybe I was wrong), requiring an item delegate to keep the original object and actually display the date.
Just out of curiosity, did you actually see the dates in the fields for which you used Python date objects as setData argument (without the QDate conversion)?

Finally, about your last sentence, it's not that "Qt knows nothing about Python". The model can "see" the stored object, but it's an unknown data type (remember, it has not been "converted", when storing data to a model it should not change its contents), thus the it has no way to know how to sort it.

When sorting, models usually call the private method isVariantLessThan() you can see here:
You'll see that the default case switch behaves as the QVariant string type, trying to transform the QVariant to a string. Since PyQt5 hides some features due to the "transparency" of data types (there are no QStrings nor QVariants by default), I'll show this with PyQt4, which probably better explains what happens under the hood. Remember that with SIP v1 (the default for PyQt4) data models usually returned QVariants, requiring the transformation to toPyObject to get the original form.

>>> from PyQt4 import QtCore
>>> intVariant = QtCore.QVariant(float(5))
>>> intVariant.typeName()
'double'
>>> intVariant.toString()
PyQt4.QtCore.QString(u'5')
>>> from datetime import date
>>> dateVariant = QtCore.QVariant(date.today())
>>> dateVariant.typeName()
'PyQt_PyObject'
>>> dateVariant.toString()
PyQt4.QtCore.QString(u'')

As you can see, the intVariant is correctly "transformed" to an internal double type and, while isVariantLessThan will obviously use the default numeric comparison, its toString method returns the string representation. In the second case, the QVariant type is indeed a "special" Qt object type, and its string is null. Some might object that, if the object has a __str__ method implemented, it should return that, but that's open to (another) debate.
At this point, lessThan will always receive False for unknown data types, leaving the whole sorting unchanged.

Well, that was educational for me too :-)
Hope this helps you as well!

Regards,
Maurizio

--
È difficile avere una convinzione precisa quando si parla delle ragioni del cuore. - "Sostiene Pereira", Antonio Tabucchi
http://www.jidesk.net

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: QSortFilterProxyModel::sort() not sorting on column types

J Barchan


On Thu, 7 Feb 2019 at 23:22, Maurizio Berti <[hidden email]> wrote:
Any kind of object can be assigned to a (any) DataRole to a valid QModelIndex.

I suspect this is the key to my misunderstanding.  When I pass a Python datetime to some PyQt function which expects a QDate (I can't think of one), or a Python decimal.Decimal where PyQt/Qt function expects a float, they get auto-converted, right?

I think that "basic" standard data types are used transparently between Python and Qt. I don't know how this exactly works behind the scenes (expecially for "advanced" numeric types, like qint8 or qlonglong), I think that standard types like int, float and strings are just used as they are represented internally by the Python interpreter, while PyQt automatically offers automatic conversion for more advanced classes whenever it makes sense.

In the old PyQt4 docs, it explicitly says:
A Python date object may be used whenever a QDate is expected.
The same appears for QDateTime and QTime, but for other "compatible" types also, like when using string objects for QByteArrays.

This means that you can use a Python date object as an argument for any method that expects a QDate (for example, the QDateTimeEdit.setDate()), including the QDate __init__ itself (that's your case), since new QDate objects can be also created using the QDate(QDate) initialization method.
So, a Python date/time is "accepted" as an argument and automatically converted to a Qt date/time object for the meanings of the method called. The original object is unchanged and there's no reference to it from the Qt side of things.
 
But when I pass those to, say, QStandardItemModel.setData(), you are saying there is no conversion and the Python object is stored as-is, right?  And when QStandardItemModel.sort()/lessThan() comes along and looks at the data which is a Python datetime or Decimal type, this does not count as the same situation (e.g. explicit argument to function) where PyQt would auto-convert as necessary --- quite possibly because we are down in C++ Qt code by then which knows nothing about Python --- so it fails.  Is that about right?

Quite correct. Remember that an item model doesn't care about the type of data stored (and it shouldn't), it's only an abstract categorization of a collection of objects.
Think of it as an advanced Python list, where you could put any kind of object in it. It wouldn't make sense to "change" the object type when you add it to the list, right?
The only scenarios in which the data type becomes important is when the data is represented (in an item view) or some level of interaction is required, such as finding data or comparing. For visual representation it behaves like printing the Python list: if the object has __str__ implemented it will show the returned string, even if the object is not a string, so it will display the content if the object is "known" as printable for Qt; but if you try to sort a non builtin Python type, you will likely get back the original sorting, since it doesn't know how to handle the data.

I wanted to do a couple of test to better demonstrate all this, but it seems that I'm not able to automatically display a Python date object in a QTableView (I thought I did it while testing yesterday, but maybe I was wrong), requiring an item delegate to keep the original object and actually display the date.
Just out of curiosity, did you actually see the dates in the fields for which you used Python date objects as setData argument (without the QDate conversion)?

Finally, about your last sentence, it's not that "Qt knows nothing about Python". The model can "see" the stored object, but it's an unknown data type (remember, it has not been "converted", when storing data to a model it should not change its contents), thus the it has no way to know how to sort it.

When sorting, models usually call the private method isVariantLessThan() you can see here:
You'll see that the default case switch behaves as the QVariant string type, trying to transform the QVariant to a string. Since PyQt5 hides some features due to the "transparency" of data types (there are no QStrings nor QVariants by default), I'll show this with PyQt4, which probably better explains what happens under the hood. Remember that with SIP v1 (the default for PyQt4) data models usually returned QVariants, requiring the transformation to toPyObject to get the original form.

>>> from PyQt4 import QtCore
>>> intVariant = QtCore.QVariant(float(5))
>>> intVariant.typeName()
'double'
>>> intVariant.toString()
PyQt4.QtCore.QString(u'5')
>>> from datetime import date
>>> dateVariant = QtCore.QVariant(date.today())
>>> dateVariant.typeName()
'PyQt_PyObject'
>>> dateVariant.toString()
PyQt4.QtCore.QString(u'')

As you can see, the intVariant is correctly "transformed" to an internal double type and, while isVariantLessThan will obviously use the default numeric comparison, its toString method returns the string representation. In the second case, the QVariant type is indeed a "special" Qt object type, and its string is null. Some might object that, if the object has a __str__ method implemented, it should return that, but that's open to (another) debate.
At this point, lessThan will always receive False for unknown data types, leaving the whole sorting unchanged.

Well, that was educational for me too :-)
Hope this helps you as well!

Regards,
Maurizio

--
È difficile avere una convinzione precisa quando si parla delle ragioni del cuore. - "Sostiene Pereira", Antonio Tabucchi
http://www.jidesk.net

Hi Maurizio,

Yet again, thank you for your interest.  Though my head is beginning to ache over all this now! :)

Again, I think you may have hit the nail on the head when you write:

but it seems that I'm not able to automatically display a Python date object in a QTableView (I thought I did it while testing yesterday, but maybe I was wrong), requiring an item delegate to keep the original object and actually display the date.
Just out of curiosity, did you actually see the dates in the fields for which you used Python date objects as setData argument (without the QDate conversion)?

Until yesterday, I would have claimed that, yes, the QTableView does show these python datetimes from the model natively.  Then I noticed in something new I was doing that they were coming out blank.  I am beginning to believe that you are quite right and it does not auto-handle/convert them....

Please understand: until a year or so ago I had never used Python or Qt.  Now I develop & maintain (on my own, so no one I can ask) an existing PyQt project with tens of thousands of lines of existing code.  Written by different people over years with not a single comment and impenetrable logic.  This means I don't always know what's happening where.

Now, if my understanding that PyQt was auto-converting datetime <-> QDate, at least in the context of model data/table view, turns out to be baseless, then this all begins to make a lot  more sense.  If the table view won't display a Python datetime, because it knows nothing about it, then I can appreciate it will have similar problem not being able to sort by it.

Since you are enjoying yourself(!), I also claimed that a column with Python type decimal.Decimal in it (that's what we use for all our monetary numbers, we need two decimal places and no loss of precision so float is no good) fails to sort in just the same way as dateime.date.  Again, there I thought some auto-conversion was going to float which the sort does know about.  You might like to see how that behaves for you?  If that too does not sort and does not display correctly (i.e. it's not handled intrinsically by model/view code), then again I wonder whether we don't have our own explicit code in, say, .data(row, col, Qt.DisplayRole) which converts to a fixed-2-decimals string output.  So again if it can't be displayed natively I would understand why it can't be sorted on.

I would do this myself today if I could.  However as of yesterday our whole web site no longer works, following a move by our ISP from PHP 5.6 to 7.x.  Reading up, I discover that for years until PHP 5.6 it used a "mysql" extension library for all database interaction, and now that has been removed and replaced with a new "mysqli" extension library, which is different, so our whole web site cannot access its database (inc. WordPress).  So today, having never done any PHP and knowing nothing about how it all works, I must urgently learn and fix the whole site... trust you understand!

Ciao, e grazie :)

--
Kindest,
Jonathan

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
Reply | Threaded
Open this post in threaded view
|

Re: QSortFilterProxyModel::sort() not sorting on column types

Maurizio Berti
Yet again, thank you for your interest.  Though my head is beginning to ache over all this now! :)

:-)


Until yesterday, I would have claimed that, yes, the QTableView does show these python datetimes from the model natively.  Then I noticed in something new I was doing that they were coming out blank.  I am beginning to believe that you are quite right and it does not auto-handle/convert them....

So I suppose I was right (meaning I was wrong in the first place): datetime object are not displayed in item views!


Please understand: until a year or so ago I had never used Python or Qt.  Now I develop & maintain (on my own, so no one I can ask) an existing PyQt project with tens of thousands of lines of existing code.  Written by different people over years with not a single comment and impenetrable logic.  This means I don't always know what's happening where.

Don't worry. I've been developing a program in the last year, it's more than 40k lines of code right now and sometimes I'm completely lost in it. I couldn't even imagine how that would feel when "inheriting" somebody else's code!
 

Since you are enjoying yourself(!), I also claimed that a column with Python type decimal.Decimal in it (that's what we use for all our monetary numbers, we need two decimal places and no loss of precision so float is no good) fails to sort in just the same way as dateime.date.  Again, there I thought some auto-conversion was going to float which the sort does know about.  You might like to see how that behaves for you?  If that too does not sort and does not display correctly (i.e. it's not handled intrinsically by model/view code), then again I wonder whether we don't have our own explicit code in, say, .data(row, col, Qt.DisplayRole) which converts to a fixed-2-decimals string output.  So again if it can't be displayed natively I would understand why it can't be sorted on.

From what you're describing (including the fact that you don't know the whole codebase yet), I'd assume that there could be some item delegate in use somewhere, which can handle non-Qt data types.
If you're not familiar with them, they're the Qt way to "display" data in an item view (tables, etc.) whenever the printable data content is not useful enough.
I use them a lot; for example, I've a sqlite database with a "tag" field which contains a json list, and I use an item delegate linked to another database table to display the content as a fancy coloured tag list with rounded corners and mouse hover highlight.
Long story short, item delegates can display custom data type content, so it is possible that this is your case: somewhere in the code a delegate was set for an item view (or a column of a table view) and that's why you can see non-Qt objects as strings, possibly by using displayText() or even paint().

 
I would do this myself today if I could. 
[...]
So today, having never done any PHP and knowing nothing about how it all works, I must urgently learn and fix the whole site... trust you understand!

Ciao, e grazie :)

Well, I know a bit about PHP (and I've learned to hate it); all I can say to you is: "good luck, I'm sorry" :-D

Grazie a te ;-)
Maurizio

--
È difficile avere una convinzione precisa quando si parla delle ragioni del cuore. - "Sostiene Pereira", Antonio Tabucchi
http://www.jidesk.net

_______________________________________________
PyQt mailing list    [hidden email]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt