[Django] #30439: ngettext broken for certain locales due to catalog merging

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[Django] #30439: ngettext broken for certain locales due to catalog merging

Django
#30439: ngettext broken for certain locales due to catalog merging
------------------------------------------------+------------------------
               Reporter:  Michal Čihař          |          Owner:  nobody
                   Type:  Bug                   |         Status:  new
              Component:  Internationalization  |        Version:  2.2
               Severity:  Normal                |       Keywords:
           Triage Stage:  Unreviewed            |      Has patch:  0
    Needs documentation:  0                     |    Needs tests:  0
Patch needs improvement:  0                     |  Easy pickings:  0
                  UI/UX:  0                     |
------------------------------------------------+------------------------
 When locales have different plural forms, the ngettext can be easily
 broken because it uses plural equation from the first loaded gettext
 catalog for all of them.

 For example in Czech locale, there are different plural equations being
 used even inside Django with either 3 or 4 plural forms. When the one with
 4 forms is loaded first, Django is looking for non existing plural in
 catalogs.

 Reproducer in Django 2.2.1:

 {{{

 >>> from django.utils import translation
 >>> translation.activate('cs')
 >>> translation.ngettext('This password is too short. It must contain at
 least %(min_length)d character.', 'This password is too short. It must
 contain at least %(min_length)d characters.', 1)
 'Heslo je příliš krátké. Musí mít délku aspoň %(min_length)d znak.'
 >>> translation.ngettext('This password is too short. It must contain at
 least %(min_length)d character.', 'This password is too short. It must
 contain at least %(min_length)d characters.', 10)
 'This password is too short. It must contain at least %(min_length)d
 characters.'
 }}}


 The second invocation is trying to find 4th plural in 3 plurals catalog.

--
Ticket URL: <https://code.djangoproject.com/ticket/30439>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/048.716a8b67511ac2abb3edeaf6b2b2b040%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30439: ngettext broken for certain locales due to catalog merging

Django
#30439: ngettext broken for certain locales due to catalog merging
-------------------------------------+-------------------------------------
     Reporter:  Michal Čihař         |                    Owner:  Claude
                                     |  Paroz
         Type:  Bug                  |                   Status:  assigned
    Component:                       |                  Version:  2.2
  Internationalization               |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------
Changes (by Claude Paroz):

 * status:  new => assigned
 * owner:  nobody => Claude Paroz
 * stage:  Unreviewed => Accepted


Comment:

 I'm partly responsible of this situation, because I'm not always merging
 translation files when only the plural string changes (essentially to save
 some .mo files which are the main cause of Django repository growing
 size). We are unfortunately completely dependent on Transifex about these
 plural string changes, which are not always desired by translation teams.

 Whatever, we should definitely commit all files with plural string
 changes.

--
Ticket URL: <https://code.djangoproject.com/ticket/30439#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/063.5a326a115f952eab4f32eb8639a55e1b%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30439: ngettext broken for certain locales due to catalog merging

Django
In reply to this post by Django
#30439: ngettext broken for certain locales due to catalog merging
-------------------------------------+-------------------------------------
     Reporter:  Michal Čihař         |                    Owner:  Claude
                                     |  Paroz
         Type:  Bug                  |                   Status:  assigned
    Component:                       |                  Version:  2.2
  Internationalization               |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Claude Paroz):

 Let's do that soon before the 2.2.2 release, so we can get new
 translations in the same commit.

--
Ticket URL: <https://code.djangoproject.com/ticket/30439#comment:2>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/063.e1a55321617001decedcdf5ad556b3ea%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30439: ngettext broken for certain locales due to catalog merging

Django
In reply to this post by Django
#30439: ngettext broken for certain locales due to catalog merging
-------------------------------------+-------------------------------------
     Reporter:  Michal Čihař         |                    Owner:  Claude
                                     |  Paroz
         Type:  Bug                  |                   Status:  assigned
    Component:                       |                  Version:  2.2
  Internationalization               |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Michal Čihař):

 The very same situation happens with third party app localization (this is
 where I noticed it first). I think it's not reasonable to expect
 everything will use same plural forms.

--
Ticket URL: <https://code.djangoproject.com/ticket/30439#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/063.0fade40180ff5093d4a6a7a232a3fff5%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30439: ngettext broken for certain locales due to catalog merging

Django
In reply to this post by Django
#30439: ngettext broken for certain locales due to catalog merging
-------------------------------------+-------------------------------------
     Reporter:  Michal Čihař         |                    Owner:  Claude
                                     |  Paroz
         Type:  Bug                  |                   Status:  assigned
    Component:                       |                  Version:  2.2
  Internationalization               |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Claude Paroz):

 Then do you have a suggestion for an alternate implementation?

--
Ticket URL: <https://code.djangoproject.com/ticket/30439#comment:4>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/063.ee4aee0abd3231c1b85df2e493cc9bff%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30439: ngettext broken for certain locales due to catalog merging

Django
In reply to this post by Django
#30439: ngettext broken for certain locales due to catalog merging
-------------------------------------+-------------------------------------
     Reporter:  Michal Čihař         |                    Owner:  Claude
                                     |  Paroz
         Type:  Bug                  |                   Status:  assigned
    Component:                       |                  Version:  2.2
  Internationalization               |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Michal Čihař):

 It would be necessary to avoid merging of the catalogs and perform lookup
 over several catalogs. With merging, the lookup in the catalog can not
 respect plural equation defined in the file. With Czech language I believe
 this is issue only because of a bug in the plural form (see below), but
 there are language were several different plural forms are widely used
 (for example [https://hosted.weblate.org/languages/lv/#information
 Latvian], where they have same number of plurals, but reverse ordering of
 the strings in one of them).


 Going back to Czech locale, the plural equation for Czech (which you
 apparently got from Transifex) doesn't make sense at all, because it
 defines 4 plural forms, but it never evaluates to 2 because `n % 1` is
 always 0:

 `(n == 1 && n % 1 == 0) ? 0 : (n >= 2 && n <= 4 && n % 1 == 0) ? 1: (n % 1
 != 0 ) ? 2 : 3`

 So the correct fix here would be revert to original formula and blame
 Transifex to fix this. Probably some damage was done to the plurals
 translation meanwhile...

 PS: I've seen such messed up plurals already in several projects which
 have decided to migrate from Transifex to [https://weblate.org/ Weblate]
 (which I develop), not sure if anybody has reported this to them.

--
Ticket URL: <https://code.djangoproject.com/ticket/30439#comment:5>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/063.2e18159952c0c446503ba6668d7ef1b8%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30439: ngettext broken for certain locales due to catalog merging

Django
In reply to this post by Django
#30439: ngettext broken for certain locales due to catalog merging
-------------------------------------+-------------------------------------
     Reporter:  Michal Čihař         |                    Owner:  Claude
                                     |  Paroz
         Type:  Bug                  |                   Status:  assigned
    Component:                       |                  Version:  2.2
  Internationalization               |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Claude Paroz):

 > It would be necessary to avoid merging of the catalogs and perform
 lookup over several catalogs. With merging, the lookup in the catalog can
 not respect plural equation defined in the file.

 Exactly, are you up to try some proof of concept patch?

 > So the correct fix here would be revert to original formula and blame
 Transifex to fix this.

 Unfortunately, it's very difficult to make Transifex team move on these
 subjects. We've been bitten by this with Polish in the past.

--
Ticket URL: <https://code.djangoproject.com/ticket/30439#comment:6>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/063.efea06abbf706903f1b7a447e7a3e125%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30439: ngettext broken for certain locales due to catalog merging

Django
In reply to this post by Django
#30439: ngettext broken for certain locales due to catalog merging
-------------------------------------+-------------------------------------
     Reporter:  Michal Čihař         |                    Owner:  Claude
                                     |  Paroz
         Type:  Bug                  |                   Status:  assigned
    Component:                       |                  Version:  2.2
  Internationalization               |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Michal Čihař):

 I was thinking about something like this:

 {{{
   class DjangoTranslation:
     def __init__(language, domain='django', localedirs=None):
         self.domain = domain
         self.translations = []
         # Here would be rest of init logic based on current code

     def merge(self, other):
         if not isinstance(other, NullTranslations):
             self.translations.append(other)

     def gettext(message):
         for translation in self.translations:
             tmsg = translation._catalog.get(message, None)
             if tmsg is not None:
                 return tmsg
         return message
 }}}

 It would perform slightly worse than current code especially in case there
 will be many apps with locales.

--
Ticket URL: <https://code.djangoproject.com/ticket/30439#comment:7>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/063.9a8d0ddaf0e67c170ef749bd065c3cc8%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30439: ngettext broken for certain locales due to catalog merging

Django
In reply to this post by Django
#30439: ngettext broken for certain locales due to catalog merging
-------------------------------------+-------------------------------------
     Reporter:  Michal Čihař         |                    Owner:  Claude
                                     |  Paroz
         Type:  Bug                  |                   Status:  assigned
    Component:                       |                  Version:  2.2
  Internationalization               |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Claude Paroz):

 Sacrificing performance is never rejoicing, but I'm not sure we have the
 choice here. And the cost shouldn't be too high hopefully.

--
Ticket URL: <https://code.djangoproject.com/ticket/30439#comment:8>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/063.20660f573d4be713666d3a74a25782c3%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30439: ngettext broken for certain locales due to catalog merging

Django
In reply to this post by Django
#30439: ngettext broken for certain locales due to catalog merging
-------------------------------------+-------------------------------------
     Reporter:  Michal Čihař         |                    Owner:  Claude
                                     |  Paroz
         Type:  Bug                  |                   Status:  assigned
    Component:                       |                  Version:  2.2
  Internationalization               |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Helder Correia):

 Strings from django.contrib.postgres are not being translated. Is this the
 same issue?

 {{{
 #!python
 >>> from django.utils.translation import activate, gettext as _
 >>> activate('pt')
 >>> _('The start of the range must not exceed the end of the range.')  #
 django/contrib/postgres/forms/ranges.py:20
 'The start of the range must not exceed the end of the range.'
 >>> _('This field cannot be null.')  #
 django/db/models/fields/__init__.py:105
 'Este campo não pode ser nulo.'
 }}}

--
Ticket URL: <https://code.djangoproject.com/ticket/30439#comment:9>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/063.fca305da9bd7939823ed66a13ba4a234%40djangoproject.com.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #30439: ngettext broken for certain locales due to catalog merging

Django
In reply to this post by Django
#30439: ngettext broken for certain locales due to catalog merging
-------------------------------------+-------------------------------------
     Reporter:  Michal Čihař         |                    Owner:  Claude
                                     |  Paroz
         Type:  Bug                  |                   Status:  assigned
    Component:                       |                  Version:  2.2
  Internationalization               |
     Severity:  Normal               |               Resolution:
     Keywords:                       |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by Claude Paroz):

 No, this is unrelated. You probably don't have `django.contrib.postgres`
 in your `INSTALLED_APPS`.

--
Ticket URL: <https://code.djangoproject.com/ticket/30439#comment:10>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/063.37a3c1832d5ffeefbb11f6721c403f9a%40djangoproject.com.