[Django] #29403: TooBig error in memcached

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[Django] #29403: TooBig error in memcached

Django
#29403: TooBig error in memcached
-------------------------------------+-------------------------------------
               Reporter:             |          Owner:  nobody
  SKisContent                        |
                   Type:  Bug        |         Status:  new
              Component:  Core       |        Version:  1.11
  (Cache system)                     |
               Severity:  Normal     |       Keywords:  memcached, TooBig
           Triage Stage:             |      Has patch:  0
  Unreviewed                         |
    Needs documentation:  0          |    Needs tests:  0
Patch needs improvement:  0          |  Easy pickings:  0
                  UI/UX:  0          |
-------------------------------------+-------------------------------------
 I'm trying to use the memcached backend cache in a project, and I'm get a
 TooBig error on some requests. The error occurs in
 ''django/core/cache/backends/memcached.py'' on line 86:

 {{{
         val = self._cache.get(key)
         if val is None:
             return default
         return val
     def set(self, key, value, timeout=DEFAULT_TIMEOUT, version=None):
         key = self.make_key(key, version=version)
 **        if not self._cache.set(key, value,
 self.get_backend_timeout(timeout)):**
             # make sure the key doesn't keep its old value in case of
 failure to set (memcached's 1MB limit)
             self._cache.delete(key)
     def delete(self, key, version=None):
         key = self.make_key(key, version=version)
         self._cache.delete(key)
 }}}


 I can reproduce it using the Django shell:

 {{{
 $ ./manage.py shell
 Python 2.7.12 (default, Nov 19 2016, 06:48:10)
 [GCC 5.4.0 20160609] on linux2
 Type "help", "copyright", "credits" or "license" for more information.
 (InteractiveConsole)
 >>> from django.core.cache import cache
 >>> cache.get('foo')
 >>> cache.set('foo', 'bar')
 >>> cache.get('foo')
 'bar'
 >>> big = [i for i in range(1024*1024)]
 >>> cache.set('big_foo', big)
 Traceback (most recent call last):
   File "<console>", line 1, in <module>
   File "/local/lib/python2.7/site-
 packages/django/core/cache/backends/memcached.py", line 86, in set
     if not self._cache.set(key, value, self.get_backend_timeout(timeout)):
 TooBig

 }}}

 I can replicate the same behavior in pylibmc:

 {{{
 $ python
 Python 2.7.12 (default, Nov 19 2016, 06:48:10)
 [GCC 5.4.0 20160609] on linux2
 Type "help", "copyright", "credits" or "license" for more information.
 >>> from pylibmc.test import make_test_client
 >>> mc = make_test_client()
 >>> mc.get('foo')
 >>> mc.set('foo', 'bar')
 True
 >>> mc.get('foo')
 'bar'
 >>> big = [i for i in range(1024*1024)]
 >>> mc.set('big_foo', big)
 Traceback (most recent call last):
   File "<console>", line 1, in <module>
 TooBig
 }}}


 The TooBig exception was added to pylibmc in 2016:
 [https://github.com/lericson/pylibmc/commit/736e21276ad04d557b68bd81b7f28be1a4b4e1ec]

 The uncaught error breaks the application. Looking at the comment about
 the 1MB size limit on the next line (line 87), it seems like the original
 intent was to unset the value and fail silently. Therefore, the **if**
 block needs to be wrapped in **try...except**.

--
Ticket URL: <https://code.djangoproject.com/ticket/29403>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/054.4c3454721be80373eef750c11a0b0d7f%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #29403: TooBig error in memcached

Django
#29403: TooBig error in memcached
-------------------------------------+-------------------------------------
     Reporter:  SKisContent          |                    Owner:  nobody
         Type:  Bug                  |                   Status:  new
    Component:  Core (Cache system)  |                  Version:  1.11
     Severity:  Normal               |               Resolution:
     Keywords:  memcached, TooBig    |             Triage Stage:
                                     |  Unreviewed
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+-------------------------------------

Comment (by SKisContent):

 To clarify, I'm using **PyLibMCCache**:

 {{{
 CACHES = {
     'default': {
         'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
         'LOCATION': '127.0.0.1:11211',
         'TIMEOUT': 1000,
         'BINARY': True,
         'OPTIONS': {
             'tcp_nodelay': True,
             'remove_failed': 4
         }
     }
 }

 }}}

--
Ticket URL: <https://code.djangoproject.com/ticket/29403#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/069.8d9139c7131245a27e02413d738a108a%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #29403: Make PyLibMCCache backend handle TooBig exception from pylibmc (was: TooBig error in memcached)

Django
In reply to this post by Django
#29403: Make PyLibMCCache backend handle TooBig exception from pylibmc
-------------------------------------+------------------------------------
     Reporter:  SKisContent          |                    Owner:  nobody
         Type:  Bug                  |                   Status:  new
    Component:  Core (Cache system)  |                  Version:  1.11
     Severity:  Normal               |               Resolution:
     Keywords:  memcached, TooBig    |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+------------------------------------
Changes (by Tim Graham):

 * stage:  Unreviewed => Accepted


Comment:

 There's [https://github.com/django/django/pull/9811 a related pull
 request].

--
Ticket URL: <https://code.djangoproject.com/ticket/29403#comment:2>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/069.31b8d66214264a03792896c12757488a%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #29403: Make PyLibMCCache backend handle TooBig exception from pylibmc

Django
In reply to this post by Django
#29403: Make PyLibMCCache backend handle TooBig exception from pylibmc
-------------------------------------+------------------------------------
     Reporter:  SKisContent          |                    Owner:  nobody
         Type:  Bug                  |                   Status:  new
    Component:  Core (Cache system)  |                  Version:  1.11
     Severity:  Normal               |               Resolution:
     Keywords:  memcached, TooBig    |             Triage Stage:  Accepted
    Has patch:  0                    |      Needs documentation:  0
  Needs tests:  0                    |  Patch needs improvement:  0
Easy pickings:  0                    |                    UI/UX:  0
-------------------------------------+------------------------------------

Comment (by Flavio Curella):

 I'm not convinced we should fail silently. I understand that some versions
 o memcached will, but I'd rather be warn the user that something went
 wrong instead of being consistent.

 If we know something went wrong we should tell the user. I'd recommend
 raising a more descriptive exception, possibly using `raise ... from ...`.

--
Ticket URL: <https://code.djangoproject.com/ticket/29403#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/069.35ff19f5cebfd8609b974b2027a75b12%40djangoproject.com.