[Django] #29427: RequestDataTooBig raised in request.py prevents Middleware from returning a valid response

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[Django] #29427: RequestDataTooBig raised in request.py prevents Middleware from returning a valid response

Django
#29427: RequestDataTooBig raised in request.py prevents Middleware from returning a
valid response
-----------------------------------------+------------------------
               Reporter:  S. Paquette    |          Owner:  nobody
                   Type:  Bug            |         Status:  new
              Component:  HTTP handling  |        Version:  1.11
               Severity:  Normal         |       Keywords:
           Triage Stage:  Unreviewed     |      Has patch:  0
    Needs documentation:  0              |    Needs tests:  0
Patch needs improvement:  0              |  Easy pickings:  0
                  UI/UX:  0              |
-----------------------------------------+------------------------
 This is effectively a request to re-open #28106, which was closed because
 the original author never replied to a request for more information.

 We need a way to return a response from a Middleware which is handling the
 RequestDataTooBig exception, but Middlewares intercepting this exception
 never generate a valid response. This seems due to how the exception
 causes self._body to never be created in request.py.

 In django.http.request, a check of the content length is done against
 settings.DATA_UPLOAD_MAX_MEMORY_SIZE at line 267.
 {{{
             # Limit the maximum request data size that will be handled in-
 memory.
             if (settings.DATA_UPLOAD_MAX_MEMORY_SIZE is not None and
                     int(self.META.get('CONTENT_LENGTH') or 0) >
 settings.DATA_UPLOAD_MAX_MEMORY_SIZE):
                 raise RequestDataTooBig('Request body exceeded
 settings.DATA_UPLOAD_MAX_MEMORY_SIZE.')

 }}}

 If the content length exceeds DATA_UPLOAD_MAX_MEMORY_SIZE, no request body
 is generated, and a RequestDataTooBig exception is raised.

 In order to detect this error and return a useful response to our users'
 browsers, we created a Middleware to catch the exception and supply an
 informative JsonResponse. However, despite the status setting correctly,
 the response itself was never being returned. Our Middleware:
 {{{
 from django.http import JsonResponse
 from django.core.exceptions import RequestDataTooBig


 class CheckSize(object):

     def __init__(self, get_response):
         self.get_response = get_response

     def __call__(self, request):

         try:
             body = request.body
         except RequestDataTooBig:
             return JsonResponse({"msg": "The file provided is too large.
 Please reduce its size and try again."}, status=400)

         response = self.get_response(request)
         return response
 }}}
 We tried placing the Middleware anywhere in the chain, and making it the
 only Middleware, but nothing worked.

 Per the author of #28106, we then added in an empty body to the request
 when the exception is raised, and that solved the problem:
 {{{
             # Limit the maximum request data size that will be handled in-
 memory.
             if (settings.DATA_UPLOAD_MAX_MEMORY_SIZE is not None and
                     int(self.META.get('CONTENT_LENGTH') or 0) >
 settings.DATA_UPLOAD_MAX_MEMORY_SIZE):
                 self._body = self.read(None)
                 raise RequestDataTooBig('Request body exceeded
 settings.DATA_UPLOAD_MAX_MEMORY_SIZE.')

 }}}

 After doing this, our response is returned. This can be reproduced on
 Django 1.11.

--
Ticket URL: <https://code.djangoproject.com/ticket/29427>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/053.39c05672e83fd1228c57234d24373b56%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #29427: RequestDataTooBig raised in request.py prevents Middleware from returning a valid response

Django
#29427: RequestDataTooBig raised in request.py prevents Middleware from returning a
valid response
-------------------------------+------------------------------------
     Reporter:  S. Paquette    |                    Owner:  nobody
         Type:  Bug            |                   Status:  new
    Component:  HTTP handling  |                  Version:  1.11
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Accepted
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+------------------------------------
Changes (by Claude Paroz):

 * stage:  Unreviewed => Accepted


Comment:

 Seems legitimate. Would you be able to write a failing test for the Django
 test suite?

--
Ticket URL: <https://code.djangoproject.com/ticket/29427#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/068.708c554d147865405c7a57b85fc8aa91%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #29427: RequestDataTooBig raised in request.py prevents Middleware from returning a valid response

Django
In reply to this post by Django
#29427: RequestDataTooBig raised in request.py prevents Middleware from returning a
valid response
-------------------------------+------------------------------------
     Reporter:  S. Paquette    |                    Owner:  oliver
         Type:  Bug            |                   Status:  assigned
    Component:  HTTP handling  |                  Version:  1.11
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Accepted
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+------------------------------------
Changes (by oliver):

 * owner:  nobody => oliver
 * status:  new => assigned


--
Ticket URL: <https://code.djangoproject.com/ticket/29427#comment:2>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/068.142c1f8398331f13545838ca027f1b71%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #29427: RequestDataTooBig raised in request.py prevents Middleware from returning a valid response

Django
In reply to this post by Django
#29427: RequestDataTooBig raised in request.py prevents Middleware from returning a
valid response
-------------------------------+------------------------------------
     Reporter:  S. Paquette    |                    Owner:  (none)
         Type:  Bug            |                   Status:  new
    Component:  HTTP handling  |                  Version:  1.11
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Accepted
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+------------------------------------
Changes (by oliver):

 * owner:  oliver => (none)
 * status:  assigned => new


--
Ticket URL: <https://code.djangoproject.com/ticket/29427#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/068.bf091937b0f8a0cd555d9d497ee95083%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #29427: RequestDataTooBig raised in request.py prevents Middleware from returning a valid response

Django
In reply to this post by Django
#29427: RequestDataTooBig raised in request.py prevents Middleware from returning a
valid response
-------------------------------+------------------------------------
     Reporter:  S. Paquette    |                    Owner:  nobody
         Type:  Bug            |                   Status:  new
    Component:  HTTP handling  |                  Version:  1.11
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Accepted
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+------------------------------------

Comment (by S. Paquette):

 Replying to [comment:1 Claude Paroz]:
 > Seems legitimate. Would you be able to write a failing test for the
 Django test suite?

 Sure thing--how's this look for django/tests/requests/tests.py?

 {{{
 from django.core.exceptions import RequestDataTooBig

 def test_req_body_exists_after_size_exceeded(self):
         """
         If a CONTENT_LENGTH > DATA_UPLOAD_MAX_MEMORY_SIZE is encountered,
 an empty
         _body attribute should still be generated in the request
         """
         with override_settings(DATA_UPLOAD_MAX_MEMORY_SIZE=12):
                 payload = FakePayload('a=1&a=2;a=3\r\n')
                 request = WSGIRequest({
                         'REQUEST_METHOD': 'POST',
                         'CONTENT_TYPE': 'application/x-www-form-
 urlencoded',
                         'CONTENT_LENGTH': len(payload),
                         'wsgi.input': payload,
                 })

                 with self.assertRaises(RequestDataTooBig):
                         request.body

                 self.assertTrue(hasattr(request,'_body'))
 }}}

--
Ticket URL: <https://code.djangoproject.com/ticket/29427#comment:2>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/068.6ac03c8b2600c5dd2fe679a1bd6d5b8f%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #29427: RequestDataTooBig raised in request.py prevents Middleware from returning a valid response

Django
In reply to this post by Django
#29427: RequestDataTooBig raised in request.py prevents Middleware from returning a
valid response
-------------------------------+------------------------------------
     Reporter:  S. Paquette    |                    Owner:  nobody
         Type:  Bug            |                   Status:  new
    Component:  HTTP handling  |                  Version:  1.11
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Accepted
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+------------------------------------
Changes (by Herbert Fortes):

 * cc: Herbert Fortes (added)


--
Ticket URL: <https://code.djangoproject.com/ticket/29427#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/068.cde96e60abaec3668ef41f0af75a3d59%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #29427: RequestDataTooBig raised in request.py prevents Middleware from returning a valid response

Django
In reply to this post by Django
#29427: RequestDataTooBig raised in request.py prevents Middleware from returning a
valid response
-------------------------------+------------------------------------
     Reporter:  S. Paquette    |                    Owner:  nobody
         Type:  Bug            |                   Status:  new
    Component:  HTTP handling  |                  Version:  1.11
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Accepted
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+------------------------------------

Comment (by Josh Schneier):

 Is there any reason that this can't be handled using an exception handling
 middleware?

 {{{#!python
 from django.http import JsonResponse
 from django.core.exceptions import RequestDataTooBig

 class HandleDataTooBigMiddleware:
     def __init__(self, get_response):
         self.get_response = get_response

     def __call__(self, request):
         return self.get_response(request)

     def process_exception(self, request, exception):
         if isinstance(exception, RequestDataTooBig):
             return JsonResponse({'info': 'File too big'}, status=400)
         return None
 }}}

--
Ticket URL: <https://code.djangoproject.com/ticket/29427#comment:4>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/068.be49e78c3b9d76e0549028c0d3ae8f72%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #29427: RequestDataTooBig raised in request.py prevents Middleware from returning a valid response

Django
In reply to this post by Django
#29427: RequestDataTooBig raised in request.py prevents Middleware from returning a
valid response
-------------------------------+------------------------------------
     Reporter:  S. Paquette    |                    Owner:  nobody
         Type:  Bug            |                   Status:  new
    Component:  HTTP handling  |                  Version:  1.11
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Accepted
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+------------------------------------

Comment (by Herbert Fortes):

 I liked [https://docs.djangoproject.com/en/2.0/topics/http/middleware
 /#process-exception process-exception] proposed by Josh Schneier. I did
 not test it though.

--
Ticket URL: <https://code.djangoproject.com/ticket/29427#comment:5>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/068.e36bb4f4858bab71c7142fd6dfdbd83c%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #29427: RequestDataTooBig raised in request.py prevents Middleware from returning a valid response

Django
In reply to this post by Django
#29427: RequestDataTooBig raised in request.py prevents Middleware from returning a
valid response
-------------------------------+------------------------------------
     Reporter:  S. Paquette    |                    Owner:  nobody
         Type:  Bug            |                   Status:  new
    Component:  HTTP handling  |                  Version:  1.11
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Accepted
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+------------------------------------

Comment (by Herbert Fortes):

 Replying to [comment:2 S. Paquette]:
 > Replying to [comment:1 Claude Paroz]:
 > > Seems legitimate. Would you be able to write a failing test for the
 Django test suite?
 >
 > Sure thing--how's this look for django/tests/requests/tests.py?

 As I see, only the PR is missing. Test and fix (self._body) are known.
 I ran the test.

--
Ticket URL: <https://code.djangoproject.com/ticket/29427#comment:6>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/068.3fa53c5f1b8f438a7845d470402770b6%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #29427: RequestDataTooBig raised in request.py prevents Middleware from returning a valid response

Django
In reply to this post by Django
#29427: RequestDataTooBig raised in request.py prevents Middleware from returning a
valid response
-------------------------------+------------------------------------
     Reporter:  S. Paquette    |                    Owner:  nobody
         Type:  Bug            |                   Status:  new
    Component:  HTTP handling  |                  Version:  1.11
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Accepted
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+------------------------------------

Comment (by S. Paquette):

 Replying to [comment:4 Josh Schneier]:
 > Is there any reason that this can't be handled using an exception
 handling middleware?
 >
 > {{{#!python
 > from django.http import JsonResponse
 > from django.core.exceptions import RequestDataTooBig
 >
 > class HandleDataTooBigMiddleware:
 >     def __init__(self, get_response):
 >         self.get_response = get_response
 >
 >     def __call__(self, request):
 >         return self.get_response(request)
 >
 >     def process_exception(self, request, exception):
 >         if isinstance(exception, RequestDataTooBig):
 >             return JsonResponse({'info': 'File too big'}, status=400)
 >         return None
 > }}}

 Per the author of #28106, this also doesn't work if the body isn't set; in
 fact that's how their Middleware is structured. (The one I pasted here is
 another option I tried when the exception handling Middleware wouldn't
 work.)

--
Ticket URL: <https://code.djangoproject.com/ticket/29427#comment:7>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/068.158e470e59b2734fc8f698debaddd22b%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Django] #29427: RequestDataTooBig raised in request.py prevents Middleware from returning a valid response

Django
In reply to this post by Django
#29427: RequestDataTooBig raised in request.py prevents Middleware from returning a
valid response
-------------------------------+------------------------------------
     Reporter:  S. Paquette    |                    Owner:  nobody
         Type:  Bug            |                   Status:  new
    Component:  HTTP handling  |                  Version:  1.11
     Severity:  Normal         |               Resolution:
     Keywords:                 |             Triage Stage:  Accepted
    Has patch:  0              |      Needs documentation:  0
  Needs tests:  0              |  Patch needs improvement:  0
Easy pickings:  0              |                    UI/UX:  0
-------------------------------+------------------------------------

Comment (by Herbert Fortes):

 S. Paquette,

 Can you do the PR? You did the test. And if the test needs
 adjustments about the style you can do it.

 If it is the first time you do a  PR to Django project, there is an
 official [https://docs.djangoproject.com/en/dev/intro/contributing/, how
 to write a patch].



 Regards

--
Ticket URL: <https://code.djangoproject.com/ticket/29427#comment:8>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-updates/068.a8f1fd52f1400d4581d6fdc502ace7d3%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.