ipdb throws AttributeError in autoreload.py with 2.2.2

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

ipdb throws AttributeError in autoreload.py with 2.2.2

Jason Johns
Hello,

About two weeks ago, work updated the base docker image to use django 2.2.2 and I noticed an issue right off the bat with using ipdb 0.12 and ipython 7.5.0 (which are also defaults for the docker image).  I really didn't have the time to dig more, and got along well enough with pdb in the meantime.  This weekend, I had some free time to poke around, and these are my findings:


In a DRF serializer helper method, I import ipdb and set a breakpoint.  The API call executes and stops here.  After anywhere from 1 to 3 seconds pause at the breakpoint, I then see the following output from ipdb> and the runserver process exits.
> /orm/service/api/serializers/helpers/facet_generator.py(62)__make_facet()
     61         import ipdb; ipdb.set_trace()
---> 62         log.info(f'Facet field: {facet_field}')
     63         log.info(f'Facet values: {facet_values}')

ipdb> Traceback (most recent call last):
  File "/orm/manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 60, in execute
    super().execute(*args, **options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 95, in handle
    self.run(**options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 102, in run
    autoreload.run_with_reloader(self.inner_run, **options)
  File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 585, in run_with_reloader
    start_django(reloader, main_func, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 570, in start_django
    reloader.run(django_main_thread)
  File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 288, in run
    self.run_loop()
  File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 294, in run_loop
    next(ticker)
  File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 334, in tick
    for filepath, mtime in self.snapshot_files():
  File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 350, in snapshot_files
    for file in self.watched_files():
  File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 249, in watched_files
    yield from iter_all_python_module_files()
  File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 103, in iter_all_python_module_files
    return iter_modules_and_files(modules, frozenset(_error_files))
  File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 120, in iter_modules_and_files
    sys_file_paths.append(module.__file__)
AttributeError: module '__main__' has no attribute '__file__'

If you suspect this is an IPython bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at [hidden email]

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    %config Application.verbose_crash=True

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/IPython/core/history.py", line 780, in writeout_cache
    self._writeout_input_cache(conn)
  File "/usr/local/lib/python3.7/site-packages/IPython/core/history.py", line 764, in _writeout_input_cache
    (self.session_number,)+line)
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 139743563077376 and this is thread id 139743865329408.



To be sure it wasn't just the project's fault, I checked out another team's project and got the same error.  This seems to be isolated to just ipdb.  Using pdb does not show the same errors

Then, I pulled the latest from django github and confirmed that it is probably related to the changes in iter_modules_and_files between 2.2.1 and 2.2.2.  Looking at ipdb's package structure, I think the problem is the usage of __main__.py, because it apparently does not have an attribute `__file__`.

To double-check this, I updated the conditional in that method to include more logging information:

if module.__name__ == '__main__':
# __main__ (usually manage.py) doesn't always have a __spec__ set.
# Handle this by falling back to using __file__, resolved below.
# See https://docs.python.org/reference/import.html#main-spec
try:
sys_file_paths.append(module.__file__)
except AttributeError:
logger.info(dir(module))
logger.info(module.__package__)
logger.error(f'Module {module} has no attribute __file__')
continue


with the corresponding output:

autoreload[INFO] ['In', 'Out', '_', '__', '___', '__builtin__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', '_dh', '_ih', '_oh', 'exit', 'get_ipython', 'quit']
None
autoreload[INFO] None
Module <module '__main__'> has no attribute __file__
autoreload[ERROR] Module <module '__main__'> has no attribute __file__


This was introduced in https://github.com/django/django/commit/b2790f74d4f38c8b297b7c1cef6875d2378f6fa6, and I know the autoreload has been an area of some frustration lately and as such I'm not sure of the best approach for handling this case.  In my local testing, catching the attribute error and continuing seem to work well, but I'm unfortunately ignorant of any side effects this would have.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/c6bc44c6-5f07-4ebe-9cb4-0ee46f6c895b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ipdb throws AttributeError in autoreload.py with 2.2.2

Tom Forbes


> , and I know the autoreload has been an area of some frustration lately and as such I'm not sure of the best approach for handling this case.

I do feel responsible for the frustration, sorry! My advice to anyone that hears a little voice in their head asking “how hard can building an autoloader be?”: ignore it. That being said, I think we’ve ironed out the worst of the issues, despite this one being pretty bad :/

On 23 Jun 2019, at 16:46, Jason Johns <[hidden email]> wrote:

Hello,

About two weeks ago, work updated the base docker image to use django 2.2.2 and I noticed an issue right off the bat with using ipdb 0.12 and ipython 7.5.0 (which are also defaults for the docker image).  I really didn't have the time to dig more, and got along well enough with pdb in the meantime.  This weekend, I had some free time to poke around, and these are my findings:


In a DRF serializer helper method, I import ipdb and set a breakpoint.  The API call executes and stops here.  After anywhere from 1 to 3 seconds pause at the breakpoint, I then see the following output from ipdb> and the runserver process exits.
> /orm/service/api/serializers/helpers/facet_generator.py(62)__make_facet()
     61         import ipdb; ipdb.set_trace()
---> 62         log.info(f'Facet field: {facet_field}')
     63         log.info(f'Facet values: {facet_values}')

ipdb> Traceback (most recent call last):
  File "/orm/manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 60, in execute
    super().execute(*args, **options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 95, in handle
    self.run(**options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 102, in run
    autoreload.run_with_reloader(self.inner_run, **options)
  File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 585, in run_with_reloader
    start_django(reloader, main_func, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 570, in start_django
    reloader.run(django_main_thread)
  File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 288, in run
    self.run_loop()
  File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 294, in run_loop
    next(ticker)
  File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 334, in tick
    for filepath, mtime in self.snapshot_files():
  File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 350, in snapshot_files
    for file in self.watched_files():
  File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 249, in watched_files
    yield from iter_all_python_module_files()
  File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 103, in iter_all_python_module_files
    return iter_modules_and_files(modules, frozenset(_error_files))
  File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 120, in iter_modules_and_files
    sys_file_paths.append(module.__file__)
AttributeError: module '__main__' has no attribute '__file__'

If you suspect this is an IPython bug, please report it at:
or send an email to the mailing list at [hidden email]

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    %config Application.verbose_crash=True

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/IPython/core/history.py", line 780, in writeout_cache
    self._writeout_input_cache(conn)
  File "/usr/local/lib/python3.7/site-packages/IPython/core/history.py", line 764, in _writeout_input_cache
    (self.session_number,)+line)
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 139743563077376 and this is thread id 139743865329408.



To be sure it wasn't just the project's fault, I checked out another team's project and got the same error.  This seems to be isolated to just ipdb.  Using pdb does not show the same errors

Then, I pulled the latest from django github and confirmed that it is probably related to the changes in iter_modules_and_files between 2.2.1 and 2.2.2.  Looking at ipdb's package structure, I think the problem is the usage of __main__.py, because it apparently does not have an attribute `__file__`.

To double-check this, I updated the conditional in that method to include more logging information:

if module.__name__ == '__main__':
# __main__ (usually manage.py) doesn't always have a __spec__ set.
# Handle this by falling back to using __file__, resolved below.
# See https://docs.python.org/reference/import.html#main-spec
try:
sys_file_paths.append(module.__file__)
except AttributeError:
logger.info(dir(module))
logger.info(module.__package__)
logger.error(f'Module {module} has no attribute __file__')
continue


with the corresponding output:

autoreload[INFO] ['In', 'Out', '_', '__', '___', '__builtin__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', '_dh', '_ih', '_oh', 'exit', 'get_ipython', 'quit']
None
autoreload[INFO] None
Module <module '__main__'> has no attribute __file__
autoreload[ERROR] Module <module '__main__'> has no attribute __file__


This was introduced in https://github.com/django/django/commit/b2790f74d4f38c8b297b7c1cef6875d2378f6fa6, and I know the autoreload has been an area of some frustration lately and as such I'm not sure of the best approach for handling this case.  In my local testing, catching the attribute error and continuing seem to work well, but I'm unfortunately ignorant of any side effects this would have.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/c6bc44c6-5f07-4ebe-9cb4-0ee46f6c895b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5A2655EA-0172-4E56-BCA9-9D5C9D3244EB%40tomforb.es.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ipdb throws AttributeError in autoreload.py with 2.2.2

Jason Johns
Wow, that was quick! Thank you!

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6b7125aa-404e-4678-8c19-578533cb52fa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.