use of subprocess module inside generator

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

use of subprocess module inside generator

Peter-309

I'm using Python 3.4.3 on Windows 7 (with latest patches) to develop a
sqlcmd module for accessing SQL Server (via Microsoft's sqlcmd.exe).  My
goal is to develop a 100% Python 3 module that's easy to use, flexible,
and by design shifts the majority of future SQL Server Python database
access maintenance to Microsoft.  So far, I've successfully and quickly
converted a few of my Python pyodbc applications/tools to use this new
module.  However, I recently ran into an apparent Python issue which
cost me a few hours to diagnose and work around.

I'm hoping that someone might know what the root cause of my issue was.
  Perhaps I've hit a bug/restriction with Python generators?

My original generator function looked like this:

def _raw_data(cl, stdout, *, opath=None, timeout=timeout):
     stdout = subprocess.check_output(cl, universal_newlines=True,
timeout=timeout)

     if opath is None:
         for line in stdout.splitlines():
             yield line.strip()
     else:
         with open(opath) as f:
             for line in f:
                 yield line.strip()

The above function appeared to work fine, if the command line directed
sqlcmd.exe to send its output to stdout.  However, if the command line
directed sqlcmd.exe to send its output to a file,
subprocess.check_output would never be called when next was called on
the returned generator.  I verified this behavior with print statements
inside my code, as well as, inside the subprocess module.

My work around was to simply move the call to subprocess.check_output
outside of the generator function (see below) to its caller (a
non-generator function).  With this minor change, everything appears to
work as expected.  OK, so am I missing something here?

def _raw_data(stdout, *, opath=None):
     if opath is None:
         for line in stdout.splitlines():
             yield line.strip()
     else:
         with open(opath) as f:
             for line in f:
                 yield line.strip()

Thank you in advance for your assistance.

Peter Santoro


Reply | Threaded
Open this post in threaded view
|

use of subprocess module inside generator

Gary Herron-2
On 05/13/2015 12:43 PM, Peter wrote:

>
> I'm using Python 3.4.3 on Windows 7 (with latest patches) to develop a
> sqlcmd module for accessing SQL Server (via Microsoft's sqlcmd.exe).  
> My goal is to develop a 100% Python 3 module that's easy to use,
> flexible, and by design shifts the majority of future SQL Server
> Python database access maintenance to Microsoft.  So far, I've
> successfully and quickly converted a few of my Python pyodbc
> applications/tools to use this new module.  However, I recently ran
> into an apparent Python issue which cost me a few hours to diagnose
> and work around.

I doubt that you've hit a bug -- we've all done this kind of thing many
times, and there's certainly no restriction on making procedure calls
within a generator -- so that's probably not the problem either.

My guess is that you've misinterpreted the failure of the original
code.  I don't know how that might be, but I do spot one oddity in your
original code which may be responsible.  See below ...

>
> I'm hoping that someone might know what the root cause of my issue
> was.  Perhaps I've hit a bug/restriction with Python generators?
>
> My original generator function looked like this:
>
> def _raw_data(cl, stdout, *, opath=None, timeout=timeout):
>     stdout = subprocess.check_output(cl, universal_newlines=True,
> timeout=timeout)

This seems muddled -- you pass in a parameter, stdout, only to
immediately overwrite its value with the output of check_output. What
was in stdout originally, and more importantly, did you expect the newly
assigned value from check_output to be returned to the calling
procedure?  If so, that's your bug, because parameters in function calls
don't work that way.

This makes sense with your workaround, since the assignment to stdout is
preserved when done outside the function.

I hope that helps.

Gary Herron



>
>     if opath is None:
>         for line in stdout.splitlines():
>             yield line.strip()
>     else:
>         with open(opath) as f:
>             for line in f:
>                 yield line.strip()
>
> The above function appeared to work fine, if the command line directed
> sqlcmd.exe to send its output to stdout.  However, if the command line
> directed sqlcmd.exe to send its output to a file,
> subprocess.check_output would never be called when next was called on
> the returned generator.  I verified this behavior with print
> statements inside my code, as well as, inside the subprocess module.
>
> My work around was to simply move the call to subprocess.check_output
> outside of the generator function (see below) to its caller (a
> non-generator function).  With this minor change, everything appears
> to work as expected.  OK, so am I missing something here?
>
> def _raw_data(stdout, *, opath=None):
>     if opath is None:
>         for line in stdout.splitlines():
>             yield line.strip()
>     else:
>         with open(opath) as f:
>             for line in f:
>                 yield line.strip()
>
> Thank you in advance for your assistance.
>
> Peter Santoro


--
Dr. Gary Herron
Department of Computer Science
DigiPen Institute of Technology
(425) 895-4418