Passing an object to a process

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Passing an object to a process

Tony Cappellini-2

On Windows XP - I've got a program running which has spawned a process using os.spawnv().

My program has a logger object instantiated that I want to pass to the process that was spawned.
I need to log errors seen by the process, to the same logfile that the program uses for logging.

I have the handle, thread ID, and process id of the process, but I see no way to share the logger object using these values.

How can I pass a logger instance to the process, from the program which spawned the process?

For legacy compatibility reasons, I'm stuck with an older version of Python that doesn't have the subprocess module and other features that are available with more modern releases of Python.


_______________________________________________
python-win32 mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-win32
Reply | Threaded
Open this post in threaded view
|

Re: Passing an object to a process

Preston Landers-2
Without getting into too philosophical of a discussion, the only sense
in which objects can truly move from one process to another is
recreating them in the other process.  Even fork() makes copies of
everything.

Have you tried pickle or other techniques of serialization? Not sure
offhand if the logger module supports pickle but it might.

You can always just create a new logger object using the same
parameters as the original (filename, etc), right? Or am I missing
something?

There IS a way to get filehandles to be shared by a child process.
But do you even need to do that if you can just recreate a new logger
object?

regards,
Preston


On Wed, Mar 14, 2012 at 1:53 PM, Tony Cappellini <[hidden email]> wrote:

>
> On Windows XP - I've got a program running which has spawned a process using
> os.spawnv().
>
> My program has a logger object instantiated that I want to pass to the
> process that was spawned.
> I need to log errors seen by the process, to the same logfile that the
> program uses for logging.
>
> I have the handle, thread ID, and process id of the process, but I see no
> way to share the logger object using these values.
>
> How can I pass a logger instance to the process, from the program which
> spawned the process?
>
> For legacy compatibility reasons, I'm stuck with an older version of Python
> that doesn't have the subprocess module and other features that are
> available with more modern releases of Python.
>
>
> _______________________________________________
> python-win32 mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/python-win32
>
_______________________________________________
python-win32 mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-win32
Reply | Threaded
Open this post in threaded view
|

Re: Passing an object to a process

Tim Roberts
In reply to this post by Tony Cappellini-2
Tony Cappellini wrote:
>
> On Windows XP - I've got a program running which has spawned a process
> using os.spawnv().
>
> My program has a logger object instantiated that I want to pass to the
> process that was spawned.

That's impossible.  Objects are just memory, and memory cannot be shared
between processes.


> I need to log errors seen by the process, to the same logfile that the
> program uses for logging.

You can certainly create a new logger instance in your spawned process
and have it write to the same file.  As long as the logging module
flushes after each write, that should work.


> For legacy compatibility reasons, I'm stuck with an older version of
> Python that doesn't have the subprocess module and other features that
> are available with more modern releases of Python.

Another alternative is to use popen instead of spawn, and have your
spawned process write its messages to stdout.  You could then echo them
on to the logfile in your original process.  That has its own buffering
problems.

--
Tim Roberts, [hidden email]
Providenza & Boekelheide, Inc.

_______________________________________________
python-win32 mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-win32
Reply | Threaded
Open this post in threaded view
|

Re: Passing an object to a process

Max Slimmer-3
In reply to this post by Tony Cappellini-2
depends what you mean by share logger object, you might be able to
have logger on both apps share a common log file.
OTH if you want realtime events, I pass a socket from one app to
another and have implemented a proxy dialog so one app writes to the
progress dialog proxy and it in turn sends data to a handler on the
other system via the socket where the actual progress dialog shows
progress.

max



On Wed, Mar 14, 2012 at 11:53 AM, Tony Cappellini <[hidden email]> wrote:

>
> On Windows XP - I've got a program running which has spawned a process using
> os.spawnv().
>
> My program has a logger object instantiated that I want to pass to the
> process that was spawned.
> I need to log errors seen by the process, to the same logfile that the
> program uses for logging.
>
> I have the handle, thread ID, and process id of the process, but I see no
> way to share the logger object using these values.
>
> How can I pass a logger instance to the process, from the program which
> spawned the process?
>
> For legacy compatibility reasons, I'm stuck with an older version of Python
> that doesn't have the subprocess module and other features that are
> available with more modern releases of Python.
>
>
> _______________________________________________
> python-win32 mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/python-win32
>
_______________________________________________
python-win32 mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-win32
Reply | Threaded
Open this post in threaded view
|

Re: Passing an object to a process

Tony Cappellini-2
In reply to this post by Preston Landers-2

>>in which objects can truly move from one process to another is
>>recreating them in the other process.  Even fork() makes copies of everything.
Recreating an object in another process means it's a different object, not a shared one.
 

>>Have you tried pickle or other techniques of serialization? Not sure
>>offhand if the logger module supports pickle but it might.
Yes. I've just tried this, even though I expected it not to work.

If process A pickles a logger object, and process B unpickles it,
referencing of an object in a different process is meaningless.

In my case, when the process attempted to write to the logger, no entries
were seen in the logfile. Surprisingly, no exceptions occurred- but this could just be a coincidence.

>>You can always just create a new logger object using the same
>>parameters as the original (filename, etc), right?
>>Or am I missing something?

That may work, and with less effort than my original idea.

But if two processes write to the logfile at the same time (especially on a multicore machine),
hard-to-read logfiles may result.
it's worth a try.

It would be great if the process could pass the info to be written to the logfile,
back to its creator and let the creator do all the writing.


_______________________________________________
python-win32 mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-win32
Reply | Threaded
Open this post in threaded view
|

Re: Passing an object to a process

Randy Syring-2
In reply to this post by Tony Cappellini-2
I've used the multiprocessing module to send data between processes:

http://docs.python.org/library/multiprocessing.html

But, its only in 2.6 and newer.  Since they are all separate processes, you could spawn a logging process and your worker processes as processes that use a newer python and then use multiprocessing.

Another option might be to send the log messages to a database and then have some process be responsible for pulling those messages out and writing to a file (if needed).

Just some thoughts.
---------------------------------------------
Randy Syring
Development & Executive Director
Level 12 Technologies (formerly Intelicom)
Direct: 502-276-0459
Office: 502-212-9913

Intelicom is now Level 12 Technologies, learn more about our name change.
Please update your address book with my new email address.

Principled People, Technology that Works

On 03/14/2012 02:53 PM, Tony Cappellini wrote:

On Windows XP - I've got a program running which has spawned a process using os.spawnv().

My program has a logger object instantiated that I want to pass to the process that was spawned.
I need to log errors seen by the process, to the same logfile that the program uses for logging.

I have the handle, thread ID, and process id of the process, but I see no way to share the logger object using these values.

How can I pass a logger instance to the process, from the program which spawned the process?

For legacy compatibility reasons, I'm stuck with an older version of Python that doesn't have the subprocess module and other features that are available with more modern releases of Python.



_______________________________________________
python-win32 mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-win32

_______________________________________________
python-win32 mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-win32
Reply | Threaded
Open this post in threaded view
|

Re: Passing an object to a process

Preston Landers-2
In reply to this post by Tony Cappellini-2
On Wed, Mar 14, 2012 at 2:55 PM, Tony Cappellini <[hidden email]> wrote:
>
>> >>in which objects can truly move from one process to another is
>> >>recreating them in the other process.  Even fork() makes copies of
>> >> everything.
>
> Recreating an object in another process means it's a different object, not a
> shared one.

Yeah, I know.  I was trying to make that point.  There's no real way
for the same object to exist in multiple processes other than SYSV
shared mem.

Truly shared memory (i.e, SYSV style) is tricky, not very portable,
and usually the wrong answer in my experience. As fas as I know stock
Python doesn't support that, and definitely never will on Windows.

The point is that you need to figure out what problem you're really
trying to solve (logging to one file from multiple processes, it
sounds like) and then find the best / simplest approach, which I can
tell you definitely doesn't involve SYSV shared memory.   It's
probably just creating separate logging objects in each process,
pointing to the same file, and protected by file locking if necessary.


>> >>Have you tried pickle or other techniques of serialization? Not sure
>> >>offhand if the logger module supports pickle but it might.
>
> Yes. I've just tried this, even though I expected it not to work.
>
> If process A pickles a logger object, and process B unpickles it,
> referencing of an object in a different process is meaningless.
>
> In my case, when the process attempted to write to the logger, no entries
> were seen in the logfile. Surprisingly, no exceptions occurred- but this
> could just be a coincidence.

Probably because the logger object, when serialized, saves a reference
to an open filehandle, which won't be automatically transfered to the
other process.

(There might ultimately be a way to make that work by inheriting
filehandles, but again, if you can find something simpler...)


>
> That may work, and with less effort than my original idea.
>
> But if two processes write to the logfile at the same time (especially on a
> multicore machine),
> hard-to-read logfiles may result.
> it's worth a try.

Yes, that type of thing can occur, but you can also get around that
with simple file locking.  By the way, that same problem certainly
exists even if you somehow shared the object between two processes -
what if the two processes made a log call at the same time?

File locking may introduce some performance issues if the logging is
very frequent, but usually you can find ways to mitigate that.

The main app I work on uses log files extensively, and the same file
is appended to by unrelated processes without a locking mechanism.
Occasionally you do see some interleaving of log entries but in my
experience it's fairly rare and in my particular case we don't care
much about that anyway.

regards,
Preston
_______________________________________________
python-win32 mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-win32
Reply | Threaded
Open this post in threaded view
|

Re: Passing an object to a process

Max Slimmer-3
If you are really just logging, use separate loggers/files, sync the
clocks, and timestamp records, then merge when you are ready to
process if you need timeline related results.


On Wed, Mar 14, 2012 at 1:29 PM, Preston Landers <[hidden email]> wrote:

> On Wed, Mar 14, 2012 at 2:55 PM, Tony Cappellini <[hidden email]> wrote:
>>
>>> >>in which objects can truly move from one process to another is
>>> >>recreating them in the other process.  Even fork() makes copies of
>>> >> everything.
>>
>> Recreating an object in another process means it's a different object, not a
>> shared one.
>
> Yeah, I know.  I was trying to make that point.  There's no real way
> for the same object to exist in multiple processes other than SYSV
> shared mem.
>
> Truly shared memory (i.e, SYSV style) is tricky, not very portable,
> and usually the wrong answer in my experience. As fas as I know stock
> Python doesn't support that, and definitely never will on Windows.
>
> The point is that you need to figure out what problem you're really
> trying to solve (logging to one file from multiple processes, it
> sounds like) and then find the best / simplest approach, which I can
> tell you definitely doesn't involve SYSV shared memory.   It's
> probably just creating separate logging objects in each process,
> pointing to the same file, and protected by file locking if necessary.
>
>
>>> >>Have you tried pickle or other techniques of serialization? Not sure
>>> >>offhand if the logger module supports pickle but it might.
>>
>> Yes. I've just tried this, even though I expected it not to work.
>>
>> If process A pickles a logger object, and process B unpickles it,
>> referencing of an object in a different process is meaningless.
>>
>> In my case, when the process attempted to write to the logger, no entries
>> were seen in the logfile. Surprisingly, no exceptions occurred- but this
>> could just be a coincidence.
>
> Probably because the logger object, when serialized, saves a reference
> to an open filehandle, which won't be automatically transfered to the
> other process.
>
> (There might ultimately be a way to make that work by inheriting
> filehandles, but again, if you can find something simpler...)
>
>
>>
>> That may work, and with less effort than my original idea.
>>
>> But if two processes write to the logfile at the same time (especially on a
>> multicore machine),
>> hard-to-read logfiles may result.
>> it's worth a try.
>
> Yes, that type of thing can occur, but you can also get around that
> with simple file locking.  By the way, that same problem certainly
> exists even if you somehow shared the object between two processes -
> what if the two processes made a log call at the same time?
>
> File locking may introduce some performance issues if the logging is
> very frequent, but usually you can find ways to mitigate that.
>
> The main app I work on uses log files extensively, and the same file
> is appended to by unrelated processes without a locking mechanism.
> Occasionally you do see some interleaving of log entries but in my
> experience it's fairly rare and in my particular case we don't care
> much about that anyway.
>
> regards,
> Preston
> _______________________________________________
> python-win32 mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/python-win32
_______________________________________________
python-win32 mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-win32
Reply | Threaded
Open this post in threaded view
|

Re: Passing an object to a process

Tony Cappellini-2
Thanks for all the advice.

The solution is simple, but neither elegant nor Pythonic. It's a kludge, but will suffice.

The process simply opens & writes a 1 line log file, then closes it. The process also increments a counter
each time the log file is written, and writes the counter value to the process log file.

The program checks to see if the file exists, reads it, then writes the process log file content to the program log file.
The program then deletes the process log file. (probably unnecessary, but prevents the possibility of the same line in the process log file
from being written to the program log file more than once.)

Performance isn't a concern, and this code won't be going to production.
It's just for debugging.

On Wed, Mar 14, 2012 at 1:49 PM, Max Slimmer <[hidden email]> wrote:
If you are really just logging, use separate loggers/files, sync the
clocks, and timestamp records, then merge when you are ready to
process if you need timeline related results.


On Wed, Mar 14, 2012 at 1:29 PM, Preston Landers <[hidden email]> wrote:
> On Wed, Mar 14, 2012 at 2:55 PM, Tony Cappellini <[hidden email]> wrote:
>>
>>> >>in which objects can truly move from one process to another is
>>> >>recreating them in the other process.  Even fork() makes copies of
>>> >> everything.
>>
>> Recreating an object in another process means it's a different object, not a
>> shared one.
>
> Yeah, I know.  I was trying to make that point.  There's no real way
> for the same object to exist in multiple processes other than SYSV
> shared mem.
>
> Truly shared memory (i.e, SYSV style) is tricky, not very portable,
> and usually the wrong answer in my experience. As fas as I know stock
> Python doesn't support that, and definitely never will on Windows.
>
> The point is that you need to figure out what problem you're really
> trying to solve (logging to one file from multiple processes, it
> sounds like) and then find the best / simplest approach, which I can
> tell you definitely doesn't involve SYSV shared memory.   It's
> probably just creating separate logging objects in each process,
> pointing to the same file, and protected by file locking if necessary.
>
>
>>> >>Have you tried pickle or other techniques of serialization? Not sure
>>> >>offhand if the logger module supports pickle but it might.
>>
>> Yes. I've just tried this, even though I expected it not to work.
>>
>> If process A pickles a logger object, and process B unpickles it,
>> referencing of an object in a different process is meaningless.
>>
>> In my case, when the process attempted to write to the logger, no entries
>> were seen in the logfile. Surprisingly, no exceptions occurred- but this
>> could just be a coincidence.
>
> Probably because the logger object, when serialized, saves a reference
> to an open filehandle, which won't be automatically transfered to the
> other process.
>
> (There might ultimately be a way to make that work by inheriting
> filehandles, but again, if you can find something simpler...)
>
>
>>
>> That may work, and with less effort than my original idea.
>>
>> But if two processes write to the logfile at the same time (especially on a
>> multicore machine),
>> hard-to-read logfiles may result.
>> it's worth a try.
>
> Yes, that type of thing can occur, but you can also get around that
> with simple file locking.  By the way, that same problem certainly
> exists even if you somehow shared the object between two processes -
> what if the two processes made a log call at the same time?
>
> File locking may introduce some performance issues if the logging is
> very frequent, but usually you can find ways to mitigate that.
>
> The main app I work on uses log files extensively, and the same file
> is appended to by unrelated processes without a locking mechanism.
> Occasionally you do see some interleaving of log entries but in my
> experience it's fairly rare and in my particular case we don't care
> much about that anyway.
>
> regards,
> Preston
> _______________________________________________
> python-win32 mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/python-win32


_______________________________________________
python-win32 mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-win32
Reply | Threaded
Open this post in threaded view
|

Re: Passing an object to a process

python@bdurham.com
In reply to this post by Tony Cappellini-2
What about logging to a database?
 
Malcolm

_______________________________________________
python-win32 mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-win32