PythonInterpreter and SystemState

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

PythonInterpreter and SystemState

Anthony Roy-5
Hi all,

There is a bug in the tracker marked as "Works For Me" regarding the fact
that PythonInterpreters all share a common system state:
http://tinyurl.com/2vhqg8

Now the reported issue is that the reporter wanted to be able to run
different scripts in different interpreters set up in different ways. The
issue is marked as won't fix due to the ability to pass in a new
SystemState to the constructor of the PythonInterpreter.

This is fine for single threaded applications, but once you have multiple
threads, you have threads after the first overriding the global system
state with it's own one, and this has the effect that earlier thread's
global environments are replaced by the latter threads environment.

I was wondering if it was worth re-opening this issue, or whether it it
worth raising as a feature request as pedronis suggests in the bug report.

Cheers,

--
Anthony Roy

76 High Street
Scapegoat Hill
Golcar
Huddersfield
West Yorkshire
HD7 4NJ

t: 01484 655010
m: 07787 777815
e: [hidden email]





-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev
Reply | Threaded
Open this post in threaded view
|

Re: PythonInterpreter and SystemState

Alan Kennedy-7
[Anthony Roy]

> There is a bug in the tracker marked as "Works For Me" regarding the fact
> that PythonInterpreters all share a common system state:
> http://tinyurl.com/2vhqg8
>
> Now the reported issue is that the reporter wanted to be able to run
> different scripts in different interpreters set up in different ways. The
> issue is marked as won't fix due to the ability to pass in a new
> SystemState to the constructor of the PythonInterpreter.
>
> This is fine for single threaded applications, but once you have multiple
> threads, you have threads after the first overriding the global system
> state with it's own one, and this has the effect that earlier thread's
> global environments are replaced by the latter threads environment.

Are you certain? Why not create the second and later threads with

another_interp = new PythonInterpreter(null,new PySystemState())

as recommended by Samuele in the bug report?

Or to be more specific, what shared parts of the PySystemState class
are causing problems for your multi-threaded apps? Are you
experiencing specific problems?

> I was wondering if it was worth re-opening this issue, or whether it it
> worth raising as a feature request as pedronis suggests in the bug report.

If there are specific attributes of the PySystemState class that are
causing problems when used in a multi-threaded situation, best to
raise a separate bug report about them. Ideally the bug report should
include a java code snippet that illustrates the precise problem.

Regards,

Alan.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev
Reply | Threaded
Open this post in threaded view
|

Re: PythonInterpreter and SystemState

Anthony Roy-5
Hi all,

>> This is fine for single threaded applications, but once you have
>> multiple
>> threads, you have threads after the first overriding the global system
>> state with it's own one, and this has the effect that earlier thread's
>> global environments are replaced by the latter threads environment.
>
> Are you certain? Why not create the second and later threads with
> another_interp = new PythonInterpreter(null,new PySystemState())
> as recommended by Samuele in the bug report?

This makes no difference - if you look at the code in PythonInterpreter,
it's clear that doing that simply changes the static PySystemState
instance in the Py class - i.e. it replaces it globally.

> Or to be more specific, what shared parts of the PySystemState class
> are causing problems for your multi-threaded apps? Are you
> experiencing specific problems?

Actually, looking deeper there are quite a few issues that prevent Jython
being used embedded in a multi-threaded environment:

1) Global variables are kept in the system state - hence different scripts
with global variables of the same name can change the value of each
other's variables!

2) Initializing the PythonInterpreter is done through a static method -
hence you cannot supply different scripts with different sys.argv for
example.

I expect there are other issues which I haven't come across yet. I would
have thought that the point of being able to create separate
PythonInterpreters would be to provide a way of having separate sandboxes
for scripts to run in, with their own environments etc. For example, you
may have a GUI app (by nature multithreaded) and want to be able to script
different parts of it in isolated environments, so that, for example,
changing a script in the editor macro part of the program didn't interfere
with the image processing part. Otherwise the PythonInterpreter may just
as well be a singleton.

> If there are specific attributes of the PySystemState class that are
> causing problems when used in a multi-threaded situation, best to
> raise a separate bug report about them. Ideally the bug report should
> include a java code snippet that illustrates the precise problem.

I think it is the overall mechanism that needs looking at - I thought I'd
discuss it here before filing a bug. I could be missing something, but it
seems to me that an inability to work within a multi-threaded environment
is a Bad Thing, since all but the most trivial Java applications are
multi-threaded, and I would guess that one of the most common uses of
Jython is as an embedded scripting language for a Java app.

Attached is a minimal example of how the global environments of completely
different scripts interfere with each other. One Java class which sets
each script running in a different thread, and the three pythons scripts
which happen to have global variables in common. Run the java program with
a command line argument (say 'new-env') to see that setting the
PySystemState makes no difference in the multi-threaded environment.

--
Anthony Roy

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev

two.py (58 bytes) Download Attachment
three.py (56 bytes) Download Attachment
one.py (56 bytes) Download Attachment
JyTest.java (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: PythonInterpreter and SystemState

Alan Kennedy-7
[Alan]
>> Are you certain? Why not create the second and later threads with
>> another_interp = new PythonInterpreter(null,new PySystemState())
>> as recommended by Samuele in the bug report?

[Anthony]
> This makes no difference - if you look at the code in PythonInterpreter,
> it's clear that doing that simply changes the static PySystemState
> instance in the Py class - i.e. it replaces it globally.

I am looking at the code in PythonInterpreter.java.

What I see is that the systemState is kept as an instance variable of
PythonInterpreter when it is passed in, i.e. it is not shared with any
other PythonInterpreter.

Which conflicts directly with your statement "it's clear that doing
that simply changes the static PySystemState instance in the Py class
- i.e. it replaces it globally."

Are we looking at the same code? I'm looking at the source of jython
2.1, as included in the jython 2.1 download.

[Anthony]
> 1) Global variables are kept in the system state - hence different scripts
> with global variables of the same name can change the value of each
> other's variables!

Well, you can pass in a globals dictionary when creating the PythonInterpreter.

> 2) Initializing the PythonInterpreter is done through a static method -
> hence you cannot supply different scripts with different sys.argv for
> example.

This is correct. Also, you cannot supply different classloaders to
different interpreters.

[Anthony]
> Attached is a minimal example of how the global environments of completely
> different scripts interfere with each other. One Java class which sets
> each script running in a different thread, and the three pythons scripts
> which happen to have global variables in common. Run the java program with
> a command line argument (say 'new-env') to see that setting the
> PySystemState makes no difference in the multi-threaded environment.

And your example never fails for me, i.e. it never reproduces the
defective behaviour you describe.

Except that you neglected to put in the PythonInterpreter.initialize
call at the beginning of the program, so it crashes when creating new
interpreters. By adding this line to the start of your program

PythonInterpreter.initialize(System.getProperties(), null, new String[] {});

I get reliable, non-error, non-interfering behaviour every time.

Specifically, when running with threads (by supplying a command line
parameter), some variation of the following appears

New System State added.
New System State added.
New System State added.
Bernard
Melchet
Percy

Each of "Bernard", "Melchett" and "Percy" appears once and only once,
every time the code is run, without fail.

Alan.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev
Reply | Threaded
Open this post in threaded view
|

Re: PythonInterpreter and SystemState

Anthony Roy-5
...
> [Anthony]
>> This makes no difference - if you look at the code in
PythonInterpreter,
>> it's clear that doing that simply changes the static PySystemState
instance in the Py class - i.e. it replaces it globally.
>
> I am looking at the code in PythonInterpreter.java.
>
> What I see is that the systemState is kept as an instance variable of
PythonInterpreter when it is passed in, i.e. it is not shared with any
other PythonInterpreter.

I see now. I'd assumed that since setState() is called at the end of the
constructor, that this pushed the systemState to the static Py.systemState
variable. Which it does, but then all exec* methods also call setState()
which resets the Py.systemState() to the local systemState.

> Are we looking at the same code? I'm looking at the source of jython
2.1, as included in the jython 2.1 download.

No - I'm looking at the 2.2a release.

...
> [Anthony]
>> Attached is a minimal example of how the global environments of
completely
>> different scripts interfere with each other. One Java class which sets
each script running in a different thread, and the three pythons
scripts
>> which happen to have global variables in common. Run the java program
with
>> a command line argument (say 'new-env') to see that setting the
PySystemState makes no difference in the multi-threaded environment.
>
> And your example never fails for me, i.e. it never reproduces the
defective behaviour you describe.

So I guess that this is something in 2.2a.

> Except that you neglected to put in the PythonInterpreter.initialize
call at the beginning of the program, so it crashes when creating new
interpreters. By adding this line to the start of your program

I get no crashes because of this on 2.2a, or 2.2b1, so something has
changed since 2.1.

> I get reliable, non-error, non-interfering behaviour every time.
>
> Specifically, when running with threads (by supplying a command line
parameter), some variation of the following appears
>
> New System State added.
> New System State added.
> New System State added.
> Bernard
> Melchet
> Percy
>
> Each of "Bernard", "Melchett" and "Percy" appears once and only once,
every time the code is run, without fail.

I also get the correct results with 2.1

Here's the result of three separate runs of the program on 2.2a, and 2.2b1:

C:\0>C:\j2sdk1.4.2_13\bin\java.exe -classpath
.;C:\Java\jython_2_2a1\jython.jar
JyTest xxx
New System State added.
New System State added.
New System State added.
Melchet
Melchet
Melchet

C:\0>C:\j2sdk1.4.2_13\bin\java.exe -classpath
.;C:\Java\jython_2_2a1\jython.jar
JyTest xxx
New System State added.
New System State added.
New System State added.
Melchet
Melchet
Percy

C:\0>C:\j2sdk1.4.2_13\bin\java.exe -classpath
.;C:\Java\jython2.2b1\jython.jar J
yTest xxx
New System State added.
New System State added.
New System State added.
Traceback (innermost last):
  File "two.py", line 1, in ?
ImportError: no module named var
Traceback (innermost last):
  File "one.py", line 1, in ?
ImportError: no module named var
Percy

C:\0>C:\j2sdk1.4.2_13\bin\java.exe -classpath
.;C:\Java\jython2.2b1\jython.jar J
yTest xxx
New System State added.
New System State added.
New System State added.
Bernard
Bernard
Bernard

It looks then like a bug in 2.2a and later. I'll see if I can track down
the problem.

We seem to be getting somewhere here...

Cheers,

--
Anthony Roy





-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev
Reply | Threaded
Open this post in threaded view
|

Re: PythonInterpreter and SystemState

Alan Kennedy-7
[Anthony]
> It looks then like a bug in 2.2a and later. I'll see if I can track down
> the problem.

Yes, it definitely makes sense that it was a bug introduced in 2.2a0 or after.

[Anthony]
> We seem to be getting somewhere here...

Indeed we do.

Thanks so much for taking the time to actually write some code and
check the facts; well done.

Regards,

Alan.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev
Reply | Threaded
Open this post in threaded view
|

Re: PythonInterpreter and SystemState

Leo User
In reply to this post by Anthony Roy-5
This whole shared static state is a real pain.  Ive
gotten around it in the past with specialised
classloaders.  That works, but its a juggling act and
easily breakable.  Ill put this on the jythonx list of
things to explore.  Im not sure where this pattern
originally came from, maybe its modeled a little too
closely on what CPython does.

leouser


--- Anthony Roy <[hidden email]> wrote:

> Hi all,
>
> >> This is fine for single threaded applications,
> but once you have
> >> multiple
> >> threads, you have threads after the first
> overriding the global system
> >> state with it's own one, and this has the effect
> that earlier thread's
> >> global environments are replaced by the latter
> threads environment.
> >
> > Are you certain? Why not create the second and
> later threads with
> > another_interp = new PythonInterpreter(null,new
> PySystemState())
> > as recommended by Samuele in the bug report?
>
> This makes no difference - if you look at the code
> in PythonInterpreter,
> it's clear that doing that simply changes the static
> PySystemState
> instance in the Py class - i.e. it replaces it
> globally.
>
> > Or to be more specific, what shared parts of the
> PySystemState class
> > are causing problems for your multi-threaded apps?
> Are you
> > experiencing specific problems?
>
> Actually, looking deeper there are quite a few
> issues that prevent Jython
> being used embedded in a multi-threaded environment:
>
> 1) Global variables are kept in the system state -
> hence different scripts
> with global variables of the same name can change
> the value of each
> other's variables!
>
> 2) Initializing the PythonInterpreter is done
> through a static method -
> hence you cannot supply different scripts with
> different sys.argv for
> example.
>
> I expect there are other issues which I haven't come
> across yet. I would
> have thought that the point of being able to create
> separate
> PythonInterpreters would be to provide a way of
> having separate sandboxes
> for scripts to run in, with their own environments
> etc. For example, you
> may have a GUI app (by nature multithreaded) and
> want to be able to script
> different parts of it in isolated environments, so
> that, for example,
> changing a script in the editor macro part of the
> program didn't interfere
> with the image processing part. Otherwise the
> PythonInterpreter may just
> as well be a singleton.
>
> > If there are specific attributes of the
> PySystemState class that are
> > causing problems when used in a multi-threaded
> situation, best to
> > raise a separate bug report about them. Ideally
> the bug report should
> > include a java code snippet that illustrates the
> precise problem.
>
> I think it is the overall mechanism that needs
> looking at - I thought I'd
> discuss it here before filing a bug. I could be
> missing something, but it
> seems to me that an inability to work within a
> multi-threaded environment
> is a Bad Thing, since all but the most trivial Java
> applications are
> multi-threaded, and I would guess that one of the
> most common uses of
> Jython is as an embedded scripting language for a
> Java app.
>
> Attached is a minimal example of how the global
> environments of completely
> different scripts interfere with each other. One
> Java class which sets
> each script running in a different thread, and the
> three pythons scripts
> which happen to have global variables in common. Run
> the java program with
> a command line argument (say 'new-env') to see that
> setting the
> PySystemState makes no difference in the
> multi-threaded environment.
>
> --
> Anthony Roy
> > import time
> var = "Melchet"
> time.sleep(1)
> print var
> > import time
> var = "Percy"
> time.sleep(1)
> print var
> > import time
> var = "Bernard"
> time.sleep(1)
> print var>
-------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get
> the chance to share your
> opinions on IT & business topics through brief
> surveys-and earn cash
>
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV>
_______________________________________________
> Jython-dev mailing list
> [hidden email]
>
https://lists.sourceforge.net/lists/listinfo/jython-dev
>



 
____________________________________________________________________________________
The fish are biting.
Get more visitors on your site using Yahoo! Search Marketing.
http://searchmarketing.yahoo.com/arp/sponsoredsearch_v2.php

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev