Python › Jython › jython-users

Fwd: is PyCode thread-safe?

Classic

List

Threaded

8 messages Options

Li Shen

Fwd: is PyCode thread-safe?

Hi,

I'm pretty new to jython and I want to use jython to execute many existing python scripts.

For performance reason, I'm wondering if I can pre-compile the python scripts, like below:

private PyCode code = new PythonInterpreter().compile((new InputStreamReader(Thread.currentThread().getContextClassLoader().getResourceAsStream(SCRIPT_NAME))));

And then reuse the compiled code in a multi-thread environment, like below:

public void someMethod() {

PythonInterpreter interpreter = new PythonInterpreter();

interpreter.exec(code);

// ...

}

Looks like the doc doesn't mention if PyCode is thead-safe. Could anyone please give out some clarifications, or suggestions if this approach is incorrect.

Thanks!

------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users

Pierre Thibault-3

Re: Fwd: is PyCode thread-safe?

2011/11/3 Li Shen <[hidden email]>

Hi,

I'm pretty new to jython and I want to use jython to execute many existing python scripts.

For performance reason, I'm wondering if I can pre-compile the python scripts, like below:

private PyCode code = new PythonInterpreter().compile((new InputStreamReader(Thread.currentThread().getContextClassLoader().getResourceAsStream(SCRIPT_NAME))));

And then reuse the compiled code in a multi-thread environment, like below:

public void someMethod() {
PythonInterpreter interpreter = new PythonInterpreter();
interpreter.exec(code);
// ...
}

Looks like the doc doesn't mention if PyCode is thead-safe. Could anyone please give out some clarifications, or suggestions if this approach is incorrect.

Thanks!

I think so. Each PythonInterpreter is an independent state machine.

Pierre Thibault

Python Developer/Développeur Python
Montréal, QC

[hidden email]

Jeff Emanuel

Re: Fwd: is PyCode thread-safe?

The interpreters ought to be constructed with independent
namespaces and sys modules, otherwise they are shared among
interpreters. See the 2-arg constuctor.

http://www.jython.org/javadoc/org/python/util/PythonInterpreter.html#PythonInterpreter(org.python.core.PyObject,
org.python.core.PySystemState)

Pierre Thibault wrote:

> 2011/11/3 Li Shen <[hidden email] <mailto:[hidden email]>>
>
> Hi,
>
> I'm pretty new to jython and I want to use jython to execute many
> existing python scripts.
>
> For performance reason, I'm wondering if I can pre-compile the
> python scripts, like below:
>
> private PyCode code = new PythonInterpreter().compile((new
> InputStreamReader(Thread.currentThread().getContextClassLoader().getResourceAsStream(SCRIPT_NAME))));
>
> And then reuse the compiled code in a multi-thread environment,
> like below:
>
> public void someMethod() {
> PythonInterpreter interpreter = new PythonInterpreter();
> interpreter.exec(code);
> // ...
> }
>
> Looks like the doc doesn't mention if PyCode is thead-safe. Could
> anyone please give out some clarifications, or suggestions if this
> approach is incorrect.
>
> Thanks!
>
>
>
> I think so. Each PythonInterpreter is an independent state machine.
>
> Pierre Thibault
>
> Python Developer/Développeur Python
> Montréal, QC
> /[hidden email] <mailto:[hidden email]>/
>
>
>
>
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
> RSA(R) Conference 2012
> Save $700 by Nov 18
> Register now
> http://p.sf.net/sfu/rsa-sfdev2dev1
> ------------------------------------------------------------------------
>
> _______________________________________________
> Jython-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/jython-users
>

Li Shen

Re: Fwd: is PyCode thread-safe?

Thanks so much Pierre & Jeff!

Jeff, did you mean that I should do something like below to create a PythonInterpreter?

PythonInterpreter interpreter = new PythonInterpreter(new PyStringMap(), new PySystemState());

Thanks!

2011/11/4 Jeff Emanuel <[hidden email]>

The interpreters ought to be constructed with independent
namespaces and sys modules, otherwise they are shared among
interpreters. See the 2-arg constuctor.

http://www.jython.org/javadoc/org/python/util/PythonInterpreter.html#PythonInterpreter(org.python.core.PyObject,
org.python.core.PySystemState)

Pierre Thibault wrote:
> 2011/11/3 Li Shen <[hidden email] <mailto:[hidden email]>>

>
> Hi,
>
> I'm pretty new to jython and I want to use jython to execute many
> existing python scripts.
>
> For performance reason, I'm wondering if I can pre-compile the
> python scripts, like below:
>
> private PyCode code = new PythonInterpreter().compile((new
> InputStreamReader(Thread.currentThread().getContextClassLoader().getResourceAsStream(SCRIPT_NAME))));
>
> And then reuse the compiled code in a multi-thread environment,
> like below:
>
> public void someMethod() {
> PythonInterpreter interpreter = new PythonInterpreter();
> interpreter.exec(code);
> // ...
> }
>
> Looks like the doc doesn't mention if PyCode is thead-safe. Could
> anyone please give out some clarifications, or suggestions if this
> approach is incorrect.
>
> Thanks!
>
>
>
> I think so. Each PythonInterpreter is an independent state machine.
>
> Pierre Thibault
>

> Python Developer/Développeur Python
> Montréal, QC

> /[hidden email] <mailto:[hidden email]>/
>
>
>
>
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
> RSA(R) Conference 2012
> Save $700 by Nov 18
> Register now
> http://p.sf.net/sfu/rsa-sfdev2dev1
> ------------------------------------------------------------------------
>
> _______________________________________________
> Jython-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/jython-users
>

------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users

Pierre Thibault-3

Re: Fwd: is PyCode thread-safe?

2011/11/4 Li Shen <[hidden email]>

Thanks so much Pierre & Jeff!

Jeff, did you mean that I should do something like below to create a PythonInterpreter?

PythonInterpreter interpreter = new PythonInterpreter(new PyStringMap(), new PySystemState());

Thanks!

Li

Hello Li,

I don't think so. Your threads are probably sharing the same modules. The modules are shared across all the threads of the process by default in Python. Having separate modules for each thread will take a lot more memory. So use it only if needed.

Pierre Thibault

Python Developer/Développeur Python
Montréal, QC

[hidden email]

Li Shen

Re: Fwd: is PyCode thread-safe?

In reply to this post by Li Shen

Hi Pierre & Jeff,

In my case, I need to execute many different python scripts with jython. I may assume those scripts could do anything they want and that's out of my control. So to ensure thread safety, the easiest approach is to create totally independent interpreter as Jeff mentioned?

Memory consumption would be another concern to us, so we may have to use the non-arg and 2-args constructors of PythonInterpreter respectively depending on the behaviors of a python script to be interpreted? This approach would require python expertise to analyze each script.

Not sure if my understandings are correct.

Thanks so much for you help!

2011/11/4 Jeff Emanuel <[hidden email]>

Hello Li,

That's what I would do, but as Pierre points out, it depends on
how independent you want your threads to be. My preference would
be that when working with separate interpreter instances, they
ought to work completely independently. If you want your threads
to share an environment, then use the no-arg constructor.

Jeff

Li Shen wrote:

Thanks so much Pierre & Jeff!

Jeff, did you mean that I should do something like below to create a PythonInterpreter?

PythonInterpreter interpreter = new PythonInterpreter(new PyStringMap(), new PySystemState());

Thanks!

Li

2011/11/4 Jeff Emanuel <[hidden email] <mailto:[hidden email]>>

The interpreters ought to be constructed with independent
namespaces and sys modules, otherwise they are shared among
interpreters. See the 2-arg constuctor.

http://www.jython.org/javadoc/org/python/util/PythonInterpreter.html#PythonInterpreter(org.python.core.PyObject

<http://www.jython.org/javadoc/org/python/util/PythonInterpreter.html#PythonInterpreter%28org.python.core.PyObject>,

org.python.core.PySystemState)

Pierre Thibault wrote:
> 2011/11/3 Li Shen <[hidden email] <mailto:[hidden email]>
<mailto:[hidden email] <mailto:[hidden email]>>>

>
> Hi,
>
> I'm pretty new to jython and I want to use jython to execute
many
> existing python scripts.
>
> For performance reason, I'm wondering if I can pre-compile the
> python scripts, like below:
>
> private PyCode code = new PythonInterpreter().compile((new
> InputStreamReader(Thread.currentThread().getContextClassLoader().getResourceAsStream(SCRIPT_NAME))));
>
> And then reuse the compiled code in a multi-thread environment,
> like below:
>
> public void someMethod() {
> PythonInterpreter interpreter = new PythonInterpreter();
> interpreter.exec(code);
> // ...
> }
>
> Looks like the doc doesn't mention if PyCode is thead-safe.
Could
> anyone please give out some clarifications, or suggestions
if this
> approach is incorrect.
>
> Thanks!
>
>
>
> I think so. Each PythonInterpreter is an independent state machine.
>
> Pierre Thibault
>
> Python Developer/Développeur Python
> Montréal, QC
> /[hidden email] <mailto:[hidden email]>

<mailto:[hidden email]

<mailto:[hidden email]>>/
>
>
>
>
>
>
------------------------------------------------------------------------
>
>
------------------------------------------------------------------------------
> RSA(R) Conference 2012
> Save $700 by Nov 18
> Register now
> http://p.sf.net/sfu/rsa-sfdev2dev1
>
------------------------------------------------------------------------
>
> _______________________________________________
> Jython-users mailing list
> [hidden email]
<mailto:[hidden email]>

> https://lists.sourceforge.net/lists/listinfo/jython-users
>

------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Jython-users mailing list
[hidden email]
<mailto:[hidden email]>
https://lists.sourceforge.net/lists/listinfo/jython-users

------------------------------------------------------------------------

------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
------------------------------------------------------------------------

_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users

Pierre Thibault-3

Re: Fwd: is PyCode thread-safe?

2011/11/4 Li Shen <[hidden email]>

Hi Pierre & Jeff,

In my case, I need to execute many different python scripts with jython. I may assume those scripts could do anything they want and that's out of my control. So to ensure thread safety, the easiest approach is to create totally independent interpreter as Jeff mentioned?

Memory consumption would be another concern to us, so we may have to use the non-arg and 2-args constructors of PythonInterpreter respectively depending on the behaviors of a python script to be interpreted? This approach would require python expertise to analyze each script.

Not sure if my understandings are correct.

Thanks so much for you help!

Li

Hello Li,

I guess it would be great to experiment then. You may want to look at the Java console to know how much you are consuming resources. Maybe you can use external processes instead of threads. I believe you may reuse the interpreter, maybe having a pool, if you want to save resources. There is a risk of conflict but not that much and if you add a little cleanup after running a script it could be solution: (clear the global name space). But I don't know what it would be in practice.

Pierre Thibault

Python Developer/Développeur Python
Montréal, QC

[hidden email]

Li Shen

Re: Fwd: is PyCode thread-safe?

Hi Pierre,

Your suggestions sounds great and we can try them in practice.

Thanks!

Li

2011/11/5 Pierre Thibault <[hidden email]>

2011/11/4 Li Shen <[hidden email]>

Hi Pierre & Jeff,

In my case, I need to execute many different python scripts with jython. I may assume those scripts could do anything they want and that's out of my control. So to ensure thread safety, the easiest approach is to create totally independent interpreter as Jeff mentioned?

Memory consumption would be another concern to us, so we may have to use the non-arg and 2-args constructors of PythonInterpreter respectively depending on the behaviors of a python script to be interpreted? This approach would require python expertise to analyze each script.

Not sure if my understandings are correct.

Thanks so much for you help!

Li

Hello Li,

I guess it would be great to experiment then. You may want to look at the Java console to know how much you are consuming resources. Maybe you can use external processes instead of threads. I believe you may reuse the interpreter, maybe having a pool, if you want to save resources. There is a risk of conflict but not that much and if you add a little cleanup after running a script it could be solution: (clear the global name space). But I don't know what it would be in practice.

Pierre Thibault

Python Developer/Développeur Python
Montréal, QC
[hidden email]

------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users