Have existing python script file that I want to execute from within a cherrypy web site/page. How can this be done 100% native to cherrypy?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Have existing python script file that I want to execute from within a cherrypy web site/page. How can this be done 100% native to cherrypy?

jibunnokage
Have existing python script file that I want to execute from within a cherrypy web site/page.  How can this be done 100% native to cherrypy?  Many examples reference java or CGI mods, that is not acceptable nor should it be IMHO.  There has to be a 100% cherrypi method to do this, no?

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Have existing python script file that I want to execute from within a cherrypy web site/page. How can this be done 100% native to cherrypy?

Sviatoslav Sydorenko-2
Just import function from that script and run it in request handler method, it is not specific to cherrypy, it works like anything else in python.

--Sviatoslav.

Sent from my phone, please pardon any typos.

On Sat, Apr 28, 2018, 23:04 <[hidden email]> wrote:
Have existing python script file that I want to execute from within a cherrypy web site/page.  How can this be done 100% native to cherrypy?  Many examples reference java or CGI mods, that is not acceptable nor should it be IMHO.  There has to be a 100% cherrypi method to do this, no?

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Have existing python script file that I want to execute from within a cherrypy web site/page. How can this be done 100% native to cherrypy?

jibunnokage
In reply to this post by jibunnokage
Yeah, I got a variant of it working.  Not sure it is pretty, but works.  We have a lot of existing python scripts we want to continue to use from console (Linux).  We just wanted to wrapper these with some simple web pages, to make them a bit easier to use.  So, I just imported the os module, can execute the script.  This is all internal stuff, nothing that has to be production hardened or such.  This lets the guys writing the python scripts for console do their thing, and lets me provide another avenue for execution.

I plan to swing back around on this one, and do more work, but using os module for now works.  I did notice one odd thing, a couple of our python classes/functions that work great from console, break when I import them to a CherryPy script, and then execute as you note.  One error is that bit shifting with the '<<' operator gripes about how it is used.  Only get that error, when the given function/routine is imported to CherryPy environment.  From console (python 2), no error at all.


On Saturday, April 28, 2018 at 1:04:56 PM UTC-7, [hidden email] wrote:
Have existing python script file that I want to execute from within a cherrypy web site/page.  How can this be done 100% native to cherrypy?  Many examples reference java or CGI mods, that is not acceptable nor should it be IMHO.  There has to be a 100% cherrypi method to do this, no?

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Have existing python script file that I want to execute from within a cherrypy web site/page. How can this be done 100% native to cherrypy?

Sviatoslav Sydorenko-2
1) don't use os module to spawn subprocesses, just have a function in those modules, import it and call it then.
2) you probably wrote scripts in a manner/syntax, which is incompatible with Python 3 and run CherryPy under Python 3. Or maybe just a way you run scripts is different when you spawn subprocesses. Anyway, I'd recommend you to not use subprocesses and do normal direct imports.
To mitigate cross python compatibility problems I strongly recommend you writing scripts using syntax, which works in both or completely switch to Python 3.
Look for a package distribution called "six", it may provide you some helpers for cross compat.

--Sviatoslav.

Sent from my phone, please pardon any typos.


On Sun, Apr 29, 2018, 06:12 <[hidden email]> wrote:
Yeah, I got a variant of it working.  Not sure it is pretty, but works.  We have a lot of existing python scripts we want to continue to use from console (Linux).  We just wanted to wrapper these with some simple web pages, to make them a bit easier to use.  So, I just imported the os module, can execute the script.  This is all internal stuff, nothing that has to be production hardened or such.  This lets the guys writing the python scripts for console do their thing, and lets me provide another avenue for execution.

I plan to swing back around on this one, and do more work, but using os module for now works.  I did notice one odd thing, a couple of our python classes/functions that work great from console, break when I import them to a CherryPy script, and then execute as you note.  One error is that bit shifting with the '<<' operator gripes about how it is used.  Only get that error, when the given function/routine is imported to CherryPy environment.  From console (python 2), no error at all.


On Saturday, April 28, 2018 at 1:04:56 PM UTC-7, [hidden email] wrote:
Have existing python script file that I want to execute from within a cherrypy web site/page.  How can this be done 100% native to cherrypy?  Many examples reference java or CGI mods, that is not acceptable nor should it be IMHO.  There has to be a 100% cherrypi method to do this, no?

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Have existing python script file that I want to execute from within a cherrypy web site/page. How can this be done 100% native to cherrypy?

Sviatoslav Sydorenko-2
I'm not talking about virtualization at all. I only mentioned https://virtualenv.pypa.io/en/stable/
There's also Pipenv which exposes a bit more full feature set https://packaging.python.org/tutorials/managing-dependencies/

Besides this, I cannot help you with your problem, because this would require you posting a real code example with a reproducible issue.

2018-05-05 2:57 GMT+02:00 Jibun no Kage <[hidden email]>:
I am under the impression you did not see my other note, showing an an example of the odd scenario I found, where the non-CherryPy environment software 'stack' accepts an enum based on a simple class definition, but the CherryPy inclusive environment software 'stack' does not.  Believe CherryPy is exposing something odd, rather than creating something odd, as I noted before.

Yes, virtualization is a best practice, and where and when we can use it, it is, but not all micro-controller environments can be emulated, for example BCM processors are very hard to emulate under Qemu.  I have design and supported virtual labs and large scale production environments for over 16 years, just about every variant of VMware, KVM, and even Hyper-V (and Microsoft Virtual Server before Hyper-V).  Moreover, running applications in slices, in such as things like docker (even LXC) is a best practice when possible, application virtualization versus OS virtualization.  But micro-controllers are often so specialized and narrow in resource scope, that is not an option.  The interesting scenario is when ARM (as now ATOM) based systems continue to scale up, from 32bit to 64bit, greater than 1GB RAM, etc., more CPU cores, etc.  Then, realistic partitioning of micro-controllers will be possible.  Looking forward to it.

Side note, much of VMware's ESXi server solution, under the hood/behind the scenes, leverages quite a bit of python in its control (console) environment.  My first experience with python came from its internal use in ESXi.

Schorschi


On 05/04/2018 15:16, Sviatoslav Sydorenko wrote:
First of all, it's not CherryPy picking interpreter, but vice versa - you run CherryPy app under a certain interpreter (better explicitly chosen).
Also, I advise you to check sys.executable to learn which binary exactly you use.
Finally, it is best practice to use a virtualenv to isolate environments for different apps you run.

--Sviatoslav.

Sent from my phone, please pardon any typos.

On Fri, May 4, 2018, 21:45 Jibun no Kage <[hidden email]> wrote:
Ah... Yes, I know how shebang works.  But, I had not worried about it, since the expectation was that everything would be completely python 2.  I went snooping... all the 'existing' child scripts, are 'banged' explicitly to Python2.  However, one of my test scripts, I was using/learning CherryPy was not explicitly qualified.  So I added a bit of code to the CherryPy script and left it NOT banged on purpose to see what CherryPy was defaulting to:

>>> Sys Get Version 2
>>> Platform Get Version 2.7.13

So CherryPy defaults to system default, and it is version 2, the child scripts are directed to version 2 via shebang.  All the modules used by the child scripts were developed in a version 2 context.  Moreover, I modified one of the child scripts to report what version bash was defaulting to when invoked by bash, a sanity check to validate, and it also reported version 2.

Child script direct invocation...
[2018-05-04 19:27:13 UTC]  >>> Platform Get Version 2.7.13
[2018-05-04 19:27:13 UTC]  >>> Sys Get Version 2

Child script indirect invocation via the os shell/bash trick...
[2018-05-04 19:29:43 UTC]  >>> Platform Get Version 2.7.13
[2018-05-04 19:29:43 UTC]  >>> Sys Get Version 2

So the OS default is 2, the scripts with os module direction are version 2,  the CherryPy defaults to 2, the subprocess defaults to 2.

So there is still something different between module load into CherryPy and invoking it, and module load into child script and then invoking it, the child script invoking always works, the module load to CherryPy then invoking, never works, the modules that exhibit the issue.  I will find it, in due course, because like we agreed, really want to import the modules and just use them directly, and avoid the os shell/bash trick.  At this point it has to be something really odd, tripping it.  Likely, it is some funky code in the module that is not very pythontic as it should be, since we do a lot of micro-controller/hardware driving python coding.

Not to confuse things, but I already found one issue and resolved it, in our development lab.  It had to do with different enum modules clashing, I discovered that 'enum34' was conflicting with 'enum' module.  On one specific system, the modules installed was incorrect, got to love when (many) developers touch stuff... oh... wait, I am one of the developers.  Ouch.

And, thanks for the comments and insight, appreciate it.

Schorschi



On 05/03/2018 22:45, Sviatoslav Sydorenko wrote:
1) You run CherryPy correctly, but it doesn't matter, because you obviously run the scripts themselves in a way that they might choose different Python env or even version. Do you know about shebang, for example? Anyway, you didn't provide sufficient details to know what happens exactly.
2) Running scripts is not CherryPy specific, it's just Python. There's no connection of your problem with CherryPy, this is not something which has to be documented.
3) My guess is that you just don't know how os.subprocess or bash executable lookup works. However I cannot be sure, because you did not show any reproducible examples of code you think you have problems with.

--Sviatoslav.

Sent from my phone, please pardon any typos.

On Fri, May 4, 2018, 03:28 Jibun no Kage <[hidden email]> wrote:
That could be, but how would anyone know?  When I run python <script name>.py (which is a CherryPy) script, I am executing the default python for the environment, which happens to be 2.7.x.  The CherryPy examples to not qualify doing anything different.  For example...

import cherrypy

class
HelloWorld(object):

    @cherrypy.expose

    def
index(self):
        return
"Hello world!"

if
__name__ == '__main__':
    cherrypy.quickstart(HelloWorld())

Run as "python <script name>.py"  This is taken straight from the online CherryPy official documentation.  So, I should be 100% python version consistent, regardless of the invocation.  Of it was otherwise, then CherryPy documentation should be explicit on this point, no?  None of the modules I am calling other than CherryPy as imported python 3 specific.  Well, less the pending backports, when I get to that point.

Schorschi

On 05/02/2018 14:53, Sviatoslav Sydorenko wrote:
It is supported, but you are doing it wrong. So it looks to me that you are spawning python3 interpreter from within python2 process.

2018-05-02 23:36 GMT+02:00 Jibun no Kage <[hidden email]>:
Thanks for the suggestions.  Unfortunately, at this time I can't address Python 3.  Python 2 only environment at the moment.  Odd, CherryPy documentation states that Python 2.7 is supported, but not surprised something tripped up.  Long term, as we revise the classes, migrate off of Python 2, expect to import and use them directly, as you suggest.

Schorschi


On 04/28/2018 23:21, Sviatoslav Sydorenko wrote:
1) don't use os module to spawn subprocesses, just have a function in those modules, import it and call it then.
2) you probably wrote scripts in a manner/syntax, which is incompatible with Python 3 and run CherryPy under Python 3. Or maybe just a way you run scripts is different when you spawn subprocesses. Anyway, I'd recommend you to not use subprocesses and do normal direct imports.
To mitigate cross python compatibility problems I strongly recommend you writing scripts using syntax, which works in both or completely switch to Python 3.
Look for a package distribution called "six", it may provide you some helpers for cross compat.

--Sviatoslav.

Sent from my phone, please pardon any typos.


On Sun, Apr 29, 2018, 06:12 <[hidden email]> wrote:
Yeah, I got a variant of it working.  Not sure it is pretty, but works.  We have a lot of existing python scripts we want to continue to use from console (Linux).  We just wanted to wrapper these with some simple web pages, to make them a bit easier to use.  So, I just imported the os module, can execute the script.  This is all internal stuff, nothing that has to be production hardened or such.  This lets the guys writing the python scripts for console do their thing, and lets me provide another avenue for execution.

I plan to swing back around on this one, and do more work, but using os module for now works.  I did notice one odd thing, a couple of our python classes/functions that work great from console, break when I import them to a CherryPy script, and then execute as you note.  One error is that bit shifting with the '<<' operator gripes about how it is used.  Only get that error, when the given function/routine is imported to CherryPy environment.  From console (python 2), no error at all.


On Saturday, April 28, 2018 at 1:04:56 PM UTC-7, [hidden email] wrote:
Have existing python script file that I want to execute from within a cherrypy web site/page.  How can this be done 100% native to cherrypy?  Many examples reference java or CGI mods, that is not acceptable nor should it be IMHO.  There has to be a 100% cherrypi method to do this, no?
--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.




--
Kind regards,

Sviatoslav Sydorenko
cell #: +380978963757






--
Kind regards,

Sviatoslav Sydorenko
cell #: +380978963757

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Have existing python script file that I want to execute from within a cherrypy web site/page. How can this be done 100% native to cherrypy?

Sviatoslav Sydorenko-2


2018-05-06 0:48 GMT+02:00 Jibun no Kage <[hidden email]>:
Not to debate, from my perspective, you are talking about virtualization, about application virtualization.  Per the link (https://virtualenv.pypa.io/en/stable/) you sent me, "In all these cases, virtualenv can help you. It creates an environment that has its own installation directories, that doesn’t share libraries with other virtualenv environments (and optionally doesn’t access the globally installed libraries either)."  That is 'application' virtualization from my perspective.  :)

Well, my assumption might be wrong, but I have a feeling that you might not realize how virtualenv works. While it contains word "virtual" it's still not a typical virtualization, nor isolation. Yes, it creates a separate "env" folder with binaries linked against some other (system) installation of python and site-packages, which won't collide with that system install of python. So you add that thing to $PATH or invoke directly.
My point is that when running app and subprocesses you may unintentionally switch versions when being not careful (for example, when you reset/erase env vars available for subshell). And since I don't know anything about how you run web app and scripts it is hard for me to point where to look closer.
So logging executable path and env vars would help you figure out more details about this.
 

As for code examples, I can't publish the explicit code, if I could, I would have.  I realize this limits your assistance, just the same, thanks for the insight, and suggestions, you have helped, I appreciate it.

You don't have to expose any of your super secret business logic or so. The rule of thumb is to post minimal amount of similar code, which might contain dummy "hello world", but still show how exactly to spawn subprocesses, which from your words looks like:

cp_app.py:
import os
import cherrypy

class RootWebApp:
    @cherrypy.expose
    def index(self):
        exec_result = os.subprocess.run(['/path/to/specific/py/env/bin/python', 'some_script.py'])  # might be different and in fact has lots of arguments to play with.
        return exec_result.stdout

    @classmethod
    def run(cls):
        cherrypy.quickstart(cls(), '', {})


def main():
    RootWebApp.run()


__name__ == '__main__' and main()

some_script.py:
def main():
    print('Hello world!')


__name__ == '__main__' and main()

As you can see, your example doesn't have to include any logic specific to your project, but to your env and the way your components interact with each other. This would help with troubleshooting.


Since your domain is UA... you are in Ukraine?  I spent most of my summers as a kid in Germany, even got to visit Hungry, Poland and Yugoslavia (when it still existed), even East Germany once, but I never got to the Ukraine.  Maybe when I retire and have time, I can get back to Eastern Europe again.

I used to live in Ukraine for my whole life, but 3 months ago I moved to Brno, Czech Republic for Red Hat :)
 


Schorschi


On 05/05/2018 02:28, Sviatoslav Sydorenko wrote:
I'm not talking about virtualization at all. I only mentioned https://virtualenv.pypa.io/en/stable/
There's also Pipenv which exposes a bit more full feature set https://packaging.python.org/tutorials/managing-dependencies/

Besides this, I cannot help you with your problem, because this would require you posting a real code example with a reproducible issue.

2018-05-05 2:57 GMT+02:00 Jibun no Kage <[hidden email]>:
I am under the impression you did not see my other note, showing an an example of the odd scenario I found, where the non-CherryPy environment software 'stack' accepts an enum based on a simple class definition, but the CherryPy inclusive environment software 'stack' does not.  Believe CherryPy is exposing something odd, rather than creating something odd, as I noted before.

Yes, virtualization is a best practice, and where and when we can use it, it is, but not all micro-controller environments can be emulated, for example BCM processors are very hard to emulate under Qemu.  I have design and supported virtual labs and large scale production environments for over 16 years, just about every variant of VMware, KVM, and even Hyper-V (and Microsoft Virtual Server before Hyper-V).  Moreover, running applications in slices, in such as things like docker (even LXC) is a best practice when possible, application virtualization versus OS virtualization.  But micro-controllers are often so specialized and narrow in resource scope, that is not an option.  The interesting scenario is when ARM (as now ATOM) based systems continue to scale up, from 32bit to 64bit, greater than 1GB RAM, etc., more CPU cores, etc.  Then, realistic partitioning of micro-controllers will be possible.  Looking forward to it.

Side note, much of VMware's ESXi server solution, under the hood/behind the scenes, leverages quite a bit of python in its control (console) environment.  My first experience with python came from its internal use in ESXi.

Schorschi


On 05/04/2018 15:16, Sviatoslav Sydorenko wrote:
First of all, it's not CherryPy picking interpreter, but vice versa - you run CherryPy app under a certain interpreter (better explicitly chosen).
Also, I advise you to check sys.executable to learn which binary exactly you use.
Finally, it is best practice to use a virtualenv to isolate environments for different apps you run.

--Sviatoslav.

Sent from my phone, please pardon any typos.

On Fri, May 4, 2018, 21:45 Jibun no Kage <[hidden email]> wrote:
Ah... Yes, I know how shebang works.  But, I had not worried about it, since the expectation was that everything would be completely python 2.  I went snooping... all the 'existing' child scripts, are 'banged' explicitly to Python2.  However, one of my test scripts, I was using/learning CherryPy was not explicitly qualified.  So I added a bit of code to the CherryPy script and left it NOT banged on purpose to see what CherryPy was defaulting to:

>>> Sys Get Version 2
>>> Platform Get Version 2.7.13

So CherryPy defaults to system default, and it is version 2, the child scripts are directed to version 2 via shebang.  All the modules used by the child scripts were developed in a version 2 context.  Moreover, I modified one of the child scripts to report what version bash was defaulting to when invoked by bash, a sanity check to validate, and it also reported version 2.

Child script direct invocation...
[2018-05-04 19:27:13 UTC]  >>> Platform Get Version 2.7.13
[2018-05-04 19:27:13 UTC]  >>> Sys Get Version 2

Child script indirect invocation via the os shell/bash trick...
[2018-05-04 19:29:43 UTC]  >>> Platform Get Version 2.7.13
[2018-05-04 19:29:43 UTC]  >>> Sys Get Version 2

So the OS default is 2, the scripts with os module direction are version 2,  the CherryPy defaults to 2, the subprocess defaults to 2.

So there is still something different between module load into CherryPy and invoking it, and module load into child script and then invoking it, the child script invoking always works, the module load to CherryPy then invoking, never works, the modules that exhibit the issue.  I will find it, in due course, because like we agreed, really want to import the modules and just use them directly, and avoid the os shell/bash trick.  At this point it has to be something really odd, tripping it.  Likely, it is some funky code in the module that is not very pythontic as it should be, since we do a lot of micro-controller/hardware driving python coding.

Not to confuse things, but I already found one issue and resolved it, in our development lab.  It had to do with different enum modules clashing, I discovered that 'enum34' was conflicting with 'enum' module.  On one specific system, the modules installed was incorrect, got to love when (many) developers touch stuff... oh... wait, I am one of the developers.  Ouch.

And, thanks for the comments and insight, appreciate it.

Schorschi



On 05/03/2018 22:45, Sviatoslav Sydorenko wrote:
1) You run CherryPy correctly, but it doesn't matter, because you obviously run the scripts themselves in a way that they might choose different Python env or even version. Do you know about shebang, for example? Anyway, you didn't provide sufficient details to know what happens exactly.
2) Running scripts is not CherryPy specific, it's just Python. There's no connection of your problem with CherryPy, this is not something which has to be documented.
3) My guess is that you just don't know how os.subprocess or bash executable lookup works. However I cannot be sure, because you did not show any reproducible examples of code you think you have problems with.

--Sviatoslav.

Sent from my phone, please pardon any typos.

On Fri, May 4, 2018, 03:28 Jibun no Kage <[hidden email]> wrote:
That could be, but how would anyone know?  When I run python <script name>.py (which is a CherryPy) script, I am executing the default python for the environment, which happens to be 2.7.x.  The CherryPy examples to not qualify doing anything different.  For example...

import cherrypy

class
HelloWorld(object):

    @cherrypy.expose

    def
index(self):
        return
"Hello world!"

if
__name__ == '__main__':
    cherrypy.quickstart(HelloWorld())

Run as "python <script name>.py"  This is taken straight from the online CherryPy official documentation.  So, I should be 100% python version consistent, regardless of the invocation.  Of it was otherwise, then CherryPy documentation should be explicit on this point, no?  None of the modules I am calling other than CherryPy as imported python 3 specific.  Well, less the pending backports, when I get to that point.

Schorschi

On 05/02/2018 14:53, Sviatoslav Sydorenko wrote:
It is supported, but you are doing it wrong. So it looks to me that you are spawning python3 interpreter from within python2 process.

2018-05-02 23:36 GMT+02:00 Jibun no Kage <[hidden email]>:
Thanks for the suggestions.  Unfortunately, at this time I can't address Python 3.  Python 2 only environment at the moment.  Odd, CherryPy documentation states that Python 2.7 is supported, but not surprised something tripped up.  Long term, as we revise the classes, migrate off of Python 2, expect to import and use them directly, as you suggest.

Schorschi


On 04/28/2018 23:21, Sviatoslav Sydorenko wrote:
1) don't use os module to spawn subprocesses, just have a function in those modules, import it and call it then.
2) you probably wrote scripts in a manner/syntax, which is incompatible with Python 3 and run CherryPy under Python 3. Or maybe just a way you run scripts is different when you spawn subprocesses. Anyway, I'd recommend you to not use subprocesses and do normal direct imports.
To mitigate cross python compatibility problems I strongly recommend you writing scripts using syntax, which works in both or completely switch to Python 3.
Look for a package distribution called "six", it may provide you some helpers for cross compat.

--Sviatoslav.

Sent from my phone, please pardon any typos.


On Sun, Apr 29, 2018, 06:12 <[hidden email]> wrote:
Yeah, I got a variant of it working.  Not sure it is pretty, but works.  We have a lot of existing python scripts we want to continue to use from console (Linux).  We just wanted to wrapper these with some simple web pages, to make them a bit easier to use.  So, I just imported the os module, can execute the script.  This is all internal stuff, nothing that has to be production hardened or such.  This lets the guys writing the python scripts for console do their thing, and lets me provide another avenue for execution.

I plan to swing back around on this one, and do more work, but using os module for now works.  I did notice one odd thing, a couple of our python classes/functions that work great from console, break when I import them to a CherryPy script, and then execute as you note.  One error is that bit shifting with the '<<' operator gripes about how it is used.  Only get that error, when the given function/routine is imported to CherryPy environment.  From console (python 2), no error at all.


On Saturday, April 28, 2018 at 1:04:56 PM UTC-7, [hidden email] wrote:
Have existing python script file that I want to execute from within a cherrypy web site/page.  How can this be done 100% native to cherrypy?  Many examples reference java or CGI mods, that is not acceptable nor should it be IMHO.  There has to be a 100% cherrypi method to do this, no?
--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.




--
Kind regards,

Sviatoslav Sydorenko
cell #: +380978963757






--
Kind regards,

Sviatoslav Sydorenko
cell #: +380978963757
--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.




--
Kind regards,

Sviatoslav Sydorenko
cell #: +380978963757

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Have existing python script file that I want to execute from within a cherrypy web site/page. How can this be done 100% native to cherrypy?

Sviatoslav Sydorenko-2
Just as a tip, in Ansible when running tests we normally try to figure out the absolute executable path and then pass that to all process spawning functions to be explicit about what we run:
https://github.com/ansible/ansible/blob/6352e67/test/runner/lib/util.py#L129

2018-05-06 2:37 GMT+02:00 Jibun no Kage <[hidden email]>:
Red Hat!  Wow.  I have quite a few friends at Red Hat, US based.  Cool.  I worked for a Fortune 50 Bank, for many years, designed several virtualization standards for same.  Red Hat has recruited from the same Fortunate 50 Bank for years, even a former boss of mine now is at RH.  I know a number of the guys on the CloudForms sales and consulting team, if you are familiar with CloudForms.

I read through the VirtualEnv documentation, yup, it is just isolation in simple form, in my world (of virtualization) we would call this a 'type' of application isolation, but... it not as sophisticated as Docker even, true.  Agree, I think your comment "My point is that when running app and subprocesses you may unintentionally switch versions" is exactly what might, likely is happening at some point, this is what I was thinking when I said the CherryPy software stack is just exposing something odd... it just happens to happen, when CherryPy is used.  It could happen via other scenarios as well of course.

To tackle the scenario, I am setting up a 'qualified' reference platform, that I am going to make everyone validate against.  The reference platform will be explicitly documented, configuration via Ansible, etc., to make sure it is always consistent.  Since I will need explicit control of what modules, versions etc. used, will be able to have greater control.  I do plan to use Pipenv as well, but will need to spend some time with it.

Schorschi


On 05/05/2018 17:19, Sviatoslav Sydorenko wrote:


2018-05-06 0:48 GMT+02:00 Jibun no Kage <[hidden email]>:
Not to debate, from my perspective, you are talking about virtualization, about application virtualization.  Per the link (https://virtualenv.pypa.io/en/stable/) you sent me, "In all these cases, virtualenv can help you. It creates an environment that has its own installation directories, that doesn’t share libraries with other virtualenv environments (and optionally doesn’t access the globally installed libraries either)."  That is 'application' virtualization from my perspective.  :)

Well, my assumption might be wrong, but I have a feeling that you might not realize how virtualenv works. While it contains word "virtual" it's still not a typical virtualization, nor isolation. Yes, it creates a separate "env" folder with binaries linked against some other (system) installation of python and site-packages, which won't collide with that system install of python. So you add that thing to $PATH or invoke directly.
My point is that when running app and subprocesses you may unintentionally switch versions when being not careful (for example, when you reset/erase env vars available for subshell). And since I don't know anything about how you run web app and scripts it is hard for me to point where to look closer.
So logging executable path and env vars would help you figure out more details about this.
 

As for code examples, I can't publish the explicit code, if I could, I would have.  I realize this limits your assistance, just the same, thanks for the insight, and suggestions, you have helped, I appreciate it.

You don't have to expose any of your super secret business logic or so. The rule of thumb is to post minimal amount of similar code, which might contain dummy "hello world", but still show how exactly to spawn subprocesses, which from your words looks like:

cp_app.py:
import os
import cherrypy

class RootWebApp:
    @cherrypy.expose
    def index(self):
        exec_result = os.subprocess.run(['/path/to/specific/py/env/bin/python', 'some_script.py'])  # might be different and in fact has lots of arguments to play with.
        return exec_result.stdout

    @classmethod
    def run(cls):
        cherrypy.quickstart(cls(), '', {})


def main():
    RootWebApp.run()


__name__ == '__main__' and main()

some_script.py:
def main():
    print('Hello world!')


__name__ == '__main__' and main()

As you can see, your example doesn't have to include any logic specific to your project, but to your env and the way your components interact with each other. This would help with troubleshooting.


Since your domain is UA... you are in Ukraine?  I spent most of my summers as a kid in Germany, even got to visit Hungry, Poland and Yugoslavia (when it still existed), even East Germany once, but I never got to the Ukraine.  Maybe when I retire and have time, I can get back to Eastern Europe again.

I used to live in Ukraine for my whole life, but 3 months ago I moved to Brno, Czech Republic for Red Hat :)
 


Schorschi


On 05/05/2018 02:28, Sviatoslav Sydorenko wrote:
I'm not talking about virtualization at all. I only mentioned https://virtualenv.pypa.io/en/stable/
There's also Pipenv which exposes a bit more full feature set https://packaging.python.org/tutorials/managing-dependencies/

Besides this, I cannot help you with your problem, because this would require you posting a real code example with a reproducible issue.

2018-05-05 2:57 GMT+02:00 Jibun no Kage <[hidden email]>:
I am under the impression you did not see my other note, showing an an example of the odd scenario I found, where the non-CherryPy environment software 'stack' accepts an enum based on a simple class definition, but the CherryPy inclusive environment software 'stack' does not.  Believe CherryPy is exposing something odd, rather than creating something odd, as I noted before.

Yes, virtualization is a best practice, and where and when we can use it, it is, but not all micro-controller environments can be emulated, for example BCM processors are very hard to emulate under Qemu.  I have design and supported virtual labs and large scale production environments for over 16 years, just about every variant of VMware, KVM, and even Hyper-V (and Microsoft Virtual Server before Hyper-V).  Moreover, running applications in slices, in such as things like docker (even LXC) is a best practice when possible, application virtualization versus OS virtualization.  But micro-controllers are often so specialized and narrow in resource scope, that is not an option.  The interesting scenario is when ARM (as now ATOM) based systems continue to scale up, from 32bit to 64bit, greater than 1GB RAM, etc., more CPU cores, etc.  Then, realistic partitioning of micro-controllers will be possible.  Looking forward to it.

Side note, much of VMware's ESXi server solution, under the hood/behind the scenes, leverages quite a bit of python in its control (console) environment.  My first experience with python came from its internal use in ESXi.

Schorschi


On 05/04/2018 15:16, Sviatoslav Sydorenko wrote:
First of all, it's not CherryPy picking interpreter, but vice versa - you run CherryPy app under a certain interpreter (better explicitly chosen).
Also, I advise you to check sys.executable to learn which binary exactly you use.
Finally, it is best practice to use a virtualenv to isolate environments for different apps you run.

--Sviatoslav.

Sent from my phone, please pardon any typos.

On Fri, May 4, 2018, 21:45 Jibun no Kage <[hidden email]> wrote:
Ah... Yes, I know how shebang works.  But, I had not worried about it, since the expectation was that everything would be completely python 2.  I went snooping... all the 'existing' child scripts, are 'banged' explicitly to Python2.  However, one of my test scripts, I was using/learning CherryPy was not explicitly qualified.  So I added a bit of code to the CherryPy script and left it NOT banged on purpose to see what CherryPy was defaulting to:

>>> Sys Get Version 2
>>> Platform Get Version 2.7.13

So CherryPy defaults to system default, and it is version 2, the child scripts are directed to version 2 via shebang.  All the modules used by the child scripts were developed in a version 2 context.  Moreover, I modified one of the child scripts to report what version bash was defaulting to when invoked by bash, a sanity check to validate, and it also reported version 2.

Child script direct invocation...
[2018-05-04 19:27:13 UTC]  >>> Platform Get Version 2.7.13
[2018-05-04 19:27:13 UTC]  >>> Sys Get Version 2

Child script indirect invocation via the os shell/bash trick...
[2018-05-04 19:29:43 UTC]  >>> Platform Get Version 2.7.13
[2018-05-04 19:29:43 UTC]  >>> Sys Get Version 2

So the OS default is 2, the scripts with os module direction are version 2,  the CherryPy defaults to 2, the subprocess defaults to 2.

So there is still something different between module load into CherryPy and invoking it, and module load into child script and then invoking it, the child script invoking always works, the module load to CherryPy then invoking, never works, the modules that exhibit the issue.  I will find it, in due course, because like we agreed, really want to import the modules and just use them directly, and avoid the os shell/bash trick.  At this point it has to be something really odd, tripping it.  Likely, it is some funky code in the module that is not very pythontic as it should be, since we do a lot of micro-controller/hardware driving python coding.

Not to confuse things, but I already found one issue and resolved it, in our development lab.  It had to do with different enum modules clashing, I discovered that 'enum34' was conflicting with 'enum' module.  On one specific system, the modules installed was incorrect, got to love when (many) developers touch stuff... oh... wait, I am one of the developers.  Ouch.

And, thanks for the comments and insight, appreciate it.

Schorschi



On 05/03/2018 22:45, Sviatoslav Sydorenko wrote:
1) You run CherryPy correctly, but it doesn't matter, because you obviously run the scripts themselves in a way that they might choose different Python env or even version. Do you know about shebang, for example? Anyway, you didn't provide sufficient details to know what happens exactly.
2) Running scripts is not CherryPy specific, it's just Python. There's no connection of your problem with CherryPy, this is not something which has to be documented.
3) My guess is that you just don't know how os.subprocess or bash executable lookup works. However I cannot be sure, because you did not show any reproducible examples of code you think you have problems with.

--Sviatoslav.

Sent from my phone, please pardon any typos.

On Fri, May 4, 2018, 03:28 Jibun no Kage <[hidden email]> wrote:
That could be, but how would anyone know?  When I run python <script name>.py (which is a CherryPy) script, I am executing the default python for the environment, which happens to be 2.7.x.  The CherryPy examples to not qualify doing anything different.  For example...

import cherrypy

class
HelloWorld(object):

    @cherrypy.expose

    def
index(self):
        return
"Hello world!"

if
__name__ == '__main__':
    cherrypy.quickstart(HelloWorld())

Run as "python <script name>.py"  This is taken straight from the online CherryPy official documentation.  So, I should be 100% python version consistent, regardless of the invocation.  Of it was otherwise, then CherryPy documentation should be explicit on this point, no?  None of the modules I am calling other than CherryPy as imported python 3 specific.  Well, less the pending backports, when I get to that point.

Schorschi

On 05/02/2018 14:53, Sviatoslav Sydorenko wrote:
It is supported, but you are doing it wrong. So it looks to me that you are spawning python3 interpreter from within python2 process.

2018-05-02 23:36 GMT+02:00 Jibun no Kage <[hidden email]>:
Thanks for the suggestions.  Unfortunately, at this time I can't address Python 3.  Python 2 only environment at the moment.  Odd, CherryPy documentation states that Python 2.7 is supported, but not surprised something tripped up.  Long term, as we revise the classes, migrate off of Python 2, expect to import and use them directly, as you suggest.

Schorschi


On 04/28/2018 23:21, Sviatoslav Sydorenko wrote:
1) don't use os module to spawn subprocesses, just have a function in those modules, import it and call it then.
2) you probably wrote scripts in a manner/syntax, which is incompatible with Python 3 and run CherryPy under Python 3. Or maybe just a way you run scripts is different when you spawn subprocesses. Anyway, I'd recommend you to not use subprocesses and do normal direct imports.
To mitigate cross python compatibility problems I strongly recommend you writing scripts using syntax, which works in both or completely switch to Python 3.
Look for a package distribution called "six", it may provide you some helpers for cross compat.

--Sviatoslav.

Sent from my phone, please pardon any typos.


On Sun, Apr 29, 2018, 06:12 <[hidden email]> wrote:
Yeah, I got a variant of it working.  Not sure it is pretty, but works.  We have a lot of existing python scripts we want to continue to use from console (Linux).  We just wanted to wrapper these with some simple web pages, to make them a bit easier to use.  So, I just imported the os module, can execute the script.  This is all internal stuff, nothing that has to be production hardened or such.  This lets the guys writing the python scripts for console do their thing, and lets me provide another avenue for execution.

I plan to swing back around on this one, and do more work, but using os module for now works.  I did notice one odd thing, a couple of our python classes/functions that work great from console, break when I import them to a CherryPy script, and then execute as you note.  One error is that bit shifting with the '<<' operator gripes about how it is used.  Only get that error, when the given function/routine is imported to CherryPy environment.  From console (python 2), no error at all.


On Saturday, April 28, 2018 at 1:04:56 PM UTC-7, [hidden email] wrote:
Have existing python script file that I want to execute from within a cherrypy web site/page.  How can this be done 100% native to cherrypy?  Many examples reference java or CGI mods, that is not acceptable nor should it be IMHO.  There has to be a 100% cherrypi method to do this, no?
--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.




--
Kind regards,

Sviatoslav Sydorenko
cell #: +380978963757






--
Kind regards,

Sviatoslav Sydorenko
cell #: +380978963757
--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.




--
Kind regards,

Sviatoslav Sydorenko
cell #: +380978963757
--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.




--
Kind regards,

Sviatoslav Sydorenko
cell #: +380978963757

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.