[SOLVED] concatenate function

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

[SOLVED] concatenate function

ferreirafm
This post was updated on .
Hi List,
I've coded three functions that I would like to concatenate. I mean, run them one after another. The third function depends on the results of the second function, which depends on the results of the first one. When I call one function after another, python runs them at the same time causing obvious errors messages. I've tried to call one of them from inside another but no way. Any clues are appreciated.
Complete code goes here:
http://ompldr.org/vZDB4OQ

Reply | Threaded
Open this post in threaded view
|

Re: concatenate function

Ian Kelly-2
On Tue, Mar 13, 2012 at 8:35 AM, ferreirafm <[hidden email]> wrote:
> Hi List,
> I've coded three functions that I would like to concatenate. I mean, run
> them one after another. The third function depends on the results of the
> second function, which depends on the results of the first one. When I call
> one function after another, python runs them at the same time causing
> obvious errors messages. I've tried to call one of them from inside another
> but no way. Any clues are appreciated.
> Complete code goes here:
> http://ompldr.org/vZDB4OQ

They don't look to me like they would run at the same time --
subprocess.call is supposed to wait for the subprocess to finish.
What error messages are you getting?
--
http://mail.python.org/mailman/listinfo/python-list
Reply | Threaded
Open this post in threaded view
|

Re: concatenate function

Robert Kern-2
In reply to this post by ferreirafm
On 3/13/12 2:35 PM, ferreirafm wrote:
> Hi List,
> I've coded three functions that I would like to concatenate. I mean, run
> them one after another. The third function depends on the results of the
> second function, which depends on the results of the first one. When I call
> one function after another, python runs them at the same time causing
> obvious errors messages. I've tried to call one of them from inside another
> but no way. Any clues are appreciated.
> Complete code goes here:
> http://ompldr.org/vZDB4OQ

Just to clarify, the Python functions are indeed running consecutively, not
concurrently. Your Python functions write scripts and then use subprocess.call()
to make qsub (an external program) to submit those scripts to a job queue. What
you are calling a "function" in your post are these scripts. Please don't call
them "functions". It's confusing.

Python is not running these scripts concurrently. Your job queue is.
subprocess.call() will wait until qsub returns. However, qsub just submits the
script to the job queue; it does not wait until the job is completed. Most
qsub-using job queues can be set up to make jobs depend on the completion of
other jobs. You will need to read the documentation of your job queue to figure
out how to do this. Once you figure out the right arguments to give to qsub,
your Python code is already more or less correct.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

--
http://mail.python.org/mailman/listinfo/python-list
Reply | Threaded
Open this post in threaded view
|

Re: concatenate function

ferreirafm
In reply to this post by Ian Kelly-2
Hi Ian,
That what I have:
> burst.py
Your job 46665 ("top_n_pdb.qsub") has been submitted
Your job 46666 ("extr_pdb.qsub") has been submitted
Your job 46667 ("combine_top.qsub") has been submitted

The first job runs quite well. The second is still runing and the third issue the following:
>  more combine_top.qsub.e46667
ERROR: Cannot open PDB file "S_3MSEB_26_0032.pdb"
ERROR:: Exit from: src/core/import_pose/import_pose.cc line: 199

Reply | Threaded
Open this post in threaded view
|

Re: concatenate function

James Elford
In reply to this post by ferreirafm
On 13/03/12 14:35, ferreirafm wrote:
> Hi List,
> I've coded three functions that I would like to concatenate. I mean, run
> them one after another. The third function depends on the results of the
> second function, which depends on the results of the first one. When I call
> one function after another, python runs them at the same time causing
> obvious errors messages. I've tried to call one of them from inside another
> but no way. Any clues are appreciated.


> Complete code goes here:
> http://ompldr.org/vZDB4OQ

Do you think you could provide a much shorter example to illustrate what
you need? In general, when you want to run one function on the result of
another, you can do something like:

<<< def increment_all(l);
... return [i+1 for i in l]

<<< increment_all(increment_all(range(3))
[2, 3, 4]

Here we apply the function increment_all to the result of the function
increment_all.

If you are talking about the "results" of each function in terms of it
mutating an object, and then the next function mutating the same object
in a (possibly) different way, then calling the functions in order will
do what you want.

l = [0, 3, 5, 2]
l.append(10) # [0, 3, 5, 2, 10]
l.sort() # [0, 2, 3, 5, 10]
l.append(3) # [0, 2, 3, 5, 10, 3]

James

>
>
>
> --
> View this message in context: http://python.6.n6.nabble.com/concatenate-function-tp4574176p4574176.html
> Sent from the Python - python-list mailing list archive at Nabble.com.

--
http://mail.python.org/mailman/listinfo/python-list
Reply | Threaded
Open this post in threaded view
|

Re: concatenate function

ferreirafm
In reply to this post by Robert Kern-2
Hi Robert,
Thanks for you kind replay and I'm sorry for my semantic mistakes.
Indeed, that's what I'm doing: qsub-ing different cshell scripts. Certainly, that's not the best approach and the only problem. I've unsuccessfully tried to set an os.environ and call qsub from it. However, subprocess.Popen seems not accept to run "qsub" over a second program. Do you have a over come to this issue?
Code goes here:
http://ompldr.org/vZDB5YQ
Reply | Threaded
Open this post in threaded view
|

Re: concatenate function

ferreirafm
In reply to this post by James Elford
Hi James, thank you for your replay. Indeed, the problem is qsub. And as warned by Robert, I don't have functions properly, but just scripts.
 
Reply | Threaded
Open this post in threaded view
|

Re: concatenate function

Robert Kern-2
In reply to this post by ferreirafm
On 3/13/12 3:59 PM, ferreirafm wrote:
> Hi Robert,
> Thanks for you kind replay and I'm sorry for my semantic mistakes.
> Indeed, that's what I'm doing: qsub-ing different cshell scripts. Certainly,
> that's not the best approach and the only problem.

It's not a problem to write out a script and have qsub run it. That's a
perfectly fine thing to do. You need to read the documentation for your job
queue to find out the right arguments to give to qsub to make it wait until the
first job finishes before executing the second job. This is not a Python
problem. You just need to find the right flags to give to qsub.

Alternately, you could just make a single .qsub script running all three of your
programs in a single job instead of making three separate .qsub scripts.

> I've unsuccessfully tried
> to set an os.environ and call qsub from it. However, subprocess.Popen seems
> not accept to run "qsub" over a second program. Do you have a over come to
> this issue?
> Code goes here:
> http://ompldr.org/vZDB5YQ

When you report a problem, you should copy-and-paste the output that you got and
also state the output that you expected. I have no idea what you mean when you
say "subprocess.Popen seems not accept to run "qsub" over a second program."

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

--
http://mail.python.org/mailman/listinfo/python-list
Reply | Threaded
Open this post in threaded view
|

Re: concatenate function

James Elford
In reply to this post by ferreirafm
On 13/03/12 16:02, ferreirafm wrote:
> Hi James, thank you for your replay. Indeed, the problem is qsub. And as
> warned by Robert, I don't have functions properly, but just scripts.
>  
>
> --
> View this message in context: http://python.6.n6.nabble.com/concatenate-function-tp4574176p4574511.html
> Sent from the Python - python-list mailing list archive at Nabble.com.

It looks like you're not calling wait() on your subprocesses: you're
effectively launching a bunch of processes, then not waiting for them to
finish before you ask the next process to operate on the same file.

If you haven't given it a good look-over already, the subprocess
documentation [1] is worth taking a little time over.

        [1]: http://docs.python.org/library/subprocess.html#popen-objects

James
--
http://mail.python.org/mailman/listinfo/python-list
Reply | Threaded
Open this post in threaded view
|

Re: concatenate function

ferreirafm
In reply to this post by Robert Kern-2
Robert Kern-2 wrote
When you report a problem, you should copy-and-paste the output that you got and
also state the output that you expected. I have no idea what you mean when you
say "subprocess.Popen seems not accept to run "qsub" over a second program."
Code goes here:
http://ompldr.org/vZDB5YQ

stdout:
$ no_name.py --toplist top_percent.list
Traceback (most recent call last):
  File "/home6/psloliveira/ferreirafm/bin/no_name.py", line 73, in <module>
    main()
  File "/home6/psloliveira/ferreirafm/bin/no_name.py", line 68, in main
    comb_slt(toplist)
  File "/home6/psloliveira/ferreirafm/bin/no_name.py", line 55, in comb_slt
    subprocess.Popen([cmd, options], env=qsub_env)
  File "/share/apps/python/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/share/apps/python/lib/python2.7/subprocess.py", line 1228, in _execute_child
    raise child_exception
OSError: [Errno 13] Permission denied
Reply | Threaded
Open this post in threaded view
|

Re: concatenate function

Robert Kern-2
On 3/13/12 6:01 PM, ferreirafm wrote:

>
> Robert Kern-2 wrote
>>
>> When you report a problem, you should copy-and-paste the output that you
>> got and
>> also state the output that you expected. I have no idea what you mean when
>> you
>> say "subprocess.Popen seems not accept to run "qsub" over a second
>> program."
>>
>
> Code goes here:
> http://ompldr.org/vZDB5YQ
>
> stdout:
> $ no_name.py --toplist top_percent.list
> Traceback (most recent call last):
>    File "/home6/psloliveira/ferreirafm/bin/no_name.py", line 73, in<module>
>      main()
>    File "/home6/psloliveira/ferreirafm/bin/no_name.py", line 68, in main
>      comb_slt(toplist)
>    File "/home6/psloliveira/ferreirafm/bin/no_name.py", line 55, in comb_slt
>      subprocess.Popen([cmd, options], env=qsub_env)
>    File "/share/apps/python/lib/python2.7/subprocess.py", line 679, in
> __init__
>      errread, errwrite)
>    File "/share/apps/python/lib/python2.7/subprocess.py", line 1228, in
> _execute_child
>      raise child_exception
> OSError: [Errno 13] Permission denied

You need to use a command list like this:

['qsub', 'combine_silent.linuxgccrelease', '-database',
'/home6/psloliveira/rosetta_database/', ...]

The program to run ("qsub", not "qsub combine_silent.linuxgccrelease") and each
individual argument must be a separate string in the list. You cannot combine
them together with spaces. The reason you get a "Permission denied" error is
that it tried to find an executable file named "qsub
combine_silent.linuxgccrelease" and, obviously, could not.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

--
http://mail.python.org/mailman/listinfo/python-list
Reply | Threaded
Open this post in threaded view
|

Re: concatenate function

Chris Rebert-6
On Tue, Mar 13, 2012 at 1:35 PM, Robert Kern <[hidden email]> wrote:

> On 3/13/12 6:01 PM, ferreirafm wrote:
>> Robert Kern-2 wrote
>>> When you report a problem, you should copy-and-paste the output that you
>>> got and
>>> also state the output that you expected. I have no idea what you mean
>>> when
>>> you
>>> say "subprocess.Popen seems not accept to run "qsub" over a second
>>> program."
>>>
>>
>> Code goes here:
>> http://ompldr.org/vZDB5YQ
>>
>> stdout:
>> $ no_name.py --toplist top_percent.list
>> Traceback (most recent call last):
>>   File "/home6/psloliveira/ferreirafm/bin/no_name.py", line 73, in<module>
>>     main()
>>   File "/home6/psloliveira/ferreirafm/bin/no_name.py", line 68, in main
>>     comb_slt(toplist)
>>   File "/home6/psloliveira/ferreirafm/bin/no_name.py", line 55, in
>> comb_slt
>>     subprocess.Popen([cmd, options], env=qsub_env)
>>   File "/share/apps/python/lib/python2.7/subprocess.py", line 679, in
>> __init__
>>     errread, errwrite)
>>   File "/share/apps/python/lib/python2.7/subprocess.py", line 1228, in
>> _execute_child
>>     raise child_exception
>> OSError: [Errno 13] Permission denied
>
>
> You need to use a command list like this:
>
> ['qsub', 'combine_silent.linuxgccrelease', '-database',
> '/home6/psloliveira/rosetta_database/', ...]
>
> The program to run ("qsub", not "qsub combine_silent.linuxgccrelease") and
> each individual argument must be a separate string in the list. You cannot
> combine them together with spaces. The reason you get a "Permission denied"
> error is that it tried to find an executable file named "qsub
> combine_silent.linuxgccrelease" and, obviously, could not.

See also the first "Note" box (and the description of "args" generally) under
http://docs.python.org/library/subprocess.html#popen-constructor

Cheers,
Chris
--
http://mail.python.org/mailman/listinfo/python-list
Reply | Threaded
Open this post in threaded view
|

Re: concatenate function

ferreirafm
Hi there,
The problem has been solved. I 've decided to run the python script as argument of qsub instead of run qsub from inside of the script itself. I also use .wait() as suggest by colleagues above.
Final code goes here:
http://ompldr.org/vZDFiag

Thank you very much for helping.