[IPython-User] Executing some function on IPython is slower than a normal python function

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[IPython-User] Executing some function on IPython is slower than a normal python function

Ivan Ogassawara
Dear all,

I'm testing some functionalities of ipython and I'm think I'm doing something wrong.

I'm testing 3 different ways to execute some math operation.

  • 1st using @parallel.parallel(view=dview, block=True) and function map
  • 2nd using single core function (python normal function)
  • 3rd using clients load balance function
the code is here:

https://stackoverflow.com/questions/26039254/executing-some-function-on-ipython-is-slower-than-a-normal-python-function

My result is:

True
True
0.040741 secs (multicore)
0.004004 secs (singlecore)
1.286592 secs (multicore_load_balance)

Why are my multicore routines slower than my single core routine? What is wrong with this approach? What can I do to fix it?

Some environment information: python3.4.1, ipython 2.2.0, numpy 1.9.0, ipcluster starting 8 Engines with LocalEngineSetLauncher


My best regards,

Ivan Ogassawara




_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user
Reply | Threaded
Open this post in threaded view
|

Re: Executing some function on IPython is slower than a normal python function

MinRK
There is always overhead in sending work to your engines and retrieving the results. Because of this, it only makes sense to use a parallel computing tool if the communication overhead is smaller than the time saved by concurrent execution. Parallelizing small tasks almost always makes them slower.

-MinRK

On Fri, Sep 26, 2014 at 6:21 AM, Ivan Ogassawara <[hidden email]> wrote:
Dear all,

I'm testing some functionalities of ipython and I'm think I'm doing something wrong.

I'm testing 3 different ways to execute some math operation.

  • 1st using @parallel.parallel(view=dview, block=True) and function map
  • 2nd using single core function (python normal function)
  • 3rd using clients load balance function
the code is here:

https://stackoverflow.com/questions/26039254/executing-some-function-on-ipython-is-slower-than-a-normal-python-function

My result is:

True
True
0.040741 secs (multicore)
0.004004 secs (singlecore)
1.286592 secs (multicore_load_balance)

Why are my multicore routines slower than my single core routine? What is wrong with this approach? What can I do to fix it?

Some environment information: python3.4.1, ipython 2.2.0, numpy 1.9.0, ipcluster starting 8 Engines with LocalEngineSetLauncher


My best regards,

Ivan Ogassawara




_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user



_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user
Reply | Threaded
Open this post in threaded view
|

Re: Executing some function on IPython is slower than a normal python function

Ivan Ogassawara
Hi MinRK, I agree with that, but I think that here the problem is the approach used. I rewrite this code using fortran and fortran with openmp (I know, i'm comparing bananas and apples, sorry) with N=10000 (more samples) and the executions time are:

(OMP) Elapsed real time =   0.335000008    
Elapsed real time =   0.833999991

with N=1000 (same sample)
(OMP) Elapsed real time =    8.10000002E-02
Elapsed real time =    8.99999961E-03

With iPython with N=1000000 it have a linear response (ipython 4 times slower)
5.546619 secs (singlecore)
20.580165 secs (ipython)


I rewrite my code to test multiprocessing native python, and I have a worst result, comparing it to ipython approach, but better than loadbalance.

My point is, how can i take advantage of ipython to do this operation? 
If I can't, do IPython have some scientific notation to be used to calculate its cost?

My best regards,

Ivan Ogassawara


2014-09-26 15:35 GMT-03:00 MinRK <[hidden email]>:
There is always overhead in sending work to your engines and retrieving the results. Because of this, it only makes sense to use a parallel computing tool if the communication overhead is smaller than the time saved by concurrent execution. Parallelizing small tasks almost always makes them slower.

-MinRK

On Fri, Sep 26, 2014 at 6:21 AM, Ivan Ogassawara <[hidden email]> wrote:
Dear all,

I'm testing some functionalities of ipython and I'm think I'm doing something wrong.

I'm testing 3 different ways to execute some math operation.

  • 1st using @parallel.parallel(view=dview, block=True) and function map
  • 2nd using single core function (python normal function)
  • 3rd using clients load balance function
the code is here:

https://stackoverflow.com/questions/26039254/executing-some-function-on-ipython-is-slower-than-a-normal-python-function

My result is:

True
True
0.040741 secs (multicore)
0.004004 secs (singlecore)
1.286592 secs (multicore_load_balance)

Why are my multicore routines slower than my single core routine? What is wrong with this approach? What can I do to fix it?

Some environment information: python3.4.1, ipython 2.2.0, numpy 1.9.0, ipcluster starting 8 Engines with LocalEngineSetLauncher


My best regards,

Ivan Ogassawara




_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user



_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user



_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user
Reply | Threaded
Open this post in threaded view
|

Re: When i use DataFrame with korean, ?? is shown

강동인



-----Original Message-----
From: "Ivan Ogassawara"<[hidden email]>
To: "Discussions about using IPython. http://ipython.org"<[hidden email]>;
Cc:
Sent: 2014-09-30 (화) 00:49:58
Subject: Re: [IPython-User] Executing some function on IPython is slower than a normal python function
 

Hi MinRK, I agree with that, but I think that here the problem is the approach used. I rewrite this code using fortran and fortran with openmp (I know, i'm comparing bananas and apples, sorry) with N=10000 (more samples) and the executions time are:
 
(OMP) Elapsed real time = 0.335000008
Elapsed real time = 0.833999991
 
with N=1000 (same sample)
(OMP) Elapsed real time = 8.10000002E-02
Elapsed real time = 8.99999961E-03
 
With iPython with N=1000000 it have a linear response (ipython 4 times slower) 
5.546619 secs (singlecore)
20.580165 secs (ipython)
 
 
I rewrite my code to test multiprocessing native python, and I have a worst result, comparing it to ipython approach, but better than loadbalance. 
 
My point is, how can i take advantage of ipython to do this operation?  
If I can't, do IPython have some scientific notation to be used to calculate its cost?
 
My best regards,
 
Ivan Ogassawara
 

2014-09-26 15:35 GMT-03:00 MinRK <[hidden email]>:
There is always overhead in sending work to your engines and retrieving the results. Because of this, it only makes sense to use a parallel computing tool if the communication overhead is smaller than the time saved by concurrent execution. Parallelizing small tasks almost always makes them slower.
 
-MinRK

On Fri, Sep 26, 2014 at 6:21 AM, Ivan Ogassawara <[hidden email]> wrote: 
Dear all, 
 

I'm testing some functionalities of ipython and I'm think I'm doing something wrong.

I'm testing 3 different ways to execute some math operation.

  • 1st using @parallel.parallel(view=dview, block=True) and function map
  • 2nd using single core function (python normal function)
  • 3rd using clients load balance function
the code is here:
 
https://stackoverflow.com/questions/26039254/executing-some-function-on-ipython-is-slower-than-a-normal-python-function
 

My result is:

True
True
0.040741 secs (multicore)
0.004004 secs (singlecore)
1.286592 secs (multicore_load_balance)

 

Why are my multicore routines slower than my single core routine? What is wrong with this approach? What can I do to fix it?

Some environment information: python3.4.1, ipython 2.2.0, numpy 1.9.0, ipcluster starting 8 Engines with LocalEngineSetLauncher

 

My best regards,

Ivan Ogassawara 

 

 

 
_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user

 

_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user

 

_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user

333.PNG (32K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: When i use DataFrame with korean, ?? is shown

Thomas Kluyver-2
Are you using Python 2 or Python 3? This is something where Python 3 is likely to be better.

If you're already on Python 3, show us the code that causes this result, and we'll look into it.

Thomas

On 4 October 2014 23:18, 강동인 <[hidden email]> wrote:



-----Original Message-----
From: "Ivan Ogassawara"<[hidden email]>
To: "Discussions about using IPython. http://ipython.org"<[hidden email]>;
Cc:
Sent: 2014-09-30 (화) 00:49:58
Subject: Re: [IPython-User] Executing some function on IPython is slower than a normal python function
 

Hi MinRK, I agree with that, but I think that here the problem is the approach used. I rewrite this code using fortran and fortran with openmp (I know, i'm comparing bananas and apples, sorry) with N=10000 (more samples) and the executions time are:
 
(OMP) Elapsed real time = 0.335000008
Elapsed real time = 0.833999991
 
with N=1000 (same sample)
(OMP) Elapsed real time = 8.10000002E-02
Elapsed real time = 8.99999961E-03
 
With iPython with N=1000000 it have a linear response (ipython 4 times slower) 
5.546619 secs (singlecore)
20.580165 secs (ipython)
 
 
I rewrite my code to test multiprocessing native python, and I have a worst result, comparing it to ipython approach, but better than loadbalance. 
 
My point is, how can i take advantage of ipython to do this operation?  
If I can't, do IPython have some scientific notation to be used to calculate its cost?
 
My best regards,
 
Ivan Ogassawara
 

2014-09-26 15:35 GMT-03:00 MinRK <[hidden email]>:
There is always overhead in sending work to your engines and retrieving the results. Because of this, it only makes sense to use a parallel computing tool if the communication overhead is smaller than the time saved by concurrent execution. Parallelizing small tasks almost always makes them slower.
 
-MinRK

On Fri, Sep 26, 2014 at 6:21 AM, Ivan Ogassawara <[hidden email]> wrote: 
Dear all, 
 

I'm testing some functionalities of ipython and I'm think I'm doing something wrong.

I'm testing 3 different ways to execute some math operation.

  • 1st using @parallel.parallel(view=dview, block=True) and function map
  • 2nd using single core function (python normal function)
  • 3rd using clients load balance function
the code is here:
 
https://stackoverflow.com/questions/26039254/executing-some-function-on-ipython-is-slower-than-a-normal-python-function
 

My result is:

True
True
0.040741 secs (multicore)
0.004004 secs (singlecore)
1.286592 secs (multicore_load_balance)

 

Why are my multicore routines slower than my single core routine? What is wrong with this approach? What can I do to fix it?

Some environment information: python3.4.1, ipython 2.2.0, numpy 1.9.0, ipcluster starting 8 Engines with LocalEngineSetLauncher

 

My best regards,

Ivan Ogassawara 

 

 

 
_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user

 

_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user

 

_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user



_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user
Reply | Threaded
Open this post in threaded view
|

Re: When i use DataFrame with korean, ?? is shown

강동인

 


i use ipython notebook , maybe with python 2 . 

 

I assume the problem is unicode and unf8. but how can i set up ?

 

When i type this code , 

-------------------------

 import pandas 

 pandas.read_csv('file name.csv')

 

---------------------------

Korean string is broken.

 

 

 

  


-----Original Message-----
From: "Thomas Kluyver"<[hidden email]>
To: "Discussions about using IPython. http://ipython.org"<[hidden email]>;
Cc:
Sent: 2014-10-06 (월) 01:38:05
Subject: Re: [IPython-User] When i use DataFrame with korean, ?? is shown
 

Are you using Python 2 or Python 3? This is something where Python 3 is likely to be better.
 
If you're already on Python 3, show us the code that causes this result, and we'll look into it.
 
Thomas 

On 4 October 2014 23:18, 강동인 <[hidden email]> wrote:



-----Original Message-----
From: "Ivan Ogassawara"<[hidden email]>
To: "Discussions about using IPython. http://ipython.org"<[hidden email]>;
Cc:
Sent: 2014-09-30 (화) 00:49:58
Subject: Re: [IPython-User] Executing some function on IPython is slower than a normal python function

Hi MinRK, I agree with that, but I think that here the problem is the approach used. I rewrite this code using fortran and fortran with openmp (I know, i'm comparing bananas and apples, sorry) with N=10000 (more samples) and the executions time are:
(OMP) Elapsed real time = 0.335000008
Elapsed real time = 0.833999991
with N=1000 (same sample)
(OMP) Elapsed real time = 8.10000002E-02
Elapsed real time = 8.99999961E-03
With iPython with N=1000000 it have a linear response (ipython 4 times slower)
5.546619 secs (singlecore)
20.580165 secs (ipython)
I rewrite my code to test multiprocessing native python, and I have a worst result, comparing it to ipython approach, but better than loadbalance.
My point is, how can i take advantage of ipython to do this operation?
If I can't, do IPython have some scientific notation to be used to calculate its cost?
My best regards,
Ivan Ogassawara

2014-09-26 15:35 GMT-03:00 MinRK <[hidden email]>:
There is always overhead in sending work to your engines and retrieving the results. Because of this, it only makes sense to use a parallel computing tool if the communication overhead is smaller than the time saved by concurrent execution. Parallelizing small tasks almost always makes them slower.
-MinRK

On Fri, Sep 26, 2014 at 6:21 AM, Ivan Ogassawara <[hidden email]> wrote:
Dear all,

I'm testing some functionalities of ipython and I'm think I'm doing something wrong.

I'm testing 3 different ways to execute some math operation.

  • 1st using @parallel.parallel(view=dview, block=True) and function map
  • 2nd using single core function (python normal function)
  • 3rd using clients load balance function
the code is here:
https://stackoverflow.com/questions/26039254/executing-some-function-on-ipython-is-slower-than-a-normal-python-function

My result is:

True
True
0.040741 secs (multicore)
0.004004 secs (singlecore)
1.286592 secs (multicore_load_balance)

 

Why are my multicore routines slower than my single core routine? What is wrong with this approach? What can I do to fix it?

Some environment information: python3.4.1, ipython 2.2.0, numpy 1.9.0, ipcluster starting 8 Engines with LocalEngineSetLauncher

My best regards,

Ivan Ogassawara

_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user


_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user


_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user

 

_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user
Reply | Threaded
Open this post in threaded view
|

Re: When i use DataFrame with korean, ?? is shown

Thomas Kluyver-2
Try playing around with different encoding= parameters for read_csv().

On 5 October 2014 09:53, 강동인 <[hidden email]> wrote:

 


i use ipython notebook , maybe with python 2 . 

 

I assume the problem is unicode and unf8. but how can i set up ?

 

When i type this code , 

-------------------------

 import pandas 

 pandas.read_csv('file name.csv')

 

---------------------------

Korean string is broken.

 

 

 

  


-----Original Message-----
From: "Thomas Kluyver"<[hidden email]>
To: "Discussions about using IPython. http://ipython.org"<[hidden email]>;
Cc:

Sent: 2014-10-06 (월) 01:38:05
Subject: Re: [IPython-User] When i use DataFrame with korean, ?? is shown
 

Are you using Python 2 or Python 3? This is something where Python 3 is likely to be better.
 
If you're already on Python 3, show us the code that causes this result, and we'll look into it.
 
Thomas 

On 4 October 2014 23:18, 강동인 <[hidden email]> wrote:



-----Original Message-----
From: "Ivan Ogassawara"<[hidden email]>
To: "Discussions about using IPython. http://ipython.org"<[hidden email]>;
Cc:
Sent: 2014-09-30 (화) 00:49:58
Subject: Re: [IPython-User] Executing some function on IPython is slower than a normal python function

Hi MinRK, I agree with that, but I think that here the problem is the approach used. I rewrite this code using fortran and fortran with openmp (I know, i'm comparing bananas and apples, sorry) with N=10000 (more samples) and the executions time are:
(OMP) Elapsed real time = 0.335000008
Elapsed real time = 0.833999991
with N=1000 (same sample)
(OMP) Elapsed real time = 8.10000002E-02
Elapsed real time = 8.99999961E-03
With iPython with N=1000000 it have a linear response (ipython 4 times slower)
5.546619 secs (singlecore)
20.580165 secs (ipython)
I rewrite my code to test multiprocessing native python, and I have a worst result, comparing it to ipython approach, but better than loadbalance.
My point is, how can i take advantage of ipython to do this operation?
If I can't, do IPython have some scientific notation to be used to calculate its cost?
My best regards,
Ivan Ogassawara

2014-09-26 15:35 GMT-03:00 MinRK <[hidden email]>:
There is always overhead in sending work to your engines and retrieving the results. Because of this, it only makes sense to use a parallel computing tool if the communication overhead is smaller than the time saved by concurrent execution. Parallelizing small tasks almost always makes them slower.
-MinRK

On Fri, Sep 26, 2014 at 6:21 AM, Ivan Ogassawara <[hidden email]> wrote:
Dear all,

I'm testing some functionalities of ipython and I'm think I'm doing something wrong.

I'm testing 3 different ways to execute some math operation.

  • 1st using @parallel.parallel(view=dview, block=True) and function map
  • 2nd using single core function (python normal function)
  • 3rd using clients load balance function
the code is here:
https://stackoverflow.com/questions/26039254/executing-some-function-on-ipython-is-slower-than-a-normal-python-function

My result is:

True
True
0.040741 secs (multicore)
0.004004 secs (singlecore)
1.286592 secs (multicore_load_balance)

 

Why are my multicore routines slower than my single core routine? What is wrong with this approach? What can I do to fix it?

Some environment information: python3.4.1, ipython 2.2.0, numpy 1.9.0, ipcluster starting 8 Engines with LocalEngineSetLauncher

My best regards,

Ivan Ogassawara

_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user


_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user


_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user

 

_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user



_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user