l = range(int(1E9))

classic Classic list List threaded Threaded
74 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

Cecil Westerhof
If I execute:
    l = range(int(1E9)

The python process gobbles up all the memory and is killed. The
problem is that after this my swap is completely used, because other
processes have swapped to it. This make those programs more slowly. Is
there a way to circumvent Python claiming all the memory?

By the way: this is CPython 2.7.8.

--
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof


Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

Grant Edwards-7
On 2015-04-30, Cecil Westerhof <Cecil at decebal.nl> wrote:
> If I execute:
>     l = range(int(1E9)
>
> The python process gobbles up all the memory and is killed. The
> problem is that after this my swap is completely used, because other
> processes have swapped to it. This make those programs more slowly.
> Is there a way to circumvent Python claiming all the memory?

I presume "don't do that" has already occured to you?

You can always use ulimit to limit the memory allowed for the process
running Python.

--
Grant Edwards               grant.b.edwards        Yow! Should I get locked
                                  at               in the PRINCICAL'S
                              gmail.com            OFFICE today -- or have
                                                   a VASECTOMY??


Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

Jon Ribbens-5
In reply to this post by Cecil Westerhof
On 2015-04-30, Cecil Westerhof <Cecil at decebal.nl> wrote:
> If I execute:
>     l = range(int(1E9)
>
> The python process gobbles up all the memory and is killed. The
> problem is that after this my swap is completely used, because other
> processes have swapped to it. This make those programs more slowly. Is
> there a way to circumvent Python claiming all the memory?
>
> By the way: this is CPython 2.7.8.

It's your operating system's job to handle processes.

If you use xrange() instead of range() then you will get an iterator
which will return each of the numbers in turn without any need to
create an enormous list of all of them.


Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

Terry Reedy
In reply to this post by Cecil Westerhof
On 4/30/2015 12:06 PM, Cecil Westerhof wrote:
> If I execute:
>      l = range(int(1E9)

you get a SyntaxError

> The python process gobbles up all the memory and is killed. The
> problem is that after this my swap is completely used, because other
> processes have swapped to it. This make those programs more slowly. Is
> there a way to circumvent Python claiming all the memory?

For this specific case, use xrange or 3.x range

> By the way: this is CPython 2.7.8.



--
Terry Jan Reedy



Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

Gary Herron-2
In reply to this post by Cecil Westerhof
On 04/30/2015 09:06 AM, Cecil Westerhof wrote:
> If I execute:
>      l = range(int(1E9)
>
> The python process gobbles up all the memory and is killed. The
> problem is that after this my swap is completely used, because other
> processes have swapped to it. This make those programs more slowly. Is
> there a way to circumvent Python claiming all the memory?
>
> By the way: this is CPython 2.7.8.

Well, that could be considered the problem.   In Python3, the range
function returns a range object which takes up almost no resources,
while in Python2 it produces a list.  Both can be iterated over, so they
produce the same result in the most common use case (i.e., iteration),
but the Python3 version generates the elements only as needed.

If you really *wanted* the list (but WHY?) in Python3, do
list(range(...)), but then you get what you deserve. :-)

Python3:

 >>> l = range(int(1E9))
 >>> l
range(0, 1000000000)


--
Dr. Gary Herron
Department of Computer Science
DigiPen Institute of Technology
(425) 895-4418



Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

Rob Gaddi
In reply to this post by Cecil Westerhof
On Thu, 30 Apr 2015 10:05:44 -0700, Gary Herron wrote:

> On 04/30/2015 09:06 AM, Cecil Westerhof wrote:
>> If I execute:
>>      l = range(int(1E9)
>>
>> The python process gobbles up all the memory and is killed. The problem
>> is that after this my swap is completely used, because other processes
>> have swapped to it. This make those programs more slowly. Is there a
>> way to circumvent Python claiming all the memory?
>>
>> By the way: this is CPython 2.7.8.
>
> Well, that could be considered the problem.   In Python3, the range
> function returns a range object which takes up almost no resources,
> while in Python2 it produces a list.  Both can be iterated over, so they
> produce the same result in the most common use case (i.e., iteration),
> but the Python3 version generates the elements only as needed.
>
> If you really *wanted* the list (but WHY?) in Python3, do
> list(range(...)), but then you get what you deserve. :-)
>
> Python3:
>
>  >>> l = range(int(1E9))
>  >>> l
> range(0, 1000000000)

This also leads to a unrelated question, Cecil.  Given that you really
are just starting to get your Python feet under you, why are you using
Python2?  Python3 is the standard now, Python2 is really just given
legacy support.  I'd understand if you were trying to maintain an old
codebase with lots of legacy code that was having "problematic"
migrations, but with the opportunity to start fresh?  Start fresh.  
You'll be happier for it.


--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order.  See above to fix.


Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

Ben Finney-10
In reply to this post by Jon Ribbens-5
Jon Ribbens <jon+usenet at unequivocal.co.uk> writes:

> On 2015-04-30, Cecil Westerhof <Cecil at decebal.nl> wrote:
> > If I execute:
> >     l = range(int(1E9)
> >
> > The python process gobbles up all the memory and is killed. [?] Is
> > there a way to circumvent Python claiming all the memory?

You seem to be asking for a way to stop a program doing exactly what
it's written to do. I don't know what kind of answer you expect.

> It's your operating system's job to handle processes.

Indeed. In this case, the program is written to gobble up memory, and
the operating system kills it. To ?circumvent? that behaviour surely
reveals the problem: that the operating system isn't handling processes
very well.

> > By the way: this is CPython 2.7.8.
>
> If you use xrange() instead of range() then you will get an iterator
> which will return each of the numbers in turn without any need to
> create an enormous list of all of them.

If you use Python 3 instead of the obsolescent Python 2, the ?range?
callable has this sensible behaviour by default.

--
 \       ?Corporation, n. An ingenious device for obtaining individual |
  `\       profit without individual responsibility.? ?Ambrose Bierce, |
_o__)                                   _The Devil's Dictionary_, 1906 |
Ben Finney



Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

Cecil Westerhof
In reply to this post by Grant Edwards-7
Op Thursday 30 Apr 2015 18:33 CEST schreef Grant Edwards:

> On 2015-04-30, Cecil Westerhof <Cecil at decebal.nl> wrote:
>> If I execute:
>> l = range(int(1E9)
>>
>> The python process gobbles up all the memory and is killed. The
>> problem is that after this my swap is completely used, because
>> other processes have swapped to it. This make those programs more
>> slowly. Is there a way to circumvent Python claiming all the
>> memory?
>
> I presume "don't do that" has already occured to you?

Well, it is just playing with the possibilities/limits of Python.
Better getting the problem now, then when I can not afford to lose
time. ;-)


> You can always use ulimit to limit the memory allowed for the
> process running Python.

That works, yes. Now I get a MemoryError and the other processes are
left alone. Now determining what are the best values.

--
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof


Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

Cecil Westerhof
In reply to this post by Jon Ribbens-5
Op Thursday 30 Apr 2015 18:55 CEST schreef Jon Ribbens:

> On 2015-04-30, Cecil Westerhof <Cecil at decebal.nl> wrote:
>> If I execute:
>> l = range(int(1E9)
>>
>> The python process gobbles up all the memory and is killed. The
>> problem is that after this my swap is completely used, because
>> other processes have swapped to it. This make those programs more
>> slowly. Is there a way to circumvent Python claiming all the
>> memory?
>>
>> By the way: this is CPython 2.7.8.
>
> It's your operating system's job to handle processes.
>
> If you use xrange() instead of range() then you will get an iterator
> which will return each of the numbers in turn without any need to
> create an enormous list of all of them.

I did it on purpose. Wanted to now the limits. Just need to use ulimit
to get a MemoryError.

--
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof


Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

Ben Finney-10
In reply to this post by Cecil Westerhof
Cecil Westerhof <Cecil at decebal.nl> writes:

> That works, yes. Now I get a MemoryError and the other processes are
> left alone. Now determining what are the best values.

I would strongly recommend that ?best values? includes ?run Python
version >= 3?.

One of the many problems you avoid by leaving Python 2 behind is that
many functions which used to return entire collections, now return lazy
evaluators (such as generators or views) which will not consume memory
the way you're describing.

Please try to learn Python using only the currently-developed Python 3.

--
 \     ?Listen: we are here on Earth to fart around. Don't let anybody |
  `\                  tell you otherwise.? ?_Timequake_, Kurt Vonnegut |
_o__)                                                                  |
Ben Finney



Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

Gisle Vanem-2
In reply to this post by Cecil Westerhof
Cecil Westerhof wrote:

> If I execute:
>      l = range(int(1E9)
>
> The python process gobbles up all the memory and is killed. The
> problem is that after this my swap is completely used, because other
> processes have swapped to it. This make those programs more slowly. Is
> there a way to circumvent Python claiming all the memory?
>
> By the way: this is CPython 2.7.8.

On what OS? If I try something similar on Win-8.1
and CPython 2.7.5 (32-bit):

  python -c "for i in range(int(1E9)): pass"
   Traceback (most recent call last):
     File "<string>", line 1, in <module>
   MemoryError


--gv


Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

Cecil Westerhof
In reply to this post by Cecil Westerhof
Op Thursday 30 Apr 2015 19:41 CEST schreef Ben Finney:

> Cecil Westerhof <Cecil at decebal.nl> writes:
>
>> That works, yes. Now I get a MemoryError and the other processes
>> are left alone. Now determining what are the best values.
>
> I would strongly recommend that ?best values? includes ?run Python
> version >= 3?.
>
> One of the many problems you avoid by leaving Python 2 behind is
> that many functions which used to return entire collections, now
> return lazy evaluators (such as generators or views) which will not
> consume memory the way you're describing.
>
> Please try to learn Python using only the currently-developed Python
> 3.

The problem with that is that in my neighbourhood 2 is still mostly
exclusively used. I want to learn also 3, but 2 has more priority.

The modules I publish on GitHub work with both. :-)

--
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof


Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

alister
In reply to this post by Cecil Westerhof
On Thu, 30 Apr 2015 20:23:31 +0200, Gisle Vanem wrote:

> Cecil Westerhof wrote:
>
>> If I execute:
>>      l = range(int(1E9)
>>
>> The python process gobbles up all the memory and is killed. The problem
>> is that after this my swap is completely used, because other processes
>> have swapped to it. This make those programs more slowly. Is there a
>> way to circumvent Python claiming all the memory?
>>
>> By the way: this is CPython 2.7.8.
>
> On what OS? If I try something similar on Win-8.1 and CPython 2.7.5
> (32-bit):
>
>   python -c "for i in range(int(1E9)): pass"
>    Traceback (most recent call last):
>      File "<string>", line 1, in <module>
>    MemoryError
>
>
> --gv

also MemoryError on Fedora 21 32 bit




--
I am a traffic light, and Alan Ginzberg kidnapped my laundry in 1927!


Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

Cecil Westerhof
In reply to this post by Rob Gaddi
Op Thursday 30 Apr 2015 19:12 CEST schreef Rob Gaddi:

> This also leads to a unrelated question, Cecil. Given that you
> really are just starting to get your Python feet under you, why are
> you using Python2? Python3 is the standard now, Python2 is really
> just given legacy support. I'd understand if you were trying to
> maintain an old codebase with lots of legacy code that was having
> "problematic" migrations, but with the opportunity to start fresh?
> Start fresh. You'll be happier for it.

I try to write code that works with both. Wen looking around in the
Netherlands there are more job opportunities with Python 2 as with
Python 3. So at the moment I find Python 2 more important, without
forgetting Python 3.

--
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof


Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

Dave Angel-4
In reply to this post by alister
On 04/30/2015 02:48 PM, alister wrote:

> On Thu, 30 Apr 2015 20:23:31 +0200, Gisle Vanem wrote:
>
>> Cecil Westerhof wrote:
>>
>>> If I execute:
>>>       l = range(int(1E9)
>>>
>>> The python process gobbles up all the memory and is killed. The problem
>>> is that after this my swap is completely used, because other processes
>>> have swapped to it. This make those programs more slowly. Is there a
>>> way to circumvent Python claiming all the memory?
>>>
>>> By the way: this is CPython 2.7.8.
>>
>> On what OS? If I try something similar on Win-8.1 and CPython 2.7.5
>> (32-bit):
>>
>>    python -c "for i in range(int(1E9)): pass"
>>     Traceback (most recent call last):
>>       File "<string>", line 1, in <module>
>>     MemoryError
>>
>>
>> --gv
>
> also MemoryError on Fedora 21 32 bit
>

That's presumably because you end up running out of address space before
you run out of swap space.  On a 64 bit system the reverse will be true,
unless you have a really *really* large swap file

ulimit is your friend if you've got a program that wants to gobble up
all of swap space.

--
DaveA


Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

Roel Schroeven-2
In reply to this post by Grant Edwards-7
Grant Edwards schreef op 2015-04-30 18:33:

> On 2015-04-30, Cecil Westerhof <Cecil at decebal.nl> wrote:
>> If I execute:
>>     l = range(int(1E9)
>>
>> The python process gobbles up all the memory and is killed. The
>> problem is that after this my swap is completely used, because other
>> processes have swapped to it. This make those programs more slowly.
>> Is there a way to circumvent Python claiming all the memory?
>
> I presume "don't do that" has already occured to you?

A few weeks ago, I had a bug in a Python script which caused it to
consume all memory, thrashed my computer; the system become unresponsive
  and it was very difficult to kill the process and get everything back
in working order.

Then I thought (foolishly) that the bug was fixed, so I ran the script
again, with the same disastrous results. And then again. Stupid, I know.

A way to limit memory usage would have been nice: the script would have
been killed before it could grind the whole system to a halt.

> You can always use ulimit to limit the memory allowed for the process
> running Python.

Sadly in my case the OS is Windows, and as far as I know it doesn't have
a ulimit equivalent for limiting memory usage.

I guess I could have used 32-bit Python instead of 64-bit Python to
limit available memory.


--
The saddest aspect of life right now is that science gathers knowledge
faster than society gathers wisdom.
   -- Isaac Asimov

Roel Schroeven



Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

Cecil Westerhof
In reply to this post by alister
Op Thursday 30 Apr 2015 20:59 CEST schreef Dave Angel:

> On 04/30/2015 02:48 PM, alister wrote:
>> On Thu, 30 Apr 2015 20:23:31 +0200, Gisle Vanem wrote:
>>
>>> Cecil Westerhof wrote:
>>>
>>>> If I execute:
>>>> l = range(int(1E9)
>>>>
>>>> The python process gobbles up all the memory and is killed. The
>>>> problem is that after this my swap is completely used, because
>>>> other processes have swapped to it. This make those programs more
>>>> slowly. Is there a way to circumvent Python claiming all the
>>>> memory?
>>>>
>>>> By the way: this is CPython 2.7.8.
>>>
>>> On what OS? If I try something similar on Win-8.1 and CPython
>>> 2.7.5 (32-bit):
>>>
>>> python -c "for i in range(int(1E9)): pass"
>>> Traceback (most recent call last):
>>> File "<string>", line 1, in <module>
>>> MemoryError
>>>
>>>
>>> --gv
>>
>> also MemoryError on Fedora 21 32 bit
>>
>
> That's presumably because you end up running out of address space
> before you run out of swap space. On a 64 bit system the reverse
> will be true, unless you have a really *really* large swap file
>
> ulimit is your friend if you've got a program that wants to gobble
> up all of swap space.

Yes, my system is openSUSE 64 bit. I really should look into ulimit.
The default is:
    core file size          (blocks, -c) 0
    data seg size           (kbytes, -d) unlimited
    scheduling priority             (-e) 0
    file size               (blocks, -f) unlimited
    pending signals                 (-i) 63768
    max locked memory       (kbytes, -l) 64
    max memory size         (kbytes, -m) unlimited
    open files                      (-n) 1024
    pipe size            (512 bytes, -p) 8
    POSIX message queues     (bytes, -q) 819200
    real-time priority              (-r) 0
    stack size              (kbytes, -s) 8192
    cpu time               (seconds, -t) unlimited
    max user processes              (-u) 63768
    virtual memory          (kbytes, -v) unlimited
    file locks                      (-x) unlimited

That should be fine-tuned.

--
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof


Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

Mark Lawrence
In reply to this post by Cecil Westerhof
On 30/04/2015 19:50, Cecil Westerhof wrote:

> Op Thursday 30 Apr 2015 19:12 CEST schreef Rob Gaddi:
>
>> This also leads to a unrelated question, Cecil. Given that you
>> really are just starting to get your Python feet under you, why are
>> you using Python2? Python3 is the standard now, Python2 is really
>> just given legacy support. I'd understand if you were trying to
>> maintain an old codebase with lots of legacy code that was having
>> "problematic" migrations, but with the opportunity to start fresh?
>> Start fresh. You'll be happier for it.
>
> I try to write code that works with both. Wen looking around in the
> Netherlands there are more job opportunities with Python 2 as with
> Python 3. So at the moment I find Python 2 more important, without
> forgetting Python 3.
>

You might find this useful then in you haven't already seen it
https://docs.python.org/3/howto/pyporting.html

I must also confess to being highly impressed, it's a breath of fresh
air having an apprentice Pythonista who is looking at doing things the
Pythonic way :)

--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence



Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

ElChino
In reply to this post by Cecil Westerhof
Mark Lawrence wrote:

> You might find this useful then in you haven't already seen it
> https://docs.python.org/3/howto/pyporting.html

The main reason I haven't switched to Python3 (from 2.7.4/MSVC),
is fear of a major breakage. How can I be certain that even if
I install to different directories, the Python2 and Python3 won't
step on each other toes. And what about all the .pyd I have around?
Do they need an update?

I'm barely familiar with virtualenv BTW; not sure another
confusing factor in the equation would be a good idea. Or
lessen my fear.




Reply | Threaded
Open this post in threaded view
|

l = range(int(1E9))

TIm Chase-3
In reply to this post by Cecil Westerhof
On 2015-04-30 22:18, Cecil Westerhof wrote:
> Op Thursday 30 Apr 2015 20:59 CEST schreef Dave Angel:
>> ulimit is your friend if you've got a program that wants to gobble
>> up all of swap space.
>
> Yes, my system is openSUSE 64 bit. I really should look into ulimit.
> The default is:
[snip]
>     max memory size         (kbytes, -m) unlimited

Note that AFAIK, "ulimit -m" doesn't work on Linux

http://unix.stackexchange.com/questions/129587/does-ulimit-m-not-work-on-modern-linux

Based on some quick testing[1], it doesn't appear to work on OpenBSD
or FreeBSD either.

-tkc


[1]
My test was to perform the following:

1) seq 20 > data.txt # to generate some random date

2) ed data.txt

3) find the PID of the "ed" process from another window using "ps ax
| grep ed"

4) make note of the current memory used for that PID using "top -p
$PID"

5) issue ",t$" in ed until the memory-used jumps up in the "top"
window (this copies the entire file to the bottom of the file,
doubling the size each time)

6) quit ed without saving ("Q")

7) issue "ulimit -m", specifying a threshold between the
before-the-memory-jump and after-the-memory-jump values

8) repeat steps 2-5 until the memory jumps up in top.  Notice that
it's possible to do this multiple times and exceed the "ulimit -m"
threshold, something the ulimit should prevent.




.


1234