Find in ipython3

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Find in ipython3

Cecil Westerhof
I am thinking about using ipython3 instead of bash. When I want to
find a file I can do the following:
    !find ~ -iname '*python*.pdf'
but is there a python way?

--
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

Reply | Threaded
Open this post in threaded view
|

Find in ipython3

Cameron Simpson
On 02Jun2015 18:13, Cecil Westerhof <Cecil at decebal.nl> wrote:
>I am thinking about using ipython3 instead of bash. When I want to
>find a file I can do the following:
>    !find ~ -iname '*python*.pdf'
>but is there a python way?

That succinct? Not out of the box, but something can easily be built on top of
the os.walk() function.

Cheers,
Cameron Simpson <cs at zip.com.au>

Perl combines all of the worst aspects of BASIC, C and line noise.
        - Keith Packard <keithp at ncd.com>

Reply | Threaded
Open this post in threaded view
|

Find in ipython3

Cecil Westerhof
In reply to this post by Cecil Westerhof
Op Thursday 4 Jun 2015 04:54 CEST schreef Cameron Simpson:

> On 02Jun2015 18:13, Cecil Westerhof <Cecil at decebal.nl> wrote:
>> I am thinking about using ipython3 instead of bash. When I want to
>> find a file I can do the following:
>> !find ~ -iname '*python*.pdf'
>> but is there a python way?
>
> That succinct? Not out of the box, but something can easily be built
> on top of the os.walk() function.

OK, I will write it. And something to implement du.

Thanks.

--
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

Reply | Threaded
Open this post in threaded view
|

Find in ipython3

Cameron Simpson
On 04Jun2015 07:09, Cecil Westerhof <Cecil at decebal.nl> wrote:

>Op Thursday 4 Jun 2015 04:54 CEST schreef Cameron Simpson:
>
>> On 02Jun2015 18:13, Cecil Westerhof <Cecil at decebal.nl> wrote:
>>> I am thinking about using ipython3 instead of bash. When I want to
>>> find a file I can do the following:
>>> !find ~ -iname '*python*.pdf'
>>> but is there a python way?
>>
>> That succinct? Not out of the box, but something can easily be built
>> on top of the os.walk() function.
>
>OK, I will write it. And something to implement du.
>Thanks.

Have fun.

For added points, do not forget that du knows about hardlinks and only counts a
file once.  And that it counts the blocks field, not the byte size (think:
sparse files).

Cheers,
Cameron Simpson <cs at zip.com.au>

Thermochromic ink is the Pet Rock ink of the New Millennium.
- overhead by WIRED at the Intelligent Printing conference Oct2006

Reply | Threaded
Open this post in threaded view
|

Find in ipython3

Grant Edwards-7
In reply to this post by Cecil Westerhof
On 2015-06-04, Cecil Westerhof <Cecil at decebal.nl> wrote:

> Op Thursday 4 Jun 2015 04:54 CEST schreef Cameron Simpson:
>
>> On 02Jun2015 18:13, Cecil Westerhof <Cecil at decebal.nl> wrote:
>>> I am thinking about using ipython3 instead of bash. When I want to
>>> find a file I can do the following:
>>> !find ~ -iname '*python*.pdf'
>>> but is there a python way?
>>
>> That succinct? Not out of the box, but something can easily be built
>> on top of the os.walk() function.
>
> OK, I will write it. And something to implement du.

Can't IPython just call the find and du utilities?

--
Grant Edwards               grant.b.edwards        Yow! Are we THERE yet?
                                  at              
                              gmail.com            

Reply | Threaded
Open this post in threaded view
|

Find in ipython3

Cecil Westerhof
Op Thursday 4 Jun 2015 16:27 CEST schreef Grant Edwards:

> On 2015-06-04, Cecil Westerhof <Cecil at decebal.nl> wrote:
>> Op Thursday 4 Jun 2015 04:54 CEST schreef Cameron Simpson:
>>
>>> On 02Jun2015 18:13, Cecil Westerhof <Cecil at decebal.nl> wrote:
>>>> I am thinking about using ipython3 instead of bash. When I want
>>>> to find a file I can do the following: !find ~ -iname
>>>> '*python*.pdf' but is there a python way?
>>>
>>> That succinct? Not out of the box, but something can easily be
>>> built on top of the os.walk() function.
>>
>> OK, I will write it. And something to implement du.
>
> Can't IPython just call the find and du utilities?

That is what
    !find ~ -iname '*python*.pdf'
does. But I do not find that aesthetically.

--
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

Reply | Threaded
Open this post in threaded view
|

Find in ipython3

Michael Torrie
In reply to this post by Cecil Westerhof
On 06/02/2015 10:13 AM, Cecil Westerhof wrote:
> I am thinking about using ipython3 instead of bash. When I want to
> find a file I can do the following:
>     !find ~ -iname '*python*.pdf'
> but is there a python way?

No more than there is a bash-native way of doing find.  Bash scripts use
a myriad of external, utility programs like find, cut, awk, sed, etc to
do heavy lifting.  The same things work just fine in iPython.  Why do
you need a "python way?"  Besides, Python isn't really a shell scripting
language, and ipython seems more like a hack to me.  Bash, Zsh, etc are
all better suited to actual shell scripting because their syntaxs
integrate the spawning external processes and building pipes in a nice
way (if you can call bash syntax nice).  That's not to say Python can't
be used for system programming. It certainly can.  And it can replace
perl easily at fast text processing.  Generator expressions are the
single most powerful tool in this.

Why not use Python for what it's good for and say pipe the results of
find into your python script?  Reinventing find poorly isn't going to
buy you anything.


Reply | Threaded
Open this post in threaded view
|

Find in ipython3

Michael Torrie
In reply to this post by Cecil Westerhof
On 06/04/2015 09:12 AM, Cecil Westerhof wrote:
>> Can't IPython just call the find and du utilities?
>
> That is what
>     !find ~ -iname '*python*.pdf'
> does. But I do not find that aesthetically.

Like I said, I find ipython to be hackish, but invoking find this way is
no more hackish than writing a 100 lines of python just to emulate it.
Using these existing external commands is more portable and more robust.

You could always write an ipython function that wraps the call to the
find command so you don't always have to see it.



Reply | Threaded
Open this post in threaded view
|

Find in ipython3

TIm Chase-3
In reply to this post by Michael Torrie
On 2015-06-04 13:09, Michael Torrie wrote:
> Why not use Python for what it's good for and say pipe the results
> of find into your python script?  Reinventing find poorly isn't
> going to buy you anything.

Until you port your app to Windows where find(1) is unavailable
natively ;-)

-tkc




Reply | Threaded
Open this post in threaded view
|

Find in ipython3

random832@fastmail.us
In reply to this post by Cecil Westerhof
On Tue, Jun 2, 2015, at 12:13, Cecil Westerhof wrote:
> I am thinking about using ipython3 instead of bash. When I want to
> find a file I can do the following:
>     !find ~ -iname '*python*.pdf'
> but is there a python way?

Python really isn't a good substitute for a shell, but the normal python
way to do this task is:

import os, os.path, fnmatch

home = os.path.expanduser('~')  # only needed since you used ~
for dirpath, dirnames, filenames in os.walk(home):
    print(dirpath)
    for filename in filenames:
        if(fnmatch.fnmatch(filename.lower(), '*python*.pdf')):
            print(os.path.join(dirpath, filename))

Note that if you have filenames with invalid unicode characters (or any
non-ASCII characters at all on Windows) you may have to do additional
processing to the filename before printing it. And of course instead of
printing it you may want to store the filenames in a list for further
processing. But these are the basic building blocks.

I don't use ipython, so I don't know what it provides if anything to
make any of this easier.

Reply | Threaded
Open this post in threaded view
|

Find in ipython3

Cameron Simpson
In reply to this post by Michael Torrie
On 04Jun2015 13:09, Michael Torrie <torriem at gmail.com> wrote:
>Why not use Python for what it's good for and say pipe the results of
>find into your python script?  Reinventing find poorly isn't going to
>buy you anything.

And several others made similar disparaging remarks. I think you're all missing
some of the point of Cecil's approach.

I suspect Cecil is a bit like me; he's writing various well understood simple
external tools in Python as an aid to learning Python and its toolbox. As an
added bonus one gets a deeper understanding of the nuances of the external
tools as well, by debugging issues in implementing them.

The point isn't always to make the shortest route to a result; Cecil clearly
already knows how to invoke the find command, for example. Sometimes the point
is to understand the path to the result in more detail.

That may not be Cecil's objective, but I've certainly benefitted from writing
my own version of things that already exist. Obviously you can't do that for
everything, life is too short, but you can do it for things of interest and as
a binus you can end up with a feature not available with the standard tool.

Cheers,
Cameron Simpson <cs at zip.com.au>

You should never bet against anything in science at odds of more than about
10^12 to 1.     - Ernest Rutherford

Reply | Threaded
Open this post in threaded view
|

Find in ipython3

Michael Torrie
On 06/04/2015 05:04 PM, Cameron Simpson wrote:
> On 04Jun2015 13:09, Michael Torrie <torriem at gmail.com> wrote:
>> Why not use Python for what it's good for and say pipe the results of
>> find into your python script?  Reinventing find poorly isn't going to
>> buy you anything.
>
> And several others made similar disparaging remarks. I think you're all missing
> some of the point of Cecil's approach.

I take your point.  However I was not intending to make a disparaging
remark and certainly didn't expect my post to be taken that way. I've
been down this road before (doing shell scripting things in Python), and
it works pretty well for many things.  Was just sharing my experience is
all.

I don't mean to discourage exploration for exploration's sake.  By all
means have fun. Python certainly is a fun language.

Reply | Threaded
Open this post in threaded view
|

Find in ipython3

Cameron Simpson
On 04Jun2015 20:23, Michael Torrie <torriem at gmail.com> wrote:

>On 06/04/2015 05:04 PM, Cameron Simpson wrote:
>> On 04Jun2015 13:09, Michael Torrie <torriem at gmail.com> wrote:
>>> Why not use Python for what it's good for and say pipe the results of
>>> find into your python script?  Reinventing find poorly isn't going to
>>> buy you anything.
>>
>> And several others made similar disparaging remarks. I think you're all missing
>> some of the point of Cecil's approach.
>
>I take your point.  However I was not intending to make a disparaging
>remark and certainly didn't expect my post to be taken that way.

And I should apologise for suggesting you were being disparaging to Cecil as
such. I simply felt that these comments (don't bother reinventing the wheel, in
various forms) gave a discouraging tone.

>I've
>been down this road before (doing shell scripting things in Python), and
>it works pretty well for many things.  Was just sharing my experience is
>all.
>I don't mean to discourage exploration for exploration's sake.  By all
>means have fun. Python certainly is a fun language.

Agreed.

Cheers,
Cameron Simpson <cs at zip.com.au>

Gabriel Genellina: See PEP 234 http://www.python.org/dev/peps/pep-0234/
Angus Rodgers:
  You've got to love a language whose documentation contains sentences
  beginning like this:
    "Among its chief virtues are the following four -- no, five -- no,
    six -- points: [...]"
from python-list at python.org

Reply | Threaded
Open this post in threaded view
|

Find in ipython3

Rustom Mody
In reply to this post by Michael Torrie
On Friday, June 5, 2015 at 9:59:22 AM UTC+5:30, Cameron Simpson wrote:

> On 04Jun2015 20:23, Michael Torrie  wrote:
> >On 06/04/2015 05:04 PM, Cameron Simpson wrote:
> >> On 04Jun2015 13:09, Michael Torrie  wrote:
> >>> Why not use Python for what it's good for and say pipe the results of
> >>> find into your python script?  Reinventing find poorly isn't going to
> >>> buy you anything.
> >>
> >> And several others made similar disparaging remarks. I think you're all missing
> >> some of the point of Cecil's approach.
> >
> >I take your point.  However I was not intending to make a disparaging
> >remark and certainly didn't expect my post to be taken that way.
>
> And I should apologise for suggesting you were being disparaging to Cecil as
> such. I simply felt that these comments (don't bother reinventing the wheel, in
> various forms) gave a discouraging tone.
>
> >I've
> >been down this road before (doing shell scripting things in Python), and
> >it works pretty well for many things.  Was just sharing my experience is
> >all.
> >I don't mean to discourage exploration for exploration's sake.  By all
> >means have fun. Python certainly is a fun language.

Chuck Moore the inventor of Forth and considered an all-time great among programmers
is in the opposite camp from the modern fad for "reuse done reinvent"
Cant find an original reference. Here is a quote:

| Chuck's philosophy is all about getting rid of everything that is not
| absolutely necessary for the task at hand, paring both the problem and
| solution down to the minimum. Compare this with the instinct that most
| programmers have to generalize every problem and abstract code into
| "reusable" modules, whether or not that code is ever actually reused. Chuck
| might say that the best way to write maintainable code is to keep the
| codebase small enough that it can be rewritten when the requirements change.
| To program in Forth you have to fight against instincts drawn in from other languages.

https://news.ycombinator.com/item?id=4623770

Reply | Threaded
Open this post in threaded view
|

Find in ipython3

Cecil Westerhof
In reply to this post by Cecil Westerhof
Op Thursday 4 Jun 2015 22:13 CEST schreef random:

> On Tue, Jun 2, 2015, at 12:13, Cecil Westerhof wrote:
>> I am thinking about using ipython3 instead of bash. When I want to
>> find a file I can do the following:
>> !find ~ -iname '*python*.pdf'
>> but is there a python way?
>
> Python really isn't a good substitute for a shell, but the normal
> python way to do this task is:
>
> import os, os.path, fnmatch
>
> home = os.path.expanduser('~')  # only needed since you used ~
> for dirpath, dirnames, filenames in os.walk(home):
> print(dirpath)
> for filename in filenames:
> if(fnmatch.fnmatch(filename.lower(), '*python*.pdf')):
> print(os.path.join(dirpath, filename))

I was already thinking along those lines. I made it:
    def find(directory, to_match):
        to_match = to_match.lower()
        results = []
        for dirpath, dirnames, filenames in os.walk(expanduser(directory)):
            for filename in filenames:
                if(fnmatch(filename.lower(), to_match)):
                    results.append(os.path.join(dirpath, filename))
        return results

> Note that if you have filenames with invalid unicode characters (or
> any non-ASCII characters at all on Windows) you may have to do
> additional processing to the filename before printing it. And of
> course instead of printing it you may want to store the filenames in
> a list for further processing. But these are the basic building
> blocks.

I have to look into it further. For one thing default the match should
be case dependent and an option used to make it independent.


> I don't use ipython, so I don't know what it provides if anything to
> make any of this easier.

I think it is useful to have it in Python also, so I should not use
ipython specific things.

In ?~/.ipython/profile_default/startup/00-init.ipy? I have:
    from utilDecebal import find

and now ?find('~', '*Python*.pdf')? gives what I want.

--
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

Reply | Threaded
Open this post in threaded view
|

Find in ipython3

Cecil Westerhof
On Friday  5 Jun 2015 09:17 CEST, Cecil Westerhof wrote:

> I was already thinking along those lines. I made it:
> def find(directory, to_match):
> to_match = to_match.lower()
> results = []
> for dirpath, dirnames, filenames in os.walk(expanduser(directory)):
> for filename in filenames:
> if(fnmatch(filename.lower(), to_match)):
> results.append(os.path.join(dirpath, filename))
> return results

I have a slightly better variant:
    def find(directory, to_match, ignore_case = False):
        to_match =  to_match + r'$'
        if ignore_case:
            p = re.compile(to_match, re.IGNORECASE)
        else:
            p = re.compile(to_match)
        results = []
        for dirpath, dirnames, filenames in os.walk(expanduser(directory)):
            for filename in filenames:
                if p.match(filename):
                    results.append(os.path.join(dirpath, filename))
        return results

Default it works now case sensitive. But I now use regular expression.
That is a lot more efficient. The old version took 4.4 seconds and
this version takes 2.4 seconds. But the ?!find? version takes about
half a second. Why is this version so much less efficient?

--
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

Reply | Threaded
Open this post in threaded view
|

Find in ipython3

Cecil Westerhof
In reply to this post by Michael Torrie
On Friday  5 Jun 2015 01:04 CEST, Cameron Simpson wrote:

> On 04Jun2015 13:09, Michael Torrie <torriem at gmail.com> wrote:
>> Why not use Python for what it's good for and say pipe the results
>> of find into your python script? Reinventing find poorly isn't
>> going to buy you anything.
>
> And several others made similar disparaging remarks. I think you're
> all missing some of the point of Cecil's approach.

My first experience with this newsgroup was very positive. Lately I
found some remarks not really encouraging (to use a British way of
talking). I am glad with this remark.


> I suspect Cecil is a bit like me; he's writing various well
> understood simple external tools in Python as an aid to learning
> Python and its toolbox. As an added bonus one gets a deeper
> understanding of the nuances of the external tools as well, by
> debugging issues in implementing them.

Yes, this is part of what I am doing: learning to work with Python.

But it is more. If I use shell functions like:
    !find ~ -iname '*python*.pdf'
then I have to switch between different way of working. Now I can use:
    find('~', '.*python.*.pdf', True)
so I do not have to switch my thinking.

And I can use:
    results = find('~', '.*python.*.pdf', True)
to do something with it.

At the moment the problem is that my Python variant takes 5 times as
much time as the shell version. Not a real big problem, but something
to think about.

And as someone said: this will also work in Windows. No plus for me,
but maybe for someone else.

--
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

Reply | Threaded
Open this post in threaded view
|

Find in ipython3

Laura Creighton-2
In reply to this post by Cecil Westerhof
The !find version is C code optimised to do one thing, find files in
your directory structure, which happens to be what you want to do.
General regular expression matching is harder.

Carl Friedrich Bolz investigated regular expression algorithms and their
implementation to see if this is the sort of task that a JIT can improve.
He blogged about it in 2 posts (part1 and part2).  There are benchmarks
for part2.  Benchmarks in part2.

see:
http://morepypy.blogspot.se/2010/05/efficient-and-elegant-regular.html
http://morepypy.blogspot.se/2010/06/jit-for-regular-expression-matching.html

You may get faster results if you use Matthew Barnett's replacement
for re here: https://pypi.python.org/pypi/regex

You will get faster results if you build your IPython shell to use PyPy,
but I would still be very surprised if it beat the C program find.

Laura


Reply | Threaded
Open this post in threaded view
|

Find in ipython3

Albert-jan Roskam
&quot;To run any command at the system shell, simply prefix it with !&quot;
See: https://ipython.org/ipython-doc/dev/interactive/tutorial.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20150606/6afec904/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Find in ipython3

Laura Creighton-2
In a message of Sat, 06 Jun 2015 15:42:27 -0700, Albert-Jan Roskam via Python-l
ist writes:
>&quot;To run any command at the system shell, simply prefix it with !&quot;
>See: https://ipython.org/ipython-doc/dev/interactive/tutorial.html

Please don't top post.

2. He knows this.  He's doing this for fun and to learn things, and for
some aesthetic reason.

12