What is considered an "advanced" topic in Python?

classic Classic list List threaded Threaded
73 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

Mike Driscoll
Hi,

I've been asked on several occasions to write about intermediate or advanced topics in Python and I was wondering what the community considers to be "intermediate" or "advanced". I realize we're all growing in our abilities with the language, so this is going to be very subjective, but I am still curious what my fellow Python developers think about this topic.

Thanks,
Mike


Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

Joel Goldstick-2
Maybe itertools or generators

On Fri, May 29, 2015 at 12:01 PM, Mike Driscoll <kyosohma at gmail.com> wrote:
> Hi,
>
> I've been asked on several occasions to write about intermediate or advanced topics in Python and I was wondering what the community considers to be "intermediate" or "advanced". I realize we're all growing in our abilities with the language, so this is going to be very subjective, but I am still curious what my fellow Python developers think about this topic.
>
> Thanks,
> Mike
> --
> https://mail.python.org/mailman/listinfo/python-list



--
Joel Goldstick
http://joelgoldstick.com


Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

Chris Angelico
In reply to this post by Mike Driscoll
On Sat, May 30, 2015 at 2:01 AM, Mike Driscoll <kyosohma at gmail.com> wrote:
> I've been asked on several occasions to write about intermediate or advanced topics in Python and I was wondering what the community considers to be "intermediate" or "advanced". I realize we're all growing in our abilities with the language, so this is going to be very subjective, but I am still curious what my fellow Python developers think about this topic.
>

Good fun! A few ideas:

How to write decorators, particularly those that take parameters.

The differences between the various number types (int, float, complex,
Fraction, Decimal) and when you'd want each one.

(CPython-specific) The dis.dis() function and what it can tell you
about how Python operates

These are all topics that have come up with my students; they're
advanced enough to be outside the scope of the course itself (the
course _uses_ decorators, but doesn't explain how to actually build
them), but not beyond the grasp of someone who's mastered Python's
fundamentals.

ChrisA


Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

Skip Montanaro
In reply to this post by Mike Driscoll
On Fri, May 29, 2015 at 11:01 AM, Mike Driscoll <kyosohma at gmail.com> wrote:
> ... I was wondering what the community considers to be "intermediate" or
"advanced".

Just about any topic on which Dave Beazley has given a keynote talk
<https://www.google.com/search?q=Dave+Beazley+keynote&oq=Dave+Beazley+keynote&aqs=chrome..69i57.5895j0j7&sourceid=chrome&es_sm=122&ie=UTF-8#q=Dave+Beazley+keynote&tbm=vid>.
:-)

Skip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20150529/77a18bb2/attachment.html>

Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

Chris Angelico
On Sat, May 30, 2015 at 2:39 AM, Skip Montanaro
<skip.montanaro at gmail.com> wrote:
> On Fri, May 29, 2015 at 11:01 AM, Mike Driscoll <kyosohma at gmail.com> wrote:
>> ... I was wondering what the community considers to be "intermediate" or
>> "advanced".
>
> Just about any topic on which Dave Beazley has given a keynote talk. :-)

Hehe. First hit is this one, with its awesome slide...

https://youtu.be/l_HBRhcgeuQ?t=3m34s

I showed that to my parents and siblings, most of whom are not
particularly adept with Python, and they found it highly amusing. As a
participant in python-ideas, I found it hilarious... and absolutely
right.

ChrisA


Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

Mike Driscoll
In reply to this post by Mike Driscoll
On Friday, May 29, 2015 at 11:08:19 AM UTC-5, Joel Goldstick wrote:
> Maybe itertools or generators
>

Yeah, I was thinking along those lines. I was also thinking about some of the cool stuff in the collections and contextlib modules.

Mike


Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

Mike Driscoll
In reply to this post by Mike Driscoll

>
> Good fun! A few ideas:
>
> How to write decorators, particularly those that take parameters.


Yes, this one always seems to trip people up.


>
> The differences between the various number types (int, float, complex,
> Fraction, Decimal) and when you'd want each one.

I hadn't considered this one


>
> (CPython-specific) The dis.dis() function and what it can tell you
> about how Python operates

I remember seeing on PyMOTW - http://pymotw.com/2/dis/  I agree that it's a module most programmers don't know about.


>
> These are all topics that have come up with my students; they're
> advanced enough to be outside the scope of the course itself (the
> course _uses_ decorators, but doesn't explain how to actually build
> them), but not beyond the grasp of someone who's mastered Python's
> fundamentals.
>
> ChrisA


Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

Mike Driscoll
In reply to this post by Mike Driscoll
On Friday, May 29, 2015 at 11:40:11 AM UTC-5, Skip Montanaro wrote:
> On Fri, May 29, 2015 at 11:01 AM, Mike Driscoll <kyos... at gmail.com> wrote:
> > ... I was wondering what the community considers to be "intermediate" or "advanced".
>
> Just about any topic on which Dave Beazley has given a keynote talk. :-)
>
>
> Skip

I've always enjoyed reading Beazley's blog...and his presentations are always neat too. Good idea.

Thanks,
Mike


Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

Todd-2
In reply to this post by Mike Driscoll
On Fri, May 29, 2015 at 6:01 PM, Mike Driscoll <kyosohma at gmail.com> wrote:

> Hi,
>
> I've been asked on several occasions to write about intermediate or
> advanced topics in Python and I was wondering what the community considers
> to be "intermediate" or "advanced". I realize we're all growing in our
> abilities with the language, so this is going to be very subjective, but I
> am still curious what my fellow Python developers think about this topic.
>
> Thanks,
> Mike
> --
> https://mail.python.org/mailman/listinfo/python-list
>

The ast module and AST hacking
The dist module and understanding the interpreter.
Code objects
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20150529/291d9111/attachment.html>

Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

sohcahtoa82@gmail.com
In reply to this post by Mike Driscoll
On Friday, May 29, 2015 at 9:02:06 AM UTC-7, Mike Driscoll wrote:
> Hi,
>
> I've been asked on several occasions to write about intermediate or advanced topics in Python and I was wondering what the community considers to be "intermediate" or "advanced". I realize we're all growing in our abilities with the language, so this is going to be very subjective, but I am still curious what my fellow Python developers think about this topic.
>
> Thanks,
> Mike

Metaclasses.

I've read about them.  I still don't understand them, why you would want them, and what you gain from them.


Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

Todd-2
In reply to this post by Todd-2
On Fri, May 29, 2015 at 7:02 PM, Todd <toddrjen at gmail.com> wrote:

> On Fri, May 29, 2015 at 6:01 PM, Mike Driscoll <kyosohma at gmail.com> wrote:
>
>> Hi,
>>
>> I've been asked on several occasions to write about intermediate or
>> advanced topics in Python and I was wondering what the community considers
>> to be "intermediate" or "advanced". I realize we're all growing in our
>> abilities with the language, so this is going to be very subjective, but I
>> am still curious what my fellow Python developers think about this topic.
>>
>> Thanks,
>> Mike
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>>
>
> The ast module and AST hacking
> The dist module and understanding the interpreter.
> Code objects
>

Sorry, miss-typed. I meant the "dis" module, not the "dist" module.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20150529/edcb3458/attachment.html>

Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

Chris Angelico
In reply to this post by Mike Driscoll
On Sat, May 30, 2015 at 2:57 AM, Mike Driscoll <kyosohma at gmail.com> wrote:
>>
>> Good fun! A few ideas:
>>
>> How to write decorators, particularly those that take parameters.
>
> Yes, this one always seems to trip people up.

It's like a Sherlock Holmes pronouncement. When you see something like
Flask's app.route(), or functools.wraps(), it's pure magic and
completely incomprehensible. But break it down into little pieces (a
function that takes a function and returns a function, then decorator
syntax, then closures and the ability to call the original, and
finally a decorator factory function, which is what a parameterized
decorator is), and it becomes elementary.

>> The differences between the various number types (int, float, complex,
>> Fraction, Decimal) and when you'd want each one.
>
> I hadn't considered this one

It's not often an actual *problem* - I've never seen anyone pick the
wrong data type and mess up their code, not in Python - but it's a
great way to explore some of the differences between real numbers and
what computers work with. Also, I like talking about Fraction and
Decimal for the simple reason that they're unobvious; you can poke
around with Python and discover int and float, and if ever you need
imaginary/complex numbers, you'll quickly come across complex, but you
might use Python for years and not realize that decimal.Decimal even
exists - nor when you'd want it.

ChrisA


Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

Ethan Furman-2
In reply to this post by sohcahtoa82@gmail.com
On 05/29/2015 10:03 AM, sohcahtoa82 at gmail.com wrote:
> On Friday, May 29, 2015 at 9:02:06 AM UTC-7, Mike Driscoll wrote:

>> I've been asked on several occasions to write about intermediate or advanced topics
>>  in Python and I was wondering what the community considers to be "intermediate" or
>>  "advanced". I realize we're all growing in our abilities with the language, so this
>>  is going to be very subjective, but I am still curious what my fellow Python
>>  developers think about this topic.
>
> Metaclasses.
>
> I've read about them.  I still don't understand them, why you would want them, and what you gain from them.

Metaclasses change the way a class behaves.

For example, the new (in 3.4) Enum class uses a metaclass.

   class SomeEnum(Enum):
      first = 1
      second = 2
      third = 3

The metaclass changes normal class behavior to:

   - support iterating: list(SomeEnum) --> [SomeEnum.first, SomeEnum.second, SomeEnum.third]
   - support a length:  len(SomeEnum) --> 3
   - not allow new instances to be created:  --> SomeEnum(1) is SomeEnum(1)  # True

--
~Ethan~


Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

Marko Rauhamaa
In reply to this post by sohcahtoa82@gmail.com
sohcahtoa82 at gmail.com:

> Metaclasses.
>
> I've read about them. I still don't understand them, why you would
> want them, and what you gain from them.

I don't think you would ever want them.


Marko


Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

Steven D'Aprano-8
In reply to this post by Mike Driscoll
On Sat, 30 May 2015 02:01 am, Mike Driscoll wrote:

> Hi,
>
> I've been asked on several occasions to write about intermediate or
> advanced topics in Python and I was wondering what the community considers
> to be "intermediate" or "advanced". I realize we're all growing in our
> abilities with the language, so this is going to be very subjective, but I
> am still curious what my fellow Python developers think about this topic.


I would consider these advanced topics:


Metaclasses.
Descriptors.
Futures.
Asynchronous processing, including multiprocessing and threads.
Writing C or Fortran extensions.
Function/code object internals.
Byte-code hacking.
Manipulating ASTs.
OS-specific features like the os.exec* functions, os.fork, daemons, etc.
Multiple inheritance, mixins, traits, etc.
Coroutines.
Dynamic programming.
Neural nets.
Distributed processing, including remote procedure calls.
Fuzz testing.


I would consider these intermediate topics:


Closures.
Using classes and functions as first-class values.
Second-order functions.
Decorators.
Functional idioms (map, filter, reduce).
Viewing byte-code using the dis module.
Unicode, code pages, codecs and encodings.
Regular expressions.
SQL and dealing with databases.
The complexities of floating point, including Decimal.
Writing your own context managers.
Generators and iterators.
Unit testing, doctests, etc.
Code generation and metaprogramming.
Writing your own classes.
State machines.
The structure of URLs (they're more than just "http://blahblah.com").
Anything to do with HTTP (client or server).
Commandline argument processing (argparse, optparse, etc.)

Some of the intermediate topics start at an intermediate level, but can go
on to include more advanced uses.



--
Steven



Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

Mike Driscoll
Hi Steven,

On Friday, May 29, 2015 at 12:55:48 PM UTC-5, Steven D'Aprano wrote:

> On Sat, 30 May 2015 02:01 am, Mike Driscoll wrote:
>
> > Hi,
> >
> > I've been asked on several occasions to write about intermediate or
> > advanced topics in Python and I was wondering what the community considers
> > to be "intermediate" or "advanced". I realize we're all growing in our
> > abilities with the language, so this is going to be very subjective, but I
> > am still curious what my fellow Python developers think about this topic.
>
>
> I would consider these advanced topics:
>
>
> Metaclasses.
> Descriptors.
> Futures.
> Asynchronous processing, including multiprocessing and threads.
> Writing C or Fortran extensions.
> Function/code object internals.
> Byte-code hacking.
> Manipulating ASTs.
> OS-specific features like the os.exec* functions, os.fork, daemons, etc.
> Multiple inheritance, mixins, traits, etc.
> Coroutines.
> Dynamic programming.
> Neural nets.
> Distributed processing, including remote procedure calls.
> Fuzz testing.
>
>
> I would consider these intermediate topics:
>
>
> Closures.
> Using classes and functions as first-class values.
> Second-order functions.
> Decorators.
> Functional idioms (map, filter, reduce).
> Viewing byte-code using the dis module.
> Unicode, code pages, codecs and encodings.
> Regular expressions.
> SQL and dealing with databases.
> The complexities of floating point, including Decimal.
> Writing your own context managers.
> Generators and iterators.
> Unit testing, doctests, etc.
> Code generation and metaprogramming.
> Writing your own classes.
> State machines.
> The structure of URLs (they're more than just "http://blahblah.com").
> Anything to do with HTTP (client or server).
> Commandline argument processing (argparse, optparse, etc.)
>
> Some of the intermediate topics start at an intermediate level, but can go
> on to include more advanced uses.
>
>
>
> --
> Steven

That's a really good list of topics. Thanks so much for sharing your ideas.

Mike


Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

sohcahtoa82@gmail.com
In reply to this post by sohcahtoa82@gmail.com
On Friday, May 29, 2015 at 10:18:29 AM UTC-7, Ethan Furman wrote:

> On 05/29/2015 10:03 AM, sohcahtoa82 at gmail.com wrote:
> > On Friday, May 29, 2015 at 9:02:06 AM UTC-7, Mike Driscoll wrote:
>
> >> I've been asked on several occasions to write about intermediate or advanced topics
> >>  in Python and I was wondering what the community considers to be "intermediate" or
> >>  "advanced". I realize we're all growing in our abilities with the language, so this
> >>  is going to be very subjective, but I am still curious what my fellow Python
> >>  developers think about this topic.
> >
> > Metaclasses.
> >
> > I've read about them.  I still don't understand them, why you would want them, and what you gain from them.
>
> Metaclasses change the way a class behaves.
>
> For example, the new (in 3.4) Enum class uses a metaclass.
>
>    class SomeEnum(Enum):
>       first = 1
>       second = 2
>       third = 3
>
> The metaclass changes normal class behavior to:
>
>    - support iterating: list(SomeEnum) --> [SomeEnum.first, SomeEnum.second, SomeEnum.third]
>    - support a length:  len(SomeEnum) --> 3
>    - not allow new instances to be created:  --> SomeEnum(1) is SomeEnum(1)  # True
>
> --
> ~Ethan~

Regarding the first two, you can implement __iter__ and __len__ functions to create that functionality, though those functions would operate on an instance of the class, not the class itself.

As for the third, can't you override the __new__ function to make attempts to create a new instance just return a previously created instance?


Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

Jason Swails
On Fri, May 29, 2015 at 5:06 PM, <sohcahtoa82 at gmail.com> wrote:

>
> > For example, the new (in 3.4) Enum class uses a metaclass.
> >
> >    class SomeEnum(Enum):
> >       first = 1
> >       second = 2
> >       third = 3
> >
> > The metaclass changes normal class behavior to:
> >
> >    - support iterating: list(SomeEnum) --> [SomeEnum.first,
> SomeEnum.second, SomeEnum.third]
> >    - support a length:  len(SomeEnum) --> 3
> >    - not allow new instances to be created:  --> SomeEnum(1) is
> SomeEnum(1)  # True
> >
> > --
> > ~Ethan~
>
> Regarding the first two, you can implement __iter__ and __len__ functions
> to create that functionality, though those functions would operate on an
> instance of the class, not the class itself.
>
> As for the third, can't you override the __new__ function to make attempts
> to create a new instance just return a previously created instance?
>

?Of course, but with metaclasses you don't *have* to (in fact, that's the
type of thing that the metaclass could do behind the scenes).

?In this case, any Enum subclass should *always* behave as Ethan described
-- without metaclasses that wouldn't necessarily be true (e.g., if the
person creating an Enum subclass didn't bother to correctly implement
__new__, __iter__, and __len__ for their subclass).

By using metaclasses, you can declare an Enum subclass as simply as Ethan
showed, since the metaclass does all of the dirty work implementing the
desired behavior at the time the class object is constructed (subclasses
inherit their parent class's metaclass).

In this use-case, you can think of it as a kind of "implicit class
factory".  Suppose you have a prescription for how you modify a class
definition (i.e., by implementing certain behavior in __new__ or __init__)
that you wrap up into some function "tweak_my_class".  The metaclass would
allow the class definition to be the equivalent of something like:

class MyClass(SubClass):
    # whatever

MyClass = tweak_my_class(MyClass)

Or as a class decorator

@tweak_my_class
class MyClass(SubClass):
    # whatever

(In fact, the `six` module allows you to implement metaclasses as
decorators to work with both Python 2 and Python 3, but I think metaclasses
are more powerful in Py3 than they are in Py2).?
?
They are cool ideas, and I've used them in my own code, but they do have a
kind of magic-ness to them -- especially in codes that you didn't write but
are working on.  As a result, I've recently started to prefer alternatives,
but in some rare cases (like Enum, for example), they are just the best
solution.

All the best,
Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20150529/07ef4598/attachment.html>

Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

Ethan Furman-2
In reply to this post by sohcahtoa82@gmail.com
On 05/29/2015 02:06 PM, sohcahtoa82 at gmail.com wrote:

> On Friday, May 29, 2015 at 10:18:29 AM UTC-7, Ethan Furman wrote:
>>
>> Metaclasses change the way a class behaves.
>>
>> For example, the new (in 3.4) Enum class uses a metaclass.
>>
>>     class SomeEnum(Enum):
>>        first = 1
>>        second = 2
>>        third = 3
>>
>> The metaclass changes normal class behavior to:
>>
>>     - support iterating: list(SomeEnum) --> [SomeEnum.first, SomeEnum.second, SomeEnum.third]
>>     - support a length:  len(SomeEnum) --> 3
>>     - not allow new instances to be created:  --> SomeEnum(1) is SomeEnum(1)  # True
>>
>> --
>> ~Ethan~
>
> Regarding the first two, you can implement __iter__ and __len__ functions to create that functionality, though those functions would operate on an instance of the class, not the class itself.

Hence the need for a metaclass, as the point is to operate on the class, not the instance.

> As for the third, can't you override the __new__ function to make attempts to create a new instance just return a previously created instance?

Yes.  In the case of Enum, however, it takes any user-defined __new__, which is needed for creating the original instances, and replaces it with a __new__ that only returns the already defined ones.

--
~Ethan~


Reply | Threaded
Open this post in threaded view
|

What is considered an "advanced" topic in Python?

Ethan Furman-2
In reply to this post by Jason Swails
On 05/29/2015 02:28 PM, Jason Swails wrote:

> ?[...] e.g., if the person creating an Enum subclass didn't bother to correctly
> implement [...] __iter__, and __len__ for their subclass

For __iter__ and __len__ to work on the Enum /class/ it must be defined in the metaclass.

--
~Ethan~


1234