Statically typing Python in Jython

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Statically typing Python in Jython

Jan Wroblewski
To Jython devs,

I am a 3rd year PhD student at University of Warsaw and I wanted to do
my Python static typing research by implementing it as a part of Jython.
Before starting, I would like to hear your opinion on combining my
research with Jython, and whether it would have a chance to be
integrated into the main Jython branch after it is complete. Below I
describe my plans.

My research
-----------
I want to statically type Python, which means constructing automatic
type inference of most objects (incl. functions and classes) present in
a Python program. The type system I am working on consists of:
- primitives,
- objects, i.e. dictionaries from strings known at compile time to types
of values contained there,
- functions,
- constructs like "TypeA or TypeB", "FunctionTypeA and FunctionTypeB".
In other words, before even running a program, I want to compute what
properties each variable present in it has, and recursively what are
properties of those properties, until I get to primitives or function
bodies.

The Python type inference algorithm I have in mind would take the Python
AST, convert it to a simpler Lucretia language and make the type
inference there. Lucretia is a language designed by Viviana Bono, Marcin
Benke and Aleksy Schubert (my PhD supervisor) for the purpose of static
typing of dynamic languages (pdf: https://arxiv.org/pdf/1206.5112.pdf).

One of the features of my project is that I would like to support full
Python language instead of subset of its features that are typeable
(like mypy does: http://mypy-lang.org/). Since it is provably impossible
to make type inference algorithm working for all Python programs, I want
to use gradual typing and treat all variables that my algorithm was not
able to process as "dynamic".

As for similar research projects, I'd say Reticulated Python by Michael
M. Vitousek, Andrew M. Kent, Jeremy G. Siek and Jim Baker is the closest
one to what I would like to achieve (pdf:
http://wphomes.soic.indiana.edu/jsiek/files/2014/03/retic-python.pdf).
Reticulated Python also does not seem to restrict allowed language
constructs, but it has slightly different type system and different
policy on handling imports (kind of at runtime instead of statically). I
noticed that Jim Baker is not only sub-author of Reticulated Python
paper, but also a major commiter to Jython, so I am interested in
hearing if some of the work from Reticulated Python is already
integrated into Jython.

Combining with Jython
---------------------
The type inference in Lucretia and combining its results with Python is
something new and I would like to implement it as a part of Jython. The
current Jython implementation would help me with parsing, execution, and
generally with lots of technical stuff. I was originally going to target
only recent versions of Python 3, but I am open for changes, even to
Python 2.7. If the general idea is welcome, I am also open to other
major changes.

I think that there could be two main applications of my research:
- Improvement of IDEs - after exposing some Jython internals connected
to my work, more precise type information could be made available to an
IDE. As many IDEs run on JVM, the integration could also be smoother.
- Producing faster, more optimized for JVM bytecode - although it is not
the main goal of Jython, more speed is always better. I believe I could
do that in situations where I could prove that no properties would be
ever dynamically added to an object, existing ones would not change
their type and I could resolve statically all called methods.

Jan Wroblewski
http://www.mimuw.edu.pl/~xi/

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev
Reply | Threaded
Open this post in threaded view
|

Re: Statically typing Python in Jython

Stefan Richthofer
Hello Jan,

just some questions/remarks from me for now.


PEP 484

In your whole email you didn't mention PEP 484 for a single time.
Will your work incorporate this official Python-typing?
(i.e. given that Jython 3 is still in work, this mainly refers to https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code)
In case you are interested:
I created a Python 2.7-workable version of typing.get_type_hints() (the original from typing-module only works for Python 3 and plainly returns None on Python 2.)
Unfortunately it currently doesn't work on Jython due to bugs.jython.org/issue2515. Anyway, that issue would need a fix for any PEP 484-related work on Jython.


Java-typing

How would your approach work with Java-typing behind the scenes (e.g. for mixed Java/Python programs)? Whenever Jython imports a Java-lib, on Java-site there is more Type-info available than the import mechanism exposes. Also, it somewhat obfuscates that info due to coercion and so. Would you also incorporate Java-type info?
What about Java-Generics? The Generics-info can be retrieved from Java too, it's a bit constrained and tricky though.

I always wanted to do some type-cleanup in Jython in the sense that Jython's Java-API could often be more precise regarding type-info. E.g. it often just returns a PyObject, even when it is clear that actually a PyTuple, a PyList<PyObject> or at least List<PyObject> etc is returned.
However, so far I lacked time and motivation to actually start this. Also not sure if this could break code, but since the types would only be specialized, everything should stay compatible.
Anyway, maybe this is more something for Jython 3. Would it benefit your approach?


Including into Jython

If your work goes well, I'd be in principle +1 on including it into Jython. Especially if it would help to make the Java/Python cross-connection more typesafe, especially when calling Python-code from Java. However, let's see what the others say; I guess it will depend on the actual benefits and features, which are currently formulated a bit vaguely I guess.


Best

Stefan



> Gesendet: Sonntag, 06. November 2016 um 15:21 Uhr
> Von: "Jan Wroblewski" <[hidden email]>
> An: [hidden email]
> Betreff: [Jython-dev] Statically typing Python in Jython
>
> To Jython devs,
>
> I am a 3rd year PhD student at University of Warsaw and I wanted to do
> my Python static typing research by implementing it as a part of Jython.
> Before starting, I would like to hear your opinion on combining my
> research with Jython, and whether it would have a chance to be
> integrated into the main Jython branch after it is complete. Below I
> describe my plans.
>
> My research
> -----------
> I want to statically type Python, which means constructing automatic
> type inference of most objects (incl. functions and classes) present in
> a Python program. The type system I am working on consists of:
> - primitives,
> - objects, i.e. dictionaries from strings known at compile time to types
> of values contained there,
> - functions,
> - constructs like "TypeA or TypeB", "FunctionTypeA and FunctionTypeB".
> In other words, before even running a program, I want to compute what
> properties each variable present in it has, and recursively what are
> properties of those properties, until I get to primitives or function
> bodies.
>
> The Python type inference algorithm I have in mind would take the Python
> AST, convert it to a simpler Lucretia language and make the type
> inference there. Lucretia is a language designed by Viviana Bono, Marcin
> Benke and Aleksy Schubert (my PhD supervisor) for the purpose of static
> typing of dynamic languages (pdf: https://arxiv.org/pdf/1206.5112.pdf).
>
> One of the features of my project is that I would like to support full
> Python language instead of subset of its features that are typeable
> (like mypy does: http://mypy-lang.org/). Since it is provably impossible
> to make type inference algorithm working for all Python programs, I want
> to use gradual typing and treat all variables that my algorithm was not
> able to process as "dynamic".
>
> As for similar research projects, I'd say Reticulated Python by Michael
> M. Vitousek, Andrew M. Kent, Jeremy G. Siek and Jim Baker is the closest
> one to what I would like to achieve (pdf:
> http://wphomes.soic.indiana.edu/jsiek/files/2014/03/retic-python.pdf).
> Reticulated Python also does not seem to restrict allowed language
> constructs, but it has slightly different type system and different
> policy on handling imports (kind of at runtime instead of statically). I
> noticed that Jim Baker is not only sub-author of Reticulated Python
> paper, but also a major commiter to Jython, so I am interested in
> hearing if some of the work from Reticulated Python is already
> integrated into Jython.
>
> Combining with Jython
> ---------------------
> The type inference in Lucretia and combining its results with Python is
> something new and I would like to implement it as a part of Jython. The
> current Jython implementation would help me with parsing, execution, and
> generally with lots of technical stuff. I was originally going to target
> only recent versions of Python 3, but I am open for changes, even to
> Python 2.7. If the general idea is welcome, I am also open to other
> major changes.
>
> I think that there could be two main applications of my research:
> - Improvement of IDEs - after exposing some Jython internals connected
> to my work, more precise type information could be made available to an
> IDE. As many IDEs run on JVM, the integration could also be smoother.
> - Producing faster, more optimized for JVM bytecode - although it is not
> the main goal of Jython, more speed is always better. I believe I could
> do that in situations where I could prove that no properties would be
> ever dynamically added to an object, existing ones would not change
> their type and I could resolve statically all called methods.
>
> Jan Wroblewski
> http://www.mimuw.edu.pl/~xi/
>
> ------------------------------------------------------------------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today. http://sdm.link/xeonphi
> _______________________________________________
> Jython-dev mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/jython-dev
>

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev
Reply | Threaded
Open this post in threaded view
|

Re: Statically typing Python in Jython

Jeff Allen-2
Hello Jan,

this sounds a very interesting project. I have given a little thought to
type inference as an optimisation technique, walking the AST to infer
type. In that context it seems more helpful to infer the implementation
type (and range), rather than the Python type. This seems to be what
Numba does: I know that the Numba team find the type hints are not fine
enough for their purposes: they want to know how wide the float is, how
big the int.

I'm not hopeful it could be incorporated into Jython (or CPython)
without implementing the core entirely differently. There seems to be so
much of the runtime where actual type is washed away to PyObject. It's
not just a matter of providing a more precise return type for library
methods. All the abstractions work against you: even adding two integers
has to return a general object because the slot (equivalently Jython's
PyObject._add) has to accommodate the possibility of returning
NotImplemented.

Jeff Allen

On 06/11/2016 18:31, Stefan Richthofer wrote:

> Hello Jan,
>
> just some questions/remarks from me for now.
>
>
> PEP 484
>
> In your whole email you didn't mention PEP 484 for a single time.
> Will your work incorporate this official Python-typing?
> (i.e. given that Jython 3 is still in work, this mainly refers to https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code)
> In case you are interested:
> I created a Python 2.7-workable version of typing.get_type_hints() (the original from typing-module only works for Python 3 and plainly returns None on Python 2.)
> Unfortunately it currently doesn't work on Jython due to bugs.jython.org/issue2515. Anyway, that issue would need a fix for any PEP 484-related work on Jython.
>
>
> Java-typing
>
> How would your approach work with Java-typing behind the scenes (e.g. for mixed Java/Python programs)? Whenever Jython imports a Java-lib, on Java-site there is more Type-info available than the import mechanism exposes. Also, it somewhat obfuscates that info due to coercion and so. Would you also incorporate Java-type info?
> What about Java-Generics? The Generics-info can be retrieved from Java too, it's a bit constrained and tricky though.
>
> I always wanted to do some type-cleanup in Jython in the sense that Jython's Java-API could often be more precise regarding type-info. E.g. it often just returns a PyObject, even when it is clear that actually a PyTuple, a PyList<PyObject> or at least List<PyObject> etc is returned.
> However, so far I lacked time and motivation to actually start this. Also not sure if this could break code, but since the types would only be specialized, everything should stay compatible.
> Anyway, maybe this is more something for Jython 3. Would it benefit your approach?
>
>
> Including into Jython
>
> If your work goes well, I'd be in principle +1 on including it into Jython. Especially if it would help to make the Java/Python cross-connection more typesafe, especially when calling Python-code from Java. However, let's see what the others say; I guess it will depend on the actual benefits and features, which are currently formulated a bit vaguely I guess.
>
>
> Best
>
> Stefan
>
>
>
>> Gesendet: Sonntag, 06. November 2016 um 15:21 Uhr
>> Von: "Jan Wroblewski" <[hidden email]>
>> An: [hidden email]
>> Betreff: [Jython-dev] Statically typing Python in Jython
>>
>> To Jython devs,
>>
>> I am a 3rd year PhD student at University of Warsaw and I wanted to do
>> my Python static typing research by implementing it as a part of Jython.
>> Before starting, I would like to hear your opinion on combining my
>> research with Jython, and whether it would have a chance to be
>> integrated into the main Jython branch after it is complete. Below I
>> describe my plans.
>>
>> My research
>> -----------
>> I want to statically type Python, which means constructing automatic
>> type inference of most objects (incl. functions and classes) present in
>> a Python program. The type system I am working on consists of:
>> - primitives,
>> - objects, i.e. dictionaries from strings known at compile time to types
>> of values contained there,
>> - functions,
>> - constructs like "TypeA or TypeB", "FunctionTypeA and FunctionTypeB".
>> In other words, before even running a program, I want to compute what
>> properties each variable present in it has, and recursively what are
>> properties of those properties, until I get to primitives or function
>> bodies.
>>
>> The Python type inference algorithm I have in mind would take the Python
>> AST, convert it to a simpler Lucretia language and make the type
>> inference there. Lucretia is a language designed by Viviana Bono, Marcin
>> Benke and Aleksy Schubert (my PhD supervisor) for the purpose of static
>> typing of dynamic languages (pdf: https://arxiv.org/pdf/1206.5112.pdf).
>>
>> One of the features of my project is that I would like to support full
>> Python language instead of subset of its features that are typeable
>> (like mypy does: http://mypy-lang.org/). Since it is provably impossible
>> to make type inference algorithm working for all Python programs, I want
>> to use gradual typing and treat all variables that my algorithm was not
>> able to process as "dynamic".
>>
>> As for similar research projects, I'd say Reticulated Python by Michael
>> M. Vitousek, Andrew M. Kent, Jeremy G. Siek and Jim Baker is the closest
>> one to what I would like to achieve (pdf:
>> http://wphomes.soic.indiana.edu/jsiek/files/2014/03/retic-python.pdf).
>> Reticulated Python also does not seem to restrict allowed language
>> constructs, but it has slightly different type system and different
>> policy on handling imports (kind of at runtime instead of statically). I
>> noticed that Jim Baker is not only sub-author of Reticulated Python
>> paper, but also a major commiter to Jython, so I am interested in
>> hearing if some of the work from Reticulated Python is already
>> integrated into Jython.
>>
>> Combining with Jython
>> ---------------------
>> The type inference in Lucretia and combining its results with Python is
>> something new and I would like to implement it as a part of Jython. The
>> current Jython implementation would help me with parsing, execution, and
>> generally with lots of technical stuff. I was originally going to target
>> only recent versions of Python 3, but I am open for changes, even to
>> Python 2.7. If the general idea is welcome, I am also open to other
>> major changes.
>>
>> I think that there could be two main applications of my research:
>> - Improvement of IDEs - after exposing some Jython internals connected
>> to my work, more precise type information could be made available to an
>> IDE. As many IDEs run on JVM, the integration could also be smoother.
>> - Producing faster, more optimized for JVM bytecode - although it is not
>> the main goal of Jython, more speed is always better. I believe I could
>> do that in situations where I could prove that no properties would be
>> ever dynamically added to an object, existing ones would not change
>> their type and I could resolve statically all called methods.
>>
>> Jan Wroblewski
>> http://www.mimuw.edu.pl/~xi/
>>
>> ------------------------------------------------------------------------------
>> Developer Access Program for Intel Xeon Phi Processors
>> Access to Intel Xeon Phi processor-based developer platforms.
>> With one year of Intel Parallel Studio XE.
>> Training and support from Colfax.
>> Order your platform today. http://sdm.link/xeonphi
>> _______________________________________________
>> Jython-dev mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/jython-dev
>>
> ------------------------------------------------------------------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today. http://sdm.link/xeonphi
> _______________________________________________
> Jython-dev mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/jython-dev
>


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev
Reply | Threaded
Open this post in threaded view
|

Re: Statically typing Python in Jython

John Hayes

Hi everyone,

I haven't followed all the subtleties of this discussion, but my naive impression is that statically typed Python variables have been dealt with, in a way, in Cython ( http://cython.org/) to enhance Python performance otherwise. It is incompatible with Jython itself per se AFAIK, but quite interesting in a general Pythonic sense IMO.

Cheers,
John


On Nov 6, 2016 6:20 PM, "Jeff Allen" <[hidden email]> wrote:
Hello Jan,

this sounds a very interesting project. I have given a little thought to
type inference as an optimisation technique, walking the AST to infer
type. In that context it seems more helpful to infer the implementation
type (and range), rather than the Python type. This seems to be what
Numba does: I know that the Numba team find the type hints are not fine
enough for their purposes: they want to know how wide the float is, how
big the int.

I'm not hopeful it could be incorporated into Jython (or CPython)
without implementing the core entirely differently. There seems to be so
much of the runtime where actual type is washed away to PyObject. It's
not just a matter of providing a more precise return type for library
methods. All the abstractions work against you: even adding two integers
has to return a general object because the slot (equivalently Jython's
PyObject._add) has to accommodate the possibility of returning
NotImplemented.

Jeff Allen

On 06/11/2016 18:31, Stefan Richthofer wrote:
> Hello Jan,
>
> just some questions/remarks from me for now.
>
>
> PEP 484
>
> In your whole email you didn't mention PEP 484 for a single time.
> Will your work incorporate this official Python-typing?
> (i.e. given that Jython 3 is still in work, this mainly refers to https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code)
> In case you are interested:
> I created a Python 2.7-workable version of typing.get_type_hints() (the original from typing-module only works for Python 3 and plainly returns None on Python 2.)
> Unfortunately it currently doesn't work on Jython due to bugs.jython.org/issue2515. Anyway, that issue would need a fix for any PEP 484-related work on Jython.
>
>
> Java-typing
>
> How would your approach work with Java-typing behind the scenes (e.g. for mixed Java/Python programs)? Whenever Jython imports a Java-lib, on Java-site there is more Type-info available than the import mechanism exposes. Also, it somewhat obfuscates that info due to coercion and so. Would you also incorporate Java-type info?
> What about Java-Generics? The Generics-info can be retrieved from Java too, it's a bit constrained and tricky though.
>
> I always wanted to do some type-cleanup in Jython in the sense that Jython's Java-API could often be more precise regarding type-info. E.g. it often just returns a PyObject, even when it is clear that actually a PyTuple, a PyList<PyObject> or at least List<PyObject> etc is returned.
> However, so far I lacked time and motivation to actually start this. Also not sure if this could break code, but since the types would only be specialized, everything should stay compatible.
> Anyway, maybe this is more something for Jython 3. Would it benefit your approach?
>
>
> Including into Jython
>
> If your work goes well, I'd be in principle +1 on including it into Jython. Especially if it would help to make the Java/Python cross-connection more typesafe, especially when calling Python-code from Java. However, let's see what the others say; I guess it will depend on the actual benefits and features, which are currently formulated a bit vaguely I guess.
>
>
> Best
>
> Stefan
>
>
>
>> Gesendet: Sonntag, 06. November 2016 um 15:21 Uhr
>> Von: "Jan Wroblewski" <[hidden email]>
>> An: [hidden email]
>> Betreff: [Jython-dev] Statically typing Python in Jython
>>
>> To Jython devs,
>>
>> I am a 3rd year PhD student at University of Warsaw and I wanted to do
>> my Python static typing research by implementing it as a part of Jython.
>> Before starting, I would like to hear your opinion on combining my
>> research with Jython, and whether it would have a chance to be
>> integrated into the main Jython branch after it is complete. Below I
>> describe my plans.
>>
>> My research
>> -----------
>> I want to statically type Python, which means constructing automatic
>> type inference of most objects (incl. functions and classes) present in
>> a Python program. The type system I am working on consists of:
>> - primitives,
>> - objects, i.e. dictionaries from strings known at compile time to types
>> of values contained there,
>> - functions,
>> - constructs like "TypeA or TypeB", "FunctionTypeA and FunctionTypeB".
>> In other words, before even running a program, I want to compute what
>> properties each variable present in it has, and recursively what are
>> properties of those properties, until I get to primitives or function
>> bodies.
>>
>> The Python type inference algorithm I have in mind would take the Python
>> AST, convert it to a simpler Lucretia language and make the type
>> inference there. Lucretia is a language designed by Viviana Bono, Marcin
>> Benke and Aleksy Schubert (my PhD supervisor) for the purpose of static
>> typing of dynamic languages (pdf: https://arxiv.org/pdf/1206.5112.pdf).
>>
>> One of the features of my project is that I would like to support full
>> Python language instead of subset of its features that are typeable
>> (like mypy does: http://mypy-lang.org/). Since it is provably impossible
>> to make type inference algorithm working for all Python programs, I want
>> to use gradual typing and treat all variables that my algorithm was not
>> able to process as "dynamic".
>>
>> As for similar research projects, I'd say Reticulated Python by Michael
>> M. Vitousek, Andrew M. Kent, Jeremy G. Siek and Jim Baker is the closest
>> one to what I would like to achieve (pdf:
>> http://wphomes.soic.indiana.edu/jsiek/files/2014/03/retic-python.pdf).
>> Reticulated Python also does not seem to restrict allowed language
>> constructs, but it has slightly different type system and different
>> policy on handling imports (kind of at runtime instead of statically). I
>> noticed that Jim Baker is not only sub-author of Reticulated Python
>> paper, but also a major commiter to Jython, so I am interested in
>> hearing if some of the work from Reticulated Python is already
>> integrated into Jython.
>>
>> Combining with Jython
>> ---------------------
>> The type inference in Lucretia and combining its results with Python is
>> something new and I would like to implement it as a part of Jython. The
>> current Jython implementation would help me with parsing, execution, and
>> generally with lots of technical stuff. I was originally going to target
>> only recent versions of Python 3, but I am open for changes, even to
>> Python 2.7. If the general idea is welcome, I am also open to other
>> major changes.
>>
>> I think that there could be two main applications of my research:
>> - Improvement of IDEs - after exposing some Jython internals connected
>> to my work, more precise type information could be made available to an
>> IDE. As many IDEs run on JVM, the integration could also be smoother.
>> - Producing faster, more optimized for JVM bytecode - although it is not
>> the main goal of Jython, more speed is always better. I believe I could
>> do that in situations where I could prove that no properties would be
>> ever dynamically added to an object, existing ones would not change
>> their type and I could resolve statically all called methods.
>>
>> Jan Wroblewski
>> http://www.mimuw.edu.pl/~xi/
>>
>> ------------------------------------------------------------------------------
>> Developer Access Program for Intel Xeon Phi Processors
>> Access to Intel Xeon Phi processor-based developer platforms.
>> With one year of Intel Parallel Studio XE.
>> Training and support from Colfax.
>> Order your platform today. http://sdm.link/xeonphi
>> _______________________________________________
>> Jython-dev mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/jython-dev
>>
> ------------------------------------------------------------------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today. http://sdm.link/xeonphi
> _______________________________________________
> Jython-dev mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/jython-dev
>


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev
Reply | Threaded
Open this post in threaded view
|

Re: Statically typing Python in Jython

Jan Wroblewski
Hi everyone,

Thank you for many responses. I'll address each of them below.

@Stefan Richthofer

PEP 484
-------
If it will be possible, I do plan to use typing syntax provided by PEP
484. I will do that if the outcome of this discussion will be that it's
preferable to work on Python 3.

PEP 484 provides some nice syntax for typing, but keep in mind that it
is largely influenced by the mypy project (http://mypy-lang.org/) and
shares some of its ideas/expressiveness. It is organized around classes
with a constant set of attributes. I find this very natural from the
developer point of view, but it causes some problems if you want to know
the exact representation of the object. Consider the following example:

class Y:
     def __init__(self, p):
         if p == 0:
             self.x = 42
         else:
             self.y = 42

def f(a: Y) -> None:
     a.x = a.x * 2

In function f, we use the x property of the argument, which is
initialized only in some cases in class Y. The above code does not
detect any errors when checking it with mypy, because class Y is defined
as object potentially having properties x and y. In order to make the
definition of f closer to actual implementation, we could write it in a
different way (the example below does not use PEP 484 notation):

def f(a: {"x": int}) -> None:
     a.x = a.x * 2

This version gives enough information to prove correctness of the
executed code. On the other hand, if Y always defined x and we could
prove that all instances of Y contain an integer x, the first typing
would also be "correct". There are times where simple class-based
descriptions are insufficient and we need to know the precise structure
of objects. I'd say that this happens when classes are not written like
in a standard statically typed language, e.g. Java.

The direction I am going in is less about making clear module interfaces
and more about program verification. The type system I am proposing is
based on objects being dictionaries from property names to types
(primitives, function bodies or, recursively, object types). My approach
is closer to how Python objects are actually represented and therefore
does not restrict allowed constructions at the level of type definitions.

In order to make the code as readable as possible, but at the same time
provide enough information for program verification, I'd like to use
most of PEP 484-like class-based types as hints or constraints for the
actual type inference, but mostly work with very precise information
about objects. In other words, I would like to combine PEP 484
class-based notation with a property-based notation that requires
properties of arguments to be of specific types instead.

Java-typing
-----------
My main focus lies in static type analysis in Python, so I did not have
any specific plans regarding using type information for different
languages. I would like to focus on pure Python first, however here are
some of my thoughts.

The fact that Java classes are immutable (at least as long as their
ClassLoader is not garbage collected) gives me a very valuable
information not present in Python: I know for sure that the set of
attributes of a class will stay constant and, if the bytecode is
correct, getfield will not produce an AttributeError-like error. I do
not think there should be any bigger problem representing a Java class
in my type system, aside from nulls. For example, let us take a look at
such a Java class:

public class LinkedListNode {
     public LinkedListNode next;
     public int value;
}

 From the bytecode I could say that it's recursive type is:

linkedListNodeType == { "next": Or(linkedListNodeType, javaNullType),
"value": int }

There would be a problem with nulls for non-private attributes, since
this would require analysis of the Java bytecode to check if they are
nullable. I did a similar kind of analysis this year for Java bytecode
and I believe proving nullability is pretty much impossible without an
input from the developer or making some assumptions (no reflection, not
loading new classes that operate on our object etc.).

I am yet to design any sort of built-in containers and polymorphism in
my type system, so I can not say for sure about handling Java generics.
Probably it would be beneficial for Jython if I could type Java and
Python collections, so that their types are as interchangeable in the
type system, as they are during the execution. Since I will have to type
built-in Python collections anyway, I will have a way to make Java
versions of these types. However, I think it will be pretty messy, since
it will essentially require me to manually create a rule for type
inference for each supported Java collection.

Type safety when running Python from Java
-----------------------------------------
I believe the work with Java API you are mentioning does not have that
much in common with what I want to do. I want to take Python classes
written by a developer and get a lot of type information about them. To
Java, this information could be made available as a converted class
bytecode that uses more specific types or through a new API. However,
this is orthogonal to any generic API, since each user program would
have different types.

You also mentioned later that you'd like to see Java-Python connection
more type-safe. I believe this could be possible, but it would become
more complex for users if they were given objects of very specific
user-defined types. Currently, I do not know enough about Jython
integration with Java to discuss this. I just want to mention that even
if I managed to compute precise type information of returned object, I
would not be able to give it to a user though a generic API. At least
not without something like first converting the whole Python project
into Java class files.

Fully converted Python class
----------------------------
As you are interested in integration of Python with Java, let me give
one of the ideas I would like to apply in later project stages to make
the code faster.

Under conditions that some restrictive Python type systems assume (such
as immutable classes), I could compile Python classes to Java classes.
Those conditions would be very restrictive, as they would require not
using many dynamic aspects of the language, but I believe it could speed
up simple, computation-intensive methods. More specifically, I think it
would be great to be able to produce a Java class like that:

public class Person {
     private String surname;
     private String name;

     public Person(String s, String n) {
         surname = s;
         name = n;
     }

     public String fullName() {
         return name + " " + surname;
     }
}

from a Python class like that:

class Person:
     def __init__(self, s, n):
         self.surname = s
         self.name = n

     def fullName(self):
         return self.name + " " + self.surname

Keep in mind that even this example would require a lot of type
information to be successfully inferred and probably performing a global
analysis using all Python code used by the project. The largest upside
of this would be removing the overhead of keeping the object as a
dictionary, getting the computations closer to bare metal after JIT
optimizations in some cases. I can see a lot of problems with that, such
as completely different API (native Java classes?) in case we want this
object to be integrated with Java. Still, I wanted to share that kind of
idea early to get feedback if it aligns with the goals of the Jython
project.

Jython 3
--------
I do believe this project is more suited for Jython 3, however I did not
find much solid information about it. Where can I find some resources
about status of Jython 3? Is this mailing list also for discussing Jython 3?

-------------------------------------------------------------

@Jeff Allen:

Thanks for letting me know about the Numba project. It might share
similar goals with mine. From a quick look I found that it seems to be
focused on numerical computations, run on manually marked pieces of code
and not analyze the Python program as a whole.

My goal is to perform static type analysis of the Python program being
ran and get type information from that. This does not necessarily need
to imply many changes in internal API, as it would work kind of aside
from the actual interpreter. Before even executing a Python program, I
would like to parse it, statically compute what it imports, then
recursively analyze that. Finally, I would like to export the type
information using a new API and maybe use it to optimize some parts of
the project by compiling some methods/classes to native-like Java
classes like in the example above (this is the part that could have a
lot in common with the Numba project).

I know that even adding two integers is a complex call, but this is
where the typing information helps me. Using your example, if I can
prove that both integers are instances of built-in Python int class, I
will also know that this class is not modifiable and will be able to
infer statically a lot of information. The problem is when I would deal
with a user class, where the __add__ method could be even dynamically
added anywhere in the code, but this is why I need to analyze the code
of the whole project. I assume that in sane cases I will be able to at
least define which files to analyze, knowing initial environment of the
execution, such as PYTHONPATH. If not, then I will have to fall back to
a "dynamic" type more often, not giving much useful information.

Do you think that kind of type inference project could be created as a
module of Jython and then maybe used to improve other parts?

-------------------------------------------------------------

@John Hayes:

The Cython project is also close to what I would like to achieve.
However, it seems to target mainly Python 2 and not use type annotations
at all.

Perhaps I should think about combining my research with Cython to
improve its type inference and broaden Python support. However, there
are various things I like about Jython, including already implemented
full support for execution of Python, the fact that it is targeting JVM,
which creates new possibilities of applications and, depending on many
things, I think I will perform some more research on programs running on
JVM in a more distant future.

Jan Wroblewski


On 11/7/2016 12:45 AM, John Hayes wrote:

> Hi everyone,
>
> I haven't followed all the subtleties of this discussion, but my naive
> impression is that statically typed Python variables have been dealt
> with, in a way, in Cython ( http://cython.org/) to enhance Python
> performance otherwise. It is incompatible with Jython itself per se
> AFAIK, but quite interesting in a general Pythonic sense IMO.
>
> Cheers,
> John
>
>
> On Nov 6, 2016 6:20 PM, "Jeff Allen" <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hello Jan,
>
>     this sounds a very interesting project. I have given a little thought to
>     type inference as an optimisation technique, walking the AST to infer
>     type. In that context it seems more helpful to infer the implementation
>     type (and range), rather than the Python type. This seems to be what
>     Numba does: I know that the Numba team find the type hints are not fine
>     enough for their purposes: they want to know how wide the float is, how
>     big the int.
>
>     I'm not hopeful it could be incorporated into Jython (or CPython)
>     without implementing the core entirely differently. There seems to be so
>     much of the runtime where actual type is washed away to PyObject. It's
>     not just a matter of providing a more precise return type for library
>     methods. All the abstractions work against you: even adding two integers
>     has to return a general object because the slot (equivalently Jython's
>     PyObject._add) has to accommodate the possibility of returning
>     NotImplemented.
>
>     Jeff Allen
>
>     On 06/11/2016 18:31, Stefan Richthofer wrote:
>      > Hello Jan,
>      >
>      > just some questions/remarks from me for now.
>      >
>      >
>      > PEP 484
>      >
>      > In your whole email you didn't mention PEP 484 for a single time.
>      > Will your work incorporate this official Python-typing?
>      > (i.e. given that Jython 3 is still in work, this mainly refers to
>     https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code
>     <https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code>)
>      > In case you are interested:
>      > I created a Python 2.7-workable version of
>     typing.get_type_hints() (the original from typing-module only works
>     for Python 3 and plainly returns None on Python 2.)
>      > Unfortunately it currently doesn't work on Jython due to
>     bugs.jython.org/issue2515 <http://bugs.jython.org/issue2515>.
>     Anyway, that issue would need a fix for any PEP 484-related work on
>     Jython.
>      >
>      >
>      > Java-typing
>      >
>      > How would your approach work with Java-typing behind the scenes
>     (e.g. for mixed Java/Python programs)? Whenever Jython imports a
>     Java-lib, on Java-site there is more Type-info available than the
>     import mechanism exposes. Also, it somewhat obfuscates that info due
>     to coercion and so. Would you also incorporate Java-type info?
>      > What about Java-Generics? The Generics-info can be retrieved from
>     Java too, it's a bit constrained and tricky though.
>      >
>      > I always wanted to do some type-cleanup in Jython in the sense
>     that Jython's Java-API could often be more precise regarding
>     type-info. E.g. it often just returns a PyObject, even when it is
>     clear that actually a PyTuple, a PyList<PyObject> or at least
>     List<PyObject> etc is returned.
>      > However, so far I lacked time and motivation to actually start
>     this. Also not sure if this could break code, but since the types
>     would only be specialized, everything should stay compatible.
>      > Anyway, maybe this is more something for Jython 3. Would it
>     benefit your approach?
>      >
>      >
>      > Including into Jython
>      >
>      > If your work goes well, I'd be in principle +1 on including it
>     into Jython. Especially if it would help to make the Java/Python
>     cross-connection more typesafe, especially when calling Python-code
>     from Java. However, let's see what the others say; I guess it will
>     depend on the actual benefits and features, which are currently
>     formulated a bit vaguely I guess.
>      >
>      >
>      > Best
>      >
>      > Stefan
>      >
>      >
>      >
>      >> Gesendet: Sonntag, 06. November 2016 um 15:21 Uhr
>      >> Von: "Jan Wroblewski" <[hidden email] <mailto:[hidden email]>>
>      >> An: [hidden email]
>     <mailto:[hidden email]>
>      >> Betreff: [Jython-dev] Statically typing Python in Jython
>      >>
>      >> To Jython devs,
>      >>
>      >> I am a 3rd year PhD student at University of Warsaw and I wanted
>     to do
>      >> my Python static typing research by implementing it as a part of
>     Jython.
>      >> Before starting, I would like to hear your opinion on combining my
>      >> research with Jython, and whether it would have a chance to be
>      >> integrated into the main Jython branch after it is complete. Below I
>      >> describe my plans.
>      >>
>      >> My research
>      >> -----------
>      >> I want to statically type Python, which means constructing automatic
>      >> type inference of most objects (incl. functions and classes)
>     present in
>      >> a Python program. The type system I am working on consists of:
>      >> - primitives,
>      >> - objects, i.e. dictionaries from strings known at compile time
>     to types
>      >> of values contained there,
>      >> - functions,
>      >> - constructs like "TypeA or TypeB", "FunctionTypeA and
>     FunctionTypeB".
>      >> In other words, before even running a program, I want to compute
>     what
>      >> properties each variable present in it has, and recursively what are
>      >> properties of those properties, until I get to primitives or
>     function
>      >> bodies.
>      >>
>      >> The Python type inference algorithm I have in mind would take
>     the Python
>      >> AST, convert it to a simpler Lucretia language and make the type
>      >> inference there. Lucretia is a language designed by Viviana
>     Bono, Marcin
>      >> Benke and Aleksy Schubert (my PhD supervisor) for the purpose of
>     static
>      >> typing of dynamic languages (pdf:
>     https://arxiv.org/pdf/1206.5112.pdf
>     <https://arxiv.org/pdf/1206.5112.pdf>).
>      >>
>      >> One of the features of my project is that I would like to
>     support full
>      >> Python language instead of subset of its features that are typeable
>      >> (like mypy does: http://mypy-lang.org/). Since it is provably
>     impossible
>      >> to make type inference algorithm working for all Python
>     programs, I want
>      >> to use gradual typing and treat all variables that my algorithm
>     was not
>      >> able to process as "dynamic".
>      >>
>      >> As for similar research projects, I'd say Reticulated Python by
>     Michael
>      >> M. Vitousek, Andrew M. Kent, Jeremy G. Siek and Jim Baker is the
>     closest
>      >> one to what I would like to achieve (pdf:
>      >>
>     http://wphomes.soic.indiana.edu/jsiek/files/2014/03/retic-python.pdf
>     <http://wphomes.soic.indiana.edu/jsiek/files/2014/03/retic-python.pdf>).
>      >> Reticulated Python also does not seem to restrict allowed language
>      >> constructs, but it has slightly different type system and different
>      >> policy on handling imports (kind of at runtime instead of
>     statically). I
>      >> noticed that Jim Baker is not only sub-author of Reticulated Python
>      >> paper, but also a major commiter to Jython, so I am interested in
>      >> hearing if some of the work from Reticulated Python is already
>      >> integrated into Jython.
>      >>
>      >> Combining with Jython
>      >> ---------------------
>      >> The type inference in Lucretia and combining its results with
>     Python is
>      >> something new and I would like to implement it as a part of
>     Jython. The
>      >> current Jython implementation would help me with parsing,
>     execution, and
>      >> generally with lots of technical stuff. I was originally going
>     to target
>      >> only recent versions of Python 3, but I am open for changes, even to
>      >> Python 2.7. If the general idea is welcome, I am also open to other
>      >> major changes.
>      >>
>      >> I think that there could be two main applications of my research:
>      >> - Improvement of IDEs - after exposing some Jython internals
>     connected
>      >> to my work, more precise type information could be made
>     available to an
>      >> IDE. As many IDEs run on JVM, the integration could also be
>     smoother.
>      >> - Producing faster, more optimized for JVM bytecode - although
>     it is not
>      >> the main goal of Jython, more speed is always better. I believe
>     I could
>      >> do that in situations where I could prove that no properties
>     would be
>      >> ever dynamically added to an object, existing ones would not change
>      >> their type and I could resolve statically all called methods.
>      >>
>      >> Jan Wroblewski
>      >> http://www.mimuw.edu.pl/~xi/
>      >>
>      >>
>     ------------------------------------------------------------------------------
>      >> Developer Access Program for Intel Xeon Phi Processors
>      >> Access to Intel Xeon Phi processor-based developer platforms.
>      >> With one year of Intel Parallel Studio XE.
>      >> Training and support from Colfax.
>      >> Order your platform today. http://sdm.link/xeonphi
>      >> _______________________________________________
>      >> Jython-dev mailing list
>      >> [hidden email]
>     <mailto:[hidden email]>
>      >> https://lists.sourceforge.net/lists/listinfo/jython-dev
>     <https://lists.sourceforge.net/lists/listinfo/jython-dev>
>      >>
>      >
>     ------------------------------------------------------------------------------
>      > Developer Access Program for Intel Xeon Phi Processors
>      > Access to Intel Xeon Phi processor-based developer platforms.
>      > With one year of Intel Parallel Studio XE.
>      > Training and support from Colfax.
>      > Order your platform today. http://sdm.link/xeonphi
>      > _______________________________________________
>      > Jython-dev mailing list
>      > [hidden email]
>     <mailto:[hidden email]>
>      > https://lists.sourceforge.net/lists/listinfo/jython-dev
>     <https://lists.sourceforge.net/lists/listinfo/jython-dev>
>      >
>
>
>     ------------------------------------------------------------------------------
>     Developer Access Program for Intel Xeon Phi Processors
>     Access to Intel Xeon Phi processor-based developer platforms.
>     With one year of Intel Parallel Studio XE.
>     Training and support from Colfax.
>     Order your platform today. http://sdm.link/xeonphi
>     _______________________________________________
>     Jython-dev mailing list
>     [hidden email]
>     <mailto:[hidden email]>
>     https://lists.sourceforge.net/lists/listinfo/jython-dev
>     <https://lists.sourceforge.net/lists/listinfo/jython-dev>
>
>
>
> ------------------------------------------------------------------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today. http://sdm.link/xeonphi
>
>
>
> _______________________________________________
> Jython-dev mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/jython-dev
>

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev