What do PyAPI_FUNC & PyAPI_DATA mean?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

What do PyAPI_FUNC & PyAPI_DATA mean?

Mark Shannon-3
Many (most?) of the function declarations in the CPython header files
are annotated with the PyAPI_FUNC declaration.
Similarly for data declarations and PyAPI_DATA

What do they mean, exactly? From the name I would expect that they are a
way of declaring a function or datum to be part of the API, but their
usage seems to be more to do with linkage.

The reason I am asking is that the new dictionary implementation
declares a few functions used to communicate between dictobject.c,
typeobject.c and ceval.c which cannot be static functions, but are not
intended to be part of the API.

Cheers,
Mark.
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: What do PyAPI_FUNC & PyAPI_DATA mean?

Benjamin Peterson-3
2012/4/23 Mark Shannon <[hidden email]>:
> Many (most?) of the function declarations in the CPython header files
> are annotated with the PyAPI_FUNC declaration.
> Similarly for data declarations and PyAPI_DATA
>
> What do they mean, exactly? From the name I would expect that they are a way
> of declaring a function or datum to be part of the API, but their usage
> seems to be more to do with linkage.

They define linkage on Windows. I actually don't know if they should
be applied to internal functions.



--
Regards,
Benjamin
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: What do PyAPI_FUNC & PyAPI_DATA mean?

Kristján Valur Jónsson
IMHO, we are _much_ too generous at applying this to almost whatever gets exposed between .c files.
I have created something called the "restricted" api for our custom python27.dll where I use different
macros (PyAPI_RFUNC, pyAPI_RDATA) to mean that things aren't exported for "restricted" builds.  We
use it to remove some of the easier access points to the dll for hackers to exploit.

Also, once declared exported this way, things become more bothersome to remove again, since once could always argue that someone out there is using these thigns.

K

> -----Original Message-----
> From: python-dev-bounces+kristjan=[hidden email]
> [mailto:python-dev-bounces+kristjan=[hidden email]] On
> Behalf Of Benjamin Peterson
> Sent: 23. apríl 2012 12:59
> To: Mark Shannon
> Cc: Python Dev
> Subject: Re: [Python-Dev] What do PyAPI_FUNC & PyAPI_DATA mean?
>
> 2012/4/23 Mark Shannon <[hidden email]>:
> > Many (most?) of the function declarations in the CPython header files
> > are annotated with the PyAPI_FUNC declaration.
> > Similarly for data declarations and PyAPI_DATA
> >
> > What do they mean, exactly? From the name I would expect that they are
> > a way of declaring a function or datum to be part of the API, but
> > their usage seems to be more to do with linkage.
>
> They define linkage on Windows. I actually don't know if they should be
> applied to internal functions.
>
>
>
> --
> Regards,
> Benjamin
> _______________________________________________
> Python-Dev mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-
> dev/kristjan%40ccpgames.com


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: What do PyAPI_FUNC & PyAPI_DATA mean?

"Martin v. Löwis"
In reply to this post by Mark Shannon-3
> What do they mean, exactly? From the name I would expect that they are a
> way of declaring a function or datum to be part of the API, but their
> usage seems to be more to do with linkage.

It means that they will be exported from the pythonXY.dll on Windows. In
Windows DLLs, it's not sufficient to make a symbol global (non-static)
to use it in an application, you also have to declare it as
__declspec(dllexport), or list it in the linker definition file (which
is not used today anymore).

Likewise, to use a symbol from a DLL, you also need to declare it
as __declspec(dllimport) in the using application. This will, in
particular, arrange for a slot in the indirect-jump assembler section
of the using DLL, so that the resulting executable will be position-
independent (except for this procedure linkage section).

As we have the same header files both for the implemenation and the
usage, this macro tricky is necessary to sometimes say dllexport,
sometimes dllimport.

Even though it's strictly needed on Windows, and strictly only for
API that we do want to expose, we apply it to all API that is
public on Unix (i.e. all Py* API), in order to avoid API being available
on Unix but not on Windows.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: What do PyAPI_FUNC & PyAPI_DATA mean?

"Martin v. Löwis"
In reply to this post by Kristján Valur Jónsson
Am 23.04.2012 15:05, schrieb Kristján Valur Jónsson:

> IMHO, we are _much_ too generous at applying this to almost whatever
> gets exposed between .c files. I have created something called the
> "restricted" api for our custom python27.dll where I use different
> macros (PyAPI_RFUNC, pyAPI_RDATA) to mean that things aren't exported
> for "restricted" builds.  We use it to remove some of the easier
> access points to the dll for hackers to exploit.
>
> Also, once declared exported this way, things become more bothersome
> to remove again, since once could always argue that someone out there
> is using these thigns.

For this, PyAPI_FUNC doesn't really matter. A symbol that is listed in
the header file is available on Unix even without such a declaration,
so listing it in the public header file is already the step that makes
it public, not specifying it as PyAPI_FUNC.

I agree that too much API is public, but the right approach is to rename
such API to _Py*, indicating to users that we don't want them to use it.
For existing API, that's tricky; for new API, I think it should be
private by default.

See also PEP 384.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: What do PyAPI_FUNC & PyAPI_DATA mean?

Kristján Valur Jónsson
You know that I'm speaking of Windows, right?
IMHO, we shouldn't put the PyAPI* stuff on functions unless they are actual API functions.
I don't know how the export tables for ELF .so objects is generated, but it surely can't
export _everything_.  Anyway, marking stuff as part of the API makes sense, and marking
functions as being part of the API makes no sense and is wasteful when they are not.
We might even have something similar for the stable api.

> -----Original Message-----
> From: "Martin v. Löwis" [mailto:[hidden email]]
> Sent: 24. apríl 2012 07:31
> To: Kristján Valur Jónsson
> Cc: Benjamin Peterson; Mark Shannon; Python Dev
> Subject: Re: [Python-Dev] What do PyAPI_FUNC & PyAPI_DATA mean?
 

> For this, PyAPI_FUNC doesn't really matter. A symbol that is listed in the
> header file is available on Unix even without such a declaration, so listing it in
> the public header file is already the step that makes it public, not specifying it
> as PyAPI_FUNC.
>
>
> I agree that too much API is public, but the right approach is to rename such
> API to _Py*, indicating to users that we don't want them to use it.
> For existing API, that's tricky; for new API, I think it should be private by
> default.


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: What do PyAPI_FUNC & PyAPI_DATA mean?

Kristján Valur Jónsson
In reply to this post by "Martin v. Löwis"
Aha, so that is the rationale.  Because the export table on unix is so
generous, we force ourselves to be generous on windows too?
I did some unix programming back in the day.  IRIX, actually (a Sys V derivative).  I'm pretty
sure we had to explicitly specify our .so exports.  But I might be mistaken.

K

> -----Original Message-----
> From: python-dev-bounces+kristjan=[hidden email]
> [mailto:python-dev-bounces+kristjan=[hidden email]] On
> Behalf Of "Martin v. Löwis"
> Sent: 24. apríl 2012 07:28
> To: Mark Shannon
> Cc: Python Dev

> Subject: Re: [Python-Dev] What do PyAPI_FUNC & PyAPI_DATA mean?
> Even though it's strictly needed on Windows, and strictly only for API that we
> do want to expose, we apply it to all API that is public on Unix (i.e. all Py*
> API), in order to avoid API being available on Unix but not on Windows.


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: What do PyAPI_FUNC & PyAPI_DATA mean?

"Martin v. Löwis"
In reply to this post by Kristján Valur Jónsson

Quoting Kristján Valur Jónsson <[hidden email]>:

> You know that I'm speaking of Windows, right?

Yes, but this may only be valid for CCP; for CPython, we certainly
have to consider Unix as well.

> IMHO, we shouldn't put the PyAPI* stuff on functions unless they are  
> actual API functions.
> I don't know how the export tables for ELF .so objects is generated,  
> but it surely can't export _everything_.

It certainly does. Any global symbol in an ELF shared library gets
exported. There are recent (10 years or so) ways to restrict this,
by declaring symbols as hidden in the object file, but exporting
everything is the default that Python uses.

> Anyway, marking stuff as part of the API makes  sense, and marking
> functions as being part of the API makes no sense and is wasteful  
> when they are not.

There are cases where it's necessary: when an extension module
uses a function that is not in the API.

> We might even have something similar for the stable api.

I don't understand. Everything in the stable api is part
of the API, and thus needs to be exported from the Python DLL.

Regards,
Martin


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: What do PyAPI_FUNC & PyAPI_DATA mean?

"Martin v. Löwis"
In reply to this post by Kristján Valur Jónsson

Quoting Kristján Valur Jónsson <[hidden email]>:

> Aha, so that is the rationale.  Because the export table on unix is so
> generous, we force ourselves to be generous on windows too?

Yes. If the code compiles and links on Unix, it shall also compile and
link on Windows.

> I did some unix programming back in the day.  IRIX, actually (a Sys  
> V derivative).  I'm pretty
> sure we had to explicitly specify our .so exports.  But I might be mistaken.

Maybe on IRIX, probably in a way that predates ELF. In the old days, on Linux,
you had to globally request address space from Linus Torvalds for  
shared libraries.
These days are long gone. ELF shared libraries are designed to give the same
experience (roughly) as static libraries, wrt. source compatibility.

Regards,
Martin


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com