Compiling the source without stat

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Compiling the source without stat

Hossein-7
Hi. I just started to port latest python 2.7.2 to another platform
(don't think you're eager to know... well it's CASIO ClassPad).
And I faced a "minor" problem... It hasn't got stat or fstat or
anything. (It supports a very limited set of c std lib).
As pyport.c suggested, i defined both DONT_HAVE_STAT and
DONT_HAVE_FSTAT, but problems only began.
It failed to compile most of import.c, particularly because it fell into
the wrong `#if !defined(PYOS_Something)' blocks. Sometimes it just fell
into an #else part which assumed stat are available. So although
HAVE_STAT is meant to control file operations, most of the source code
aren't implement to use it. You see how "minor" the problem was?
So now I need advice from developers.
Is there a fix for it? What a question... definitely no replacement to stat.
It's 99% definite that I can't compile further without touching the
source code. I have to #define my own PYOS_whatever and handle files in
my own way. In that case where should my special file handling cases go?
I saw some marshal.c code which seemed it wants to abstract away
platform's file handling from source code; but from what I understood it
can't be made to use alternate file handling methods.
If there is anything I should do (maybe show you my handmade
pyconfig.h?) tell me.
[My first post in a mailing list... Should I say] Best Regards, Hossein
[in here?]
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Compiling the source without stat

Petri Lehtinen
Hossein wrote:

> Hi. I just started to port latest python 2.7.2 to another platform
> (don't think you're eager to know... well it's CASIO ClassPad).
> And I faced a "minor" problem... It hasn't got stat or fstat or
> anything. (It supports a very limited set of c std lib).
> As pyport.c suggested, i defined both DONT_HAVE_STAT and
> DONT_HAVE_FSTAT, but problems only began.
> It failed to compile most of import.c, particularly because it fell
> into the wrong `#if !defined(PYOS_Something)' blocks. Sometimes it
> just fell into an #else part which assumed stat are available. So
> although HAVE_STAT is meant to control file operations, most of the
> source code aren't implement to use it. You see how "minor" the
> problem was?
> So now I need advice from developers.
> Is there a fix for it? What a question... definitely no replacement to stat.
> It's 99% definite that I can't compile further without touching the
> source code. I have to #define my own PYOS_whatever and handle files
> in my own way. In that case where should my special file handling
> cases go? I saw some marshal.c code which seemed it wants to
> abstract away platform's file handling from source code; but from
> what I understood it can't be made to use alternate file handling
> methods.
> If there is anything I should do (maybe show you my handmade
> pyconfig.h?) tell me.

See http://bugs.python.org/issue12082. Currently neither Python 2.x
nor 3.x can be compiled without stat() or fstat(). Python 2.7 almost
compiles, but Python 3 depends heavily on them.

The problem boils down to the fact that you cannot really check
whether a filesystem entry is a directory without calling stat() or
fstat().

My personal opinion is that support for DONT_HAVE_STAT and
DONT_HAVE_FSTAT defines should be dropped because they don't work, and
would only be useful in a very limited set of cases.

> [My first post in a mailing list... Should I say] Best Regards,
n> Hossein [in here?]

Yeah, why not? :)

Regards, Petri
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Compiling the source without stat

"Martin v. Löwis"
In reply to this post by Hossein-7
> It's 99% definite that I can't compile further without touching the
> source code. I have to #define my own PYOS_whatever and handle files in
> my own way. In that case where should my special file handling cases go?

It's difficult to say how to proceed. On one hand, I don't see an
overwhelming need to support systems without stat, and am tempted
to say that you are on your own.

On the other hand, it appears that people keep asking for it, from
time to time. So if it was possible to support such systems without
making the code too convoluted, it may be worth supporting it.

One thing seems clear: without stat(), we cannot possibly support
.pyc files, at least not in __pycache__. So one consequence of
a lacking stat should be that all the code dealing with caching of
byte code files needs to be disabled. Supporting .pyc as modules
might still be possible.

It's questionable how to deal with path searching in the absence
of stat. Testing for the presence of a file is possible in principle
by trying to open the file, and closing it when it was found to be
present. So in the places where we only check for the presence of
a file, an alternative implementation could be provided.

In any case, it needs someone to champion such a project, preferably
in an ongoing manner (i.e. several years). So if you are interested,
you should
- volunteer to maintain stat-less systems for some time
- create a port of Python 3 that works stat-less
- come back to python-dev for review to determine whether it's
  worth to support such systems.

Alternatively, you can just make your own fork of Python, which
you may or may not publish.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Compiling the source without stat

Terry Reedy
In reply to this post by Petri Lehtinen
On 12/14/2011 2:26 PM, Petri Lehtinen wrote:

> The problem boils down to the fact that you cannot really check
> whether a filesystem entry is a directory without calling stat() or
> fstat().
>
> My personal opinion is that support for DONT_HAVE_STAT and
> DONT_HAVE_FSTAT defines should be dropped because they don't work, and
> would only be useful in a very limited set of cases.

At present, it seems to be an attractive nuisance, tempting people like
Hossein to try something that does not work.

--
Terry Jan Reedy

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Compiling the source without stat

Hrvoje Niksic-2
In reply to this post by Hossein-7
On 12/14/2011 05:04 PM, Hossein wrote:
> If there is anything I should do

You can determine what the code that calls stat() is trying to do, and
implement that with other primitives that your platform provides.  For
example, you can determine whether a file exists by trying to open it in
read-only mode and checking the error.  You can find whether a
filesystem path names a directory by trying to chdir into it and
checking the error.  You can find the size of a regular file by opening
it and seeking to the end.  These substitutions would not be acceptable
for a desktop system, but may be perfectly adequate for an embedded one
that doesn't provide stat() in the first place.  Either way, I expect
that you will need to modify the sources.

Finally, are you 100% sure that your platform doesn't provide another
API similar to stat()?
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Compiling the source without stat

Hossein Azadmanesh
It does have its own file handling functions: Opening, getting the size,
enumerating directories, etc.
It has its own limitations too. No dates supported, folders only one
level deep, maximum 99 files inside each folder, etc.
There is not a function called stat. But I am considering faking it,
will explain in another reply.
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Compiling the source without stat

Hossein-7
In reply to this post by Petri Lehtinen
I wanted to say something in the bug page petri showed (
http://bugs.python.org/issue12082 ) however I though about first
discussing it here. If faking a stat struct and a function to fill it
solves the problem, and checking for existing files and folders is the
only thing that python needs to be compiled (i'm talking about 2.7) then
it's possible to fail-check it by just trying to open the file.

If you don't want to change the stat mechanism, you can create a new
#define which can let user point it to his own faked stat function and
struct.
I'm currently trying to fake stat to see what happens next, but I guess
I will have more problems with file handling later.

By the way, some people with the same problem there said they "used"
python by setting the Py_DontWriteBytecodeFlag flag, but here my problem
is that i can't compile it. Dunno what they really did.
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Compiling the source without stat

Victor STINNER
Le jeudi 15 décembre 2011 15:29:23 vous avez écrit :
> If faking a stat struct and a function to fill it
> solves the problem, and checking for existing files and folders is the
> only thing that python needs to be compiled (i'm talking about 2.7) then
> it's possible to fail-check it by just trying to open the file.

It's better to only work on Python 3.3. I consider "support platform without
stat" as a new feature, and new features are only accepted in Python 3.3.

Victor
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Compiling the source without stat

"Martin v. Löwis"
In reply to this post by Hossein-7
Am 15.12.2011 12:59, schrieb Hossein:
> I wanted to say something in the bug page petri showed (
> http://bugs.python.org/issue12082 ) however I though about first
> discussing it here. If faking a stat struct and a function to fill it
> solves the problem, and checking for existing files and folders is the
> only thing that python needs to be compiled (i'm talking about 2.7) then
> it's possible to fail-check it by just trying to open the file.

That's not true. It also looks at the file modification time.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com