1
Vote

pyc.py: generates exes which fail on using std python libs

description

I tried to generate a standalone exe for a tool I have (3 modules) but it bombs out at startup with:
 
Unhandled Exception: IronPython.Runtime.Exceptions.ImportException: No module named os
at DLRCachedCode.__main__$1(CodeContext $globalContext, FunctionCode $functionCode)
at IronPython.Compiler.OnDiskScriptCode.Run()
at IronPython.Runtime.PythonContext.InitializeModule(String fileName, ModuleContext moduleContext, ScriptCode scriptCode, ModuleOptions options)
at IronPython.Runtime.ModuleLoader.load_module(CodeContext context, String fullName)
at Microsoft.Scripting.Interpreter.FuncCallInstruction`4.Run(InterpretedFrame frame)
at Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame frame)
at Microsoft.Scripting.Interpreter.LightLambda.Run4[T0,T1,T2,T3,TRet](T0 arg0, T1 arg1, T2 arg2, T3 arg3)
 
I can reproduce with this simple foo.py module
 

foo.py

import os
print "Hello"
print "OS:",os.name
print "Bye"

end foo.py

 
which I compile with the following:
 
ipy -m pyc /standalone /main:foo.py /target:exe /out:foo
or
ipy -m pyc /main:foo.py /target:exe /out:foo2
 
Starting either foo.exe or foo2.exe fails with above error.
 
If I remove the import os and the line that uses on.name, I get an executable that works...

file attachments

comments

slide_o_mix wrote Mar 15, 2012 at 8:38 PM

What pyc.py needs to do is try and find all the modules that are needed and automatically compile them as well.

ddewaleffe wrote Mar 16, 2012 at 1:23 PM

I started to look at this last night... Hope to get this done over the week-end.
More news soon...

slide_o_mix wrote Mar 16, 2012 at 2:01 PM

ModuleFinder won't work for IronPython as it stands now, so there may need to be another avenue found.

ddewaleffe wrote Mar 17, 2012 at 4:10 PM

I attach findeps.py and my modified pyc.py (packaged in zip)

This relies on ast.py to recursively find imports and generate a list of files to be compiled along the main ones.
It works for the small examples I tried.
It may miss dependencies if ast.py fails to parse a source file (e.g: subprocess.py, random.py)

slide_o_mix wrote Mar 17, 2012 at 5:35 PM

What license would you attribute to finddeps.py? I think we would need it something similar to Apache in order to include it.

ddewaleffe wrote Mar 18, 2012 at 9:44 PM

Here is a new version of finddeps.py:
  • includes mention of licensing (apache v2)
  • falls back of re based parsing (imperfect) for the cases where ast fails.... (see secondBestGuess() function)

ddewaleffe wrote Mar 19, 2012 at 7:21 AM

Fixed bug in latest finddeps.py. Wrong type returned...

ddewaleffe wrote Mar 19, 2012 at 7:27 AM

However, still does not work completely. It fails for complex modules like 'logging' implemented in a directory... More to do for tonight...

ddewaleffe wrote Mar 27, 2012 at 7:05 PM

Got a bit further. This latest package is able to get at the packages init.py files too and tries m.n....mm.nn.py as well as m.n....mm.py or similarly for init.py
So I get many more of the needed module...

However, if I generate pyc.exe using ipy pyc.py .....
the generated executable seems to contain the needed modules but fails to work as expected. Further I get no errors.... So at this stage I am a bit lost....
I am not claiming this code is the cleanest....

On the samples I tried, it seems that one quickly pulls in quite a lot of the std lib... Making me wonder if, indeed packaging the whole of the stdlib in zip file would not just be right...
But is it possible to package this also in the generated executable.

My hope is really to get to a single .exe that contains all that's needed....

slide_o_mix wrote Mar 28, 2012 at 10:49 PM

So you think it would be good to have a way of embedding the stdlib as a zip into the executable instead of compiling it all up?

ddewaleffe wrote Mar 29, 2012 at 4:27 AM

My primary concern is to be able to produce a real standalone exe. 1 file to copy somewhere. No requirement to copy/install anything else. Dont care if big...

This first trial I made sort of partly work. For instance it works on:
import os.path
print "Hello"
print os.path.realpath('.')
print "running on", os.name
print "Bye"

But it does not (no error message at all) for pyc.py. Don't know why at the moment. It surely worth looking at why it fails....

Looking at the list of dependencies pulled in in each case, they are many.... so why not include the whole of stdlib at once instead of trying to analyze it by parsing at every compilation. This compiled version could be cached to avoid doing it every time...

Now we have 2 ways to do this:
a. package the zip file and rely on zimpimport doing its work
b. compile all the files of the library and package them all in the executable

Sign in to add a comment or to set email notifications