Nuitka Compiler for Python

I like developing using Python.  I like its program structure, its performance and the way it supports coroutines.  What I don’t like is distributing Python programs.  Python installations from one computer to the next are notoriously different from one another.  I may have two seemingly similar Linux computers running the same version of Python and a program may run well on one and may not find a dependency on another.

While the language and run-time system are well-defined, the way in which search-paths are set up and resolved does not seem to be. (See http://stackoverflow.com/questions/25715039/python-interplay-between-lib-site-packages-site-py-and-lib-site-py). Virtualenv aims to help, but the contract between Python and “the system” seems confusing.  The result for me has that it has been difficult to produce shrink-wrapped programs for distribution from Python source.

The types of programs I’ve developed usually have dependencies on locally-developed SWIG-generated shared-libraries (.so), so my case may not be typical.  (However, it is interesting :-)  I’ve looked at using Cython to compile Python modules, and it works well for building an extension module in C, or for extending a C program with Python.  However, if your aim is software construction using Python for the distribution of a complete application, then Cython seems lacking.

Nuitka

Nuitka (http://nuitka.net) is a “Python Compiler.”   Nuitka can compile just one or a few modules, or it can compile an entire Python program.  It takes a higher-level view of the job of compiling an entire project.

In standalone mode, its aim is to comprehend an entire Python program and emit a compiled version of that program with all libraries (shared objects) included. The compiled version captures all of the dependencies of the program on the system and places them in a binary executable and distribution directory.  The resulting distribution should then work on any system with the same ABI (Application Binary Interface) as the compiling system.

In my tests, I’ve found it to be very capable.  This article describes some of what I’ve learned.

Basic Use

In basic use, compiling a Python program for distribution is as simple as:

 % nuitka --standalone main.py

When completed, Nuitka will produce a directory `main.dist` containing a main executable and the collection of all shared libraries used by your project.  Because Nuitka walks the parse tree of all of the Python used by your program, the main executable includes the compiled version of all modules used by your program.

main.dist/main.exe - the main executable

The copies of shared libraries found includes many system libraries, as well as the shared-objects accompanying locally-compiled extension modules.

main.dist/libc.so.6 
main.dist/libstdc++.so.6
main.dist/_curses.so
main.dist/libsqlite3.so.0
main.dist/mymod/submod/_my_swig_mod.so 

You can run the binary of the resulting distribution like this

% main.dist/main.exe

and it will run exactly as if you had typed

% python main.py

Nuitka parses your entire Python program.  It begins at your “main.py” and builds a parse tree of all statements.  “import” statements are handled at compile time, and imported modules are included in the parse tree.

This is different from some other compilation systems that compile Python code on a module-by-module basis.

Nuitka analyzes the Python program for all of the shared objects it references and gathers copies of them for inclusion in the distribution.

Nuitka compiles Python into C++ and then compiles that C++.  Nuitka optimizes the intermediate form so that the resulting code can perform much faster than the original interpreted version.

While speed may be the attraction of compilation for some, producing a shrink-wrapped binary distribution was more interesting to me.

Analyzing your Distribution

If you are preparing a distribution, it is important to make sure that Nuitka has found all of the components your application relies on. You can see the dynamic libraries your distribution is loading by using the Linux `ldd` command.

 % ldd main.dist/main.exe

It is also useful to watch what your distribution is loading dynamically by setting the PYTHONVERBOSE environment variable.

 % PYTHONVERBOSE=1 main.dist/main.exe

You will get a very detailed listing of all of the imports your python program performs.  If your program is dynamically loading a module, you will see something like this

 import mymod.submod.my_swig_mod # dlopen("/home/foo/blah/.../mymod/submod/_my_swig_mod.so")

In my case, my SWIG module “my_swig_mod’ was loading its shared object dynamically.  I could see the location it was loading from, and it was obvious it was not part of the packed distribution.

Issues with Dynamically Loaded shared objects (SWIG)

Nuitka does a good job at finding shared objects that are loaded as a result of an “import” statement, but it cannot find shared objects that are loaded dynamically.  While you can ask explicitly ask Nuitka to include specific Python modules, it still may not find an associated “.so” file since the library remains dynamically loaded.

If you want to include such modules in your distribution, they may need a little help.

I ran into this while using a custom module generated by [SWIG – “Simple Wrapper Interface Generator”](https://swig.org) .  In the description here, I’ll use a fictitious module called “my_swig_mod” that is a sub-module of a module called “mymod.submod”.

The difficult Python code is shown below.  It is generated by SWIG itself (https://github.com/swig/swig/blob/master/Source/Modules/python.cxx), and so the only way to “fix” it is to alter SWIG.

This code attempts to find the “.so” file related to module “my_swig_mod.py”.  SWIG expects that the shared object should live in the same directory as the Python wrapper and that it should begin with an underscore.

==== my_swig_mod.py
from sys import version_info
if version_info >= (2,6,0):
    def swig_import_helper():
        from os.path import dirname
        import imp
        fp = None
        try:
            fp, pathname, description = imp.find_module('_my_swig_mod', [dirname(__file__)])
        except ImportError:
            import _my_swig_mod
            return _my_swig_mod
        if fp is not None:
            try:
                _mod = imp.load_module('_my_swig_mod', fp, pathname, description)
            finally:
                fp.close()
            return _mod
    _my_swig_mod = swig_import_helper()
    del swig_import_helper
else:
    import _my_swig_mod
del version_info

SWIG tries hard to produce code that can run on many different release-levels of Python system.  To accomplish its goals, the SWIG Python handler performs some tricky things (like the excerpt shown
above.)

If you are using Nuitka to produce a distribution, your aim is different.  You have exactly ONE Python release that you are compiling to.  In my case I was using Python 2.7.  In my case, the entire block of code in the file “my_swig_mod.py” above can be replaced with the direct import below.

==== my_swig_mod.py
import _my_swig_mod

Nuitka handles this import nicely.

Conclusion

Software construction and distribution seems to be going through a period of rethinking at the current time. In the past few years I have seen the proliferation of systems like ‘virtualenv’ for Python and ‘rbenv’ for Ruby. These applications patch the system to help in running applications that may have conflicting dependencies. These tools can help in supporting a few broadly different language versions (Python 2.7, Python 3.2), (Ruby 1.8.7, Ruby 2.1.1). While these tools can help in setting up a few development environments, they do not do much to isolate dependencies and help with application distribution.

I see an interesting corollary between Nuitka and the excitement behind [Docker](https://www.docker.com/) containers. A container describes

  • the file system
  • and libraries

necessary to run

  • a binary

on a Linux ABI. A Dockerfile is a recipe for producing a container and building the binary. Nuitka analyzes a Python program and produces a somewhat similar container: its distributon. A Nuitka distribution includes a binary and the shared objects necessary to run on an ABI (it does not include a file system). Nuitka distributions are shrink-wrapped applications ready to run.

Appendix

A quick comparison to [Cython](http://cython.org/)

Cython is an optimizing static compiler for Python and an extended language called Cython.

  • it is module based
  • it does not walk through the dependency tree
  • does automatically compile nested modules: when a top-level module
    is composed of other modules
  • it does not find shared objects used by your application

11 thoughts on “Nuitka Compiler for Python”

  1. Cython has one other big problem: It is similar to Python, but it is not Python.
    It is a new single purpose programming language, with bad support for text editors, no formal definition and few users. As good and useful as it is, it is a fringe language for writing numerical extension modules.
    Once you make the decision to convert a module to Cython, it cannot be run in Python anymore (excluding the pure module that is not used much).

    Nuitka on the other hand runs on unmodified Python code and has a much wider user base beyond the numerical community.

    It would be great if the Cython community could leverage Nuitka and make the numpy specific optimizations a part of Nuitka.

  2. Have you any experience to make a portable python app which uses gui libs like pygobject or qt ? Does Nuitka do the job?
    Thanks.

  3. Im trying, this because I couldnt manage to compile my program that uses >tkinterrequests< modules.
    I have many errors…..like
    "Cannot find 'tkinter' as relative or absolute import."
    When i try my code, without the tkinter GUI , i get something like this

    " Cannot find 'http.cookies' in package 'requests' as relative or absolute import."

    Python 3 is quite nice to learn for non programers like me, but too much bugy to distribut…compile….share…

  4. i just tried compile my main.py with tkinter & others imports & all is working.
    I have just copy TCL folder in my app distribution.

    I use :
    -Mingw-64
    -Python27 (Nuitka need find a directory called “python27”, python27 is used by nuitka for compilation)
    -Python34 (The version i use for code my apps)

    Command :
    nuitka –windows-disable-console –recurse-all –standalone FICHIER.py

    Result:
    FICHIER.exe 7.84Mo
    All dependencies included FICHIER.exe 30Mo

    Tested on the computer of my brother (He haven’t Python) , works :)

    –windows-disable-console (i don’t want show console for my tkinter application)
    –recurse-all (Search for all modules)
    –standalone (python distri with your app)

    it’s happens modules are not included , but you can fix that . Read the missing modules in error & copy the right directory , files in the exe’s folder. I hope this project don’t die , i really love python but the problem is about disbrution , deployment.
    With nuitka we can make standalone application , sources are more protected (i know nothing is 100%safe) and you gain perfomance.

    Sorry for my english .

  5. Just tried to compile my Python program as a “standalone”, but it failed because it couldn’t find a Tk.framework. Nuitka did say when I started it that “The version ‘3.5’ is not currently supported. Expect problems.”, so I can’t say I wasn’t warned. I know Python 3.5 is new and developers are still catching up with it, but it’s a pain when you have to wait several months before promising software like Nuitka and py2exe become usable for you.

  6. You mention “Tk.framework”. Does that mean you were using Nuitka on OS X? It can be helpful to share some of the details of the platform you tested on.
    Good luck!

  7. OS X 10.10.5, Python 3.5. Program uses modules: tkinter, array, deque from collections, urandom from os, os.path, sys, time, pack from struct. It links to an external .txt file and two folders on the same directory as the main program. The original version imported four Cythonised modules (.so files), but I re-wrote them into pure Python versions and incorporated them into the main program as functions for the Nuitka test.

  8. Good info. someone else had trouble with tkinter and py3. i know from my own experience that Tk.framework on OS X works fine with the builtin python but does not always play nice with others. Hopefully this info can help figure this out.

  9. I had a module (actually a compiler written in python to compile an in-house language). I compiled it using both Cython and Nuitka. I did not use any of the Cython construct, so it was pure python code. Cython gave me 40% speed improvement and Nuitka gave me about 20% (versions february 2016). I know with Cython I could squeeze more if I’d use their type directives but I’d lose Python level debugging capability in case I need it with non-Python compatible code. For now, both systems seem capable and it’s interesting to learn where Nuitka will be 1 year from now. However Cython seems to optimize more and better at this point at least for my scenario.

  10. Tackling this for days now, tried many different ways, environments (32bit windows) and still no success.
    So I’ve seen this https://github.com/kivy/kivy/wiki/Packaging-Kivy-apps-written-in-Python-3,-targeting-Windows-using-Nuitka tried with this kivy sdk as well as standalone installation with numpy/scipy installed from .whl or even winpython… and the closest I’ve got was woking build – but only on the build system. I guess it’s some kind of dependency issue?

    If anyone is interested or have any idea how to createa standalone build out of this simple app: http://pastebin.com/KwSd0kNx to work out of the box on any modern windows system it would be great help.

    Just for the record, I can build simple “Hello world” apps that have few imports, but with kivy had no luck. And I’ve tried many different ways:
    nuitka –msvc=14.0 –python-version=3.4 –recurse-all –standalone main.py
    nuitka –msvc=14.0 –python-version=3.4–recurse-plugins=dependencies.py –standalone main.py

    I’ve tried with –mingw as well (though even simple hello world app didn’t work for me, not sure how to set it up properly).
    And no matter what I try, there are always dependency warnings during the build time (“Can’t import as a global import” which I guess means it cant find all required modules).

    So… does anybody have any experience/idea how to solve this?

Leave a Reply

Your email address will not be published. Required fields are marked *