Nuitka Compiler for Python

I like developing using Python.  I like its program structure, its performance and the way it supports coroutines.  What I don’t like is distributing Python programs.  Python installations from one computer to the next are notoriously different from one another.  I may have two seemingly similar Linux computers running the same version of Python and a program may run well on one and may not find a dependency on another.

While the language and run-time system are well-defined, the way in which search-paths are set up and resolved does not seem to be. (See Virtualenv aims to help, but the contract between Python and “the system” seems confusing.  The result for me has that it has been difficult to produce shrink-wrapped programs for distribution from Python source.

The types of programs I’ve developed usually have dependencies on locally-developed SWIG-generated shared-libraries (.so), so my case may not be typical.  (However, it is interesting :-)  I’ve looked at using Cython to compile Python modules, and it works well for building an extension module in C, or for extending a C program with Python.  However, if your aim is software construction using Python for the distribution of a complete application, then Cython seems lacking.


Nuitka ( is a “Python Compiler.”   Nuitka can compile just one or a few modules, or it can compile an entire Python program.  It takes a higher-level view of the job of compiling an entire project.
Continue reading Nuitka Compiler for Python

Real-Time Streaming Data Meetup

On November 21 I was pleased to participate in a meetup entitled Real-Time Streaming Data.  The organizer of this meetup assembles a wide variety of presenters and topics under the umbrella topic of “Large-Scale Production Engineering.” Chris (the organizer) does a remarkable job of keeping a pipeline of interesting talks coming.  I’m particularly interested in the January talk humorously entitled “Whatever happened to IPV6?”

Continue reading Real-Time Streaming Data Meetup

Verilog and Python together Again: Cocotb

It has been a few years since there has been a new project embedding the Python interpreter into Verilog. There is a new one on the block, called “Cocotb” built with a definite focus on writing testbenches and running regressions. It is written in a modern dialect of Python and hosted on Git. It is well-documented. I’m impressed with what I’ve seen.

This article gives a little bit of a history of Python and Verilog and describes how Cocotb fits in.

A Brief History

ScriptEDA (2001) [,] was primarily an example of using SWIG to link a Python interpreter into a Verilog simulator using PLI/VPI.  ScriptEDA handled TCK, Perl and Python in roughly the same way.

Continue reading Verilog and Python together Again: Cocotb

Twisted: txWS and Autobahn and Resource together

The web is organized in terms of “resources”, and many web frameworks make it easy to define a website as a collection of resources managed by an HTTP Resource handler.  Twisted is no exception: it has an excellent Web Server that manages a forest of Resources.  In Twisted terms, its web server is a case of one particular type of Protocol handler: an HTTP server that manages resources.

Sometimes it’s convenient to organize a web-site as a collection of Protocol handlers that each manage a different portion of the Resource forest.  There are many reasons for doing this, but most come down to when the characteristics of some of the resources are better handled by one type of server over another.

Apache installations are configured to do this all the time.  It is customary for static resources (files) to be served by a handler that is optimized for file serving.  Dynamic portions of the website can then be handled by Rails or Django.  In common use, one organizes their web site as a collection of handlers, each mounted at a different URL prefix.

"/static/*"             StaticHandler
"/mywebsite/*"          DynamicHandler

Streaming uploads are another type of traffic that is often better served by its own type of protocol handler.  In Twisted, the default Resource handler buffers an entire request before handling it.  If your application wants to handle upload packets as they arrive (i.e., a streaming upload), you need a custom handler.

"/static/*"             StaticHandler
"/mywebsite/*"          DynamicHandler
"/uploads/*"            UploadHandler


And then there are websockets.  The WebSocket protocol has been evolving and changing over the past few years.  A WebSockets protocol implements a full-duplex channel between the client and server.  Right now, Twisted’s Resources don’t co-mingle with the two popular WebSockets implementations txWS and Autobahn.  Out of the box, Twisted makes it easy to set up a handler for an HTTP Site on different ports, but it doesn’t make it easy to set up two different Protocol handlers at different URI prefixes.

When I have needed to run a standard HTTP website along-side a Twisted WebSocket, I’ve organized my site something like the code below.

reactor.listenTCP(80, HTTPFactory())
reactor.listenTCP(8080, WebSocketFactory())

This isn’t exactly what I wanted to do.  What I wanted was something that dispatched HTTP requests to entirely different protocol handlers based on the URI prefix.

   "/mywebsocket/*" = WebSocketFactory(),
   "/everythingelse/*" = HTTPFactory(),

I wanted to have a little proxy factory that could be configured to route a connection to the proper handler.  This way, I could write specialized Protocol handlers for special applications (streaming media in my case), and still use the proven parts of Twisted’s Resources for everything else.  This way I could drop in either one of the websockets toolkits (txWS or Autobahn) in a Twisted application.

A Proxy Toolkit

I wrote a little toolkit for proxying an incoming TCP connection and dispatching it to one of a collection of Twisted services.  The first few packets of the connection are collected for the dispatcher to make its decision.  These packets are replayed into the chosen service and then the original connection is spliced into the new service.  The service can implement a full-duplex connection like a WebSocket if needed.  A service may be out-of-process as well.  The toolkit can proxy traffic besides HTTP since it operates at Layer 4 of the OSI model.

You can read about the toolkit here at  It comes with a couple of examples that show how to use it.

Update: 2012-11-05

Autobahn recently added a Resource handler. There is a Twisted repository branch adding txWS as a standard resource; it is not released yet.