On November 21 I was pleased to participate in a meetup entitled Real-Time Streaming Data. The organizer of this meetup assembles a wide variety of presenters and topics under the umbrella topic of “Large-Scale Production Engineering.” Chris (the organizer) does a remarkable job of keeping a pipeline of interesting talks coming. I’m particularly interested in the January talk humorously entitled “Whatever happened to IPV6?”
The web is organized in terms of “resources”, and many web frameworks make it easy to define a website as a collection of resources managed by an HTTP Resource handler. Twisted is no exception: it has an excellent Web Server that manages a forest of Resources. In Twisted terms, its web server is a case of one particular type of Protocol handler: an HTTP server that manages resources.
Sometimes it’s convenient to organize a web-site as a collection of Protocol handlers that each manage a different portion of the Resource forest. There are many reasons for doing this, but most come down to when the characteristics of some of the resources are better handled by one type of server over another.
Apache installations are configured to do this all the time. It is customary for static resources (files) to be served by a handler that is optimized for file serving. Dynamic portions of the website can then be handled by Rails or Django. In common use, one organizes their web site as a collection of handlers, each mounted at a different URL prefix.
"/static/*" StaticHandler "/mywebsite/*" DynamicHandler
Streaming uploads are another type of traffic that is often better served by its own type of protocol handler. In Twisted, the default Resource handler buffers an entire request before handling it. If your application wants to handle upload packets as they arrive (i.e., a streaming upload), you need a custom handler.
"/static/*" StaticHandler "/mywebsite/*" DynamicHandler "/uploads/*" UploadHandler
And then there are websockets. The WebSocket protocol has been evolving and changing over the past few years. A WebSockets protocol implements a full-duplex channel between the client and server. Right now, Twisted’s Resources don’t co-mingle with the two popular WebSockets implementations txWS and Autobahn. Out of the box, Twisted makes it easy to set up a handler for an HTTP Site on different ports, but it doesn’t make it easy to set up two different Protocol handlers at different URI prefixes.
When I have needed to run a standard HTTP website along-side a Twisted WebSocket, I’ve organized my site something like the code below.
reactor.listenTCP(80, HTTPFactory()) reactor.listenTCP(8080, WebSocketFactory())
This isn’t exactly what I wanted to do. What I wanted was something that dispatched HTTP requests to entirely different protocol handlers based on the URI prefix.
reactor.listenTCP(80, ProxyFactory( "/mywebsocket/*" = WebSocketFactory(), "/everythingelse/*" = HTTPFactory(), ))
I wanted to have a little proxy factory that could be configured to route a connection to the proper handler. This way, I could write specialized Protocol handlers for special applications (streaming media in my case), and still use the proven parts of Twisted’s Resources for everything else. This way I could drop in either one of the websockets toolkits (txWS or Autobahn) in a Twisted application.
A Proxy Toolkit
I wrote a little toolkit for proxying an incoming TCP connection and dispatching it to one of a collection of Twisted services. The first few packets of the connection are collected for the dispatcher to make its decision. These packets are replayed into the chosen service and then the original connection is spliced into the new service. The service can implement a full-duplex connection like a WebSocket if needed. A service may be out-of-process as well. The toolkit can proxy traffic besides HTTP since it operates at Layer 4 of the OSI model.
You can read about the toolkit here at http://github.com/sheffler/StreamProx. It comes with a couple of examples that show how to use it.
Autobahn recently added a Resource handler. There is a Twisted repository branch adding txWS as a standard resource; it is not released yet.