Sensr Time-Lapse Playback with the DVR View

Sensr.net has been collecting vast amounts of image data from lots of cameras for well over a year now.  Until now, the best way to search and view the images has been with a grid-based view based on a metaphor of a photo gallery.  Recently, Sensr.net added a “DVR” view that uses advanced HTML and Javascript to present a fluid view of an
image sequence.

As Sensr.net receives image streams it applies sophisticated motion-detection algorithms to suppress redundant or irrelevant images.  This helps users during playback by allowing the images from an entire hour to be reviewed in only minutes.  It’s also interesting (and sometimes amusing) to see the shots that Sensr.net captures.

One of our users has been capturing morning walks with her dog using an iPhone.  (See the Lifecasting article for a HOWTO if you’d like to do this yourself.)  Watching what she saw as a slideshow or by single-stepping through the frames using the “Next” and “Prev” buttons is an interesting way to share her experiences!  Follow the link to try it out yourself.

http://sensr.net/zoomgallery/351/dvr.html?interval=hour&reft=1278426317&idx=16

I captured a street scene from a restaurant window while on my annual trip to Provincetown.  (also via iPhone.)  Take a look below.  This sequence is amusing when played with the “PLAY” button.  This one takes a while to load, so you may have to wait a little bit.

http://sensr.net/zoomgallery/345/dvr.html?interval=hour&reft=1278089700&idx=10

Adam Beguelin recorded one of these DVR views as a quicktime movie. (http://www.beguelin.com/2010/07/quicktime-screen-recording.html)  He posted it to Youtube too.  Pretty cool to watch.

These views are less-than full-motion video – they capture a experiences as images sequences that can be played back faster than realtime, or frozen as photos.  I find the time-lapse sequences to be extremely engaging in a way that is different from video.  What do you think?

WebSocket Upgrade Handshake. EventMachine HttpServer and Null characters.

I took advantage of some holiday downtime to experiment with WebSockets a bit.  As of December, Chrome implements HTML5 WebSockets natively.  I think we can expect more implementations shortly.

My experiments were inspired by this blog entry and my own ongoing experiments with EventMachine.  I learned two things from this exercise:

  1. The WebSocket Upgrade Handshake should be sent as a contiguous stream of bytes.
  2. Null characters in WebSockets streams may confuse existing HTTP Protocol handlers.

Upgrade Handshake

The websocket protocol (IETF Spec) is absolutely clear about the fact that the first three lines of the Upgrade Protocol Handshake must match “character-for-character” as laid out in the specification. I wanted to know if these first three lines could be sent as separate frames.  In experimenting with the Chrome implementation I learned that the three lines should be sent as one multi-line string.

Using an EventMachine Connection to listen on a port, I returned the handshake using three separate send_data calls.  Chrome failed to recognize this as a handshake by refusing to report an open connection.

send_data("HTTP/1.1 101 Web Socket Protocol Handshake\r\n")
send_data("Upgrade: WebSocket\r\n")
send_data("Connection: Upgrade\r\n")
send_data("WebSocket-Origin: #{origin}\r\n")
send_data("WebSocket-Location: #{location}\r\n")
send_data("\r\n")

I changed the handshake to the following.  This succeeded in creating an open connection between the browser and server.

upgrade = "HTTP/1.1 101 Web Socket Protocol Handshake\r\n"
upgrade << "Upgrade: WebSocket\r\n"
upgrade << "Connection: Upgrade\r\n"
upgrade << "WebSocket-Origin: #{origin}\r\n"
upgrade << "WebSocket-Location: #{location}\r\n"
upgrade << "\r\n"
send_data(upgrade)

Null Characters

In another experiment, I wanted to subclass EventMachine::HttpServer to modify it to be a WebSocket server. The idea was to detect WebSocket connections and upgrade the connection if a WebSocket is detected. This technique would use the HttpServer header parsing mechanism only and then would get the Request handler out of the way to allow the bytes that are sent in the socket stream to flow freely.

HttpServer has a mode (dont_accumulate_post) that tells it to call method (receive_post_data) on the bytes received in a Request body as they are received, rather than to accumulate the entire body. I reasoned that I could receive bytes in the WebSocket stream by using this option and overriding receive_post_data. This didn’t work. [Yes, I realize all of this is an abuse of HTTP.]

The reason that this didn’t work is that each message in a WebSocket stream starts with a Null (\000) byte. EventMachine::HttpServer is written in C and uses a string function (strpbrk) to do request body parsing. Most of what HttpServer sees is strings that start with Null.

Of course, the WebSockets protocol is not a subset of HTTP exactly, and I shouldn’t expect this to work, but now I know.

I did succeed in subclassing EventMachine::HttpServer to give me the byte streams. Here’s an outline of what I did. This puts a big conditional before the processing of every packet received, but it achives the effect I want. HttpServer parses the headers and I get the byte stream. My existing HTTP header-processing (cookies, sessions) works with my WebSocket stream.

class WsServer < EM::Connection
  include EM::HttpServer

  def post_init
    super
    no_environment_strings
    dont_accumulate_post        # dont read all of the post data
    @header_processing = true
  end

  def process_http_request
    ...
    @header_processing = false
  end

  def receive_data(*args)
    if @header_processing
      super
    else
      receive_post_data(args)
    end
  end

  def receive_post_data(x)
    puts "GOT POST DATA:#{x.inspect}:"
  end
end

Conclusion

The WebSocket protocol superficially looks like a subset of HTTP, and even the terminology “upgrade” suggests that it is only a refinement.  Strictly speaking however, it is a different (but related) protocol.  It will be a while until servers, caches and middleware get the details sorted out.

HTML5 and WebSockets

The next revision of HTML (HTML5) goes far beyond updating text styles and markup. It adds support for media playback, adds a new Canvas object for 2-D drawing, and even supports real-time video compositing by using Javascript to generate image frames from real-time data streams.  HTML5 will help make the web even more interactive and alive than it is now with “Web 2.0” apps.

html5-logo_2

I think one of the most interesting additions is the API for WebSockets. The WebSockets API will let Javascript on a web-page open TCP connections to web-services to send and receive data through internet pipes. A WebSocket enabled Javascript program can also modify DOM elements on the page and draw on a Canvas. With these capabilities, the web-page will become more of a full-fledged interactive application with full-duplex communication to one or more hosts.

AJAX technologies already make web pages feel more interactive, but these pages have restricted capabilities: they can only request information from servers and do not asynchronously receive updates from a data source. “Comet” (cleaning product joke intended) is the name given to a new breed of application that implements asynchronous behavior using existing technologies in an ad-hoc manner. The term “Comet” encompasses a variety of techniques that WebSockets will help simplify.

Right now, a developer can implement something like WebSockets using Java or Flash, but both of these have the drawback that they require the user to install a runtime. Some people have been able to implement something like “server-push” using a technique called “forever frames” in which raw Javascript is piped to a browser. In other applications, server-polling suffices to give the illusion of live data.  None of these techniques satisfy the goals of being truly portable and robust with zero-install overhead.

The idea with HTML5 WebSockets is to unify all of these approaches and put the functionality in Javascript. If successful, this new capability will enable real-time live apps that live in the browser. It may even transform web pages from clients into servers. Imagine starting a home-automation server by opening a web page: that’s not the way we think of the browser right now.