On November 21 I was pleased to participate in a meetup entitled Real-Time Streaming Data. The organizer of this meetup assembles a wide variety of presenters and topics under the umbrella topic of “Large-Scale Production Engineering.” Chris (the organizer) does a remarkable job of keeping a pipeline of interesting talks coming. I’m particularly interested in the January talk humorously entitled “Whatever happened to IPV6?”
At this meetup, there was one lightning talk and three longer talks. The first, entited “Realtime by proxy” was delivered by Justin Kerneges of Fanout IO. Justin described Pushpin, a reverse-proxy server that helps to write streaming services. The Pushpin proxy communicates to a backend web application using regular HTTP requests. The proxy itself, implements a new Content-Type (“application/grip-instruct”) that is routed to a long-polling request in an application or browser-page application. What Justin described was a way to reuse much of your existing framework to add realtime-ness in an incremental way.
You can see the Github of the repository here. https://github.com/fanout/pushpin
Next up was a talk about DataTorrent: a platform for real-time production operations. DataTorrent is a Hadoop application for realtime stream processing. Using DataTorrent, an algorithmic stream operation is translated into a DAG (Directed Acyclic Graph) of
operators, each of which is mapped onto a node in a Hadoop cluster. The application focus is realtime systems monitoring. The presenters showed a demonstration of a dashboard view of their compute center with realtime updates of loads and response times.
My own talk was about an application that delivers segmented-audio packets to a one-page browser application that renders audio using Audio tags or the newer AudioContext. The problem to be solved is that in the rendering of live audio, there is a realtime delivery deadline for audio packets, after which the audio packet is no longer relevant. During periods of network congestion, audio packets can be delayed, and if they are delayed too long they are no longer needed. A smart audio server needs to be able to monitor the network congestion and adapt. Autobahn websockets makes this possible.
The last talk was about PubNub by their CTO Stephen Blum. PubNub is a distributed network for routing and managing Push notifications. Stephen talked a little bit about how they route messages around their global network and how you use it. I was most impressed with the fact that their service is implemented by a distributed global network and that the issues of redundancy and resiliency have been considered from
the start. PubNub helps developers write realtime applications by taking care of the infrastructure for routing messages and guaranteeing their delivery.
It was all in all a very interesting evening, and the questions from the audience and discussions afterward showed a very engaged gathering of almost 200 professionals. I’ll look forward to attending more of these in the future.