Live Site Traffic map Last Updated Friday, June 4, 2010

 

Origin

Infographics are one of Constant Contact Labs’ big areas of interest, and when we encounter interesting data sets we can’t help but experiment with them. The Live Site Traffic map is a great example - it began as a side exploration into data garnered from an internal network monitoring project, but ended up presenting such an arresting visual that it took on a life of its own.

Labs had been experimenting with a live stream of logging information coming from one of Constant Contact’s data centers, and we noticed that one of the fields coming over the connection was dedicated to geocoding site visitors’ IP addresses. When we had a few spare minutes, we headed to Google Maps and manually looked up some of the latitudes and longitudes that were scrolling past. Interestingly, the first one we looked up originated in the caribbean! This was a bit unexpected. Maybe, we thought, other people in Engineering would think this data was cool, too.

Architecture & Development

Geocoding IPs can be a time- and energy-intensive process, so we were pleasantly surprised to find that our data center monitoring devices were doing the work for us. Our friends in Operations poked a bit deeper and figured out that it was a lookup-table-driven coding system, which explained the speediness. With the pre-coded data stream available, we merely needed to create a distribution and display infrastructure. Great! But how would we deploy a map that streams well over a thousand data points per second without bogging down the monitoring device?

Fortunately, concomitant work on the main network monitoring project had given us a Java-based metrics server that sat in our Network Operations Center. The metrics server connected securely to a few different data sources, including our geocoding source, and then made the aggregated data available to interested LAN clients over a simple SSL socket connection. We made sure to sneak the geocoding fields into the metrics server’s data aggregation, and then were able to write a metrics client to connect, request a geocoded data stream, and display the results. This approach meant that the metrics server could make just one connection to the remote data center, and then share the results with as many clients as we cared to run over the speedy local LAN.

With our Security and Network Operations folks happy, we moved on to writing the Live Site Traffic visualization itself. We chose Processing, a Java-based graphics programming language originally from MIT, as our weapon of choice. (Jim usually does, given the option.) Processing provided for cross-platform deployment, which meant that we could run our map on both Macs and Windows machines, and one of its renderers was OpenGL-based, which let us tap the rarely-used power of our workstation graphics cards. The OpenGL renderer was the critical component that let us provide a high-def (720p), 30+ fps, smoothly zooming world map with thousands of animated data points.

Deployment

The Live Site Traffic map made a quiet debut in September 2009 on a TV mounted in the Engineering kitchen. Partly this was due to it being a nerdy little display; partly this was because we weren’t sure how stable the code was, and we wanted to keep an eye on it. After a few days it began to develop a reputation around the office as a neat little idea. We were surprised and delighted - we made something that appealed to non-engineers!

Current State

We’re writing this in February 2010. As Constant Contact has continued its renovations in Waltham, more new TVs have been added to more new kitchens, and we’ve been asked to spread instances of the map around the offices. Our decision to use the metrics server instead of connecting each map instance to the data center directly has proven to be a smart one: we now have about as many map instances running concurrently as we do sockets available on the data center monitoring device.

We’re not actively developing the Live Site Traffic map right now, but we are keeping track of feature requests and general feedback. We’re also taking the continued interest as a sign that there’s plenty more we can do to engage people using custom information visualization. The Live Site Traffic map has given us a number of additional ideas for ways to use the office TVs, and we hope to find time soon to head off on another interesting visualization tangent.

Demo

You can find a demo video of the Live Site Traffic map on YouTube:

And if you’re in the area, you can find the real thing in the reception area of our newly renovated offices.