Evan Savage // Blog Archive

Fitbit: APIs, crossfilter, and d3.js

In this post, I present fitbit-crossfilter, which uses the Fitbit API, crossfilter and d3.js to provide an interactive visualization for exploratory analysis.

The Inspiration #

It was early April 2012. By this point, I'd been through a stint of pen-and-paper self-tracking for panic recovery. I'd just received my Fitbit in the mail.

Earlier that year, I applied to the EECS PhD program at UC Berkeley with this statement of purpose. I was fascinated by this idea that pervasive gameplay really could make us all better, that somewhere beyond the rat wheel of gamification was hidden a Shangri-La of game-driven awesome.

That unfortunately didn't pan out, and I was left with the age-old question:

What do I do with this idea?

It was around this time that, in a moment of exquisite digital serendipity, Meetup suggested I check out the Bay Area Quantified Self Meetup Group.

Quantified Self? What's that?. As I explored the group page, I felt a rush of clarity: this was exactly what I'd been doing! There's a whole community of people turning their lives into games in the name of self-betterment!

I bit the bullet and forked over hard cash to sign up for QS Show&Tell #25 at the California College of the Arts. It was everything I'd hoped for. One presenter dissected 30 years of medical data and correlated it with his marital status. Another showed off a cyclist threat detection system cobbled together by mounting a webcam and sonar unit to his handlebars. There was a rich vein of inquiry into awesome here. I was hooked.

Beau Gunderson of Singly presented zeo-crossfilter. That was the turning point. I saw what he had done and said

Hey, I can build that!

And so fitbit-crossfilter was born.

The Tools #

As mentioned, fitbit-crossfilter is a mashup between the Fitbit API, crossfilter, and d3.js. I'll go over each part with examples.

Fitbit API #

The Fitbit API uses OAuth for authentication. If you've never confronted OAuth before, it can be confusing. To compound the confusion, every API provider seems to do it slightly differently. The official Fitbit docs are opaque, the OAuth specs are even more opaque, and the unofficial apis.io listing is just wrong:

$ curl -X GET -u '<username>:<password>' http://api.fitbit.com/1/user/-/profile.json 2>/dev/null | jsonpp
{
  "errors": [
    {
      "errorType": "oauth",
      "fieldName": "n/a",
      "message": "No Authorization header provided in the request. Each call to Fitbit API should be OAuth signed"
    }
  ]
}

I turned to oauth2, a Python library that makes it easier to carry out this handshake. First, we get a temporary access token:

# Fill in your app parameters here.
FITBIT_APP_KEY = '<app key>'
FITBIT_APP_SECRET = '<app secret>'

import oauth2
consumer = oauth2.Consumer(key=FITBIT_APP_KEY, secret=FITBIT_APP_SECRET)
client = oauth2.Client(consumer)
resp, content = client.request('http://api.fitbit.com/oauth/request_token, 'GET')
token = oauth2.Token.from_string(content)
# NOTE: the auth URL uses www.fitbit.com as the domain, NOT api.fitbit.com
auth_url = 'http://www.fitbit.com/oauth/authorize?oauth_token={0}'.format(token.key)
print auth_url

Now we need an OAuth verifier. This will be used to retrieve the real access credentials. Visit auth_url in your browser, log into Fitbit, and click Allow. You'll be redirected to the OAuth callback specified in your app. Use the value of the oauth_verifier GET param on your token from before to keep going:

token.set_verifier('<oauth_verifier>')
client = oauth2.Client(consumer, token)
resp, content = client.request('http://api.fitbit.com/oauth/access_token', 'POST')
access_token = oauth2.Token.from_string(content)

With this, we can now retrieve useful information:

request_url = 'http://api.fitbit.com/1/user/-/profile.json'
oauth_request = oauth2.Request.from_consumer_and_token(consumer, token=access_token, http_url=request_url)
# Despite what the docs say, you need to generate a plaintext signature.
oauth_request.sign_request(oauth2.SignatureMethod_PLAINTEXT(), consumer, access_token)
headers = oauth_request.to_header(realm='api.fitbit.com')

import httplib
connection = httplib.HTTPSConnection('api.fitbit.com')
connection.request('GET', request_url, headers=headers)
resp = connection.getresponse()

import json
data = json.loads(resp.read())

I encountered a few difficulties in figuring this out:

You can see the full implementation here, along with an example of its use.

crossfilter #

Square's crossfilter is a JavaScript library for efficiently performing multidimensional range queries. I've included an interactive example below.

crossfilter uses two types of objects to represent a multidimensional dataset:

The totally-ordered part is essential, since that makes it possible to perform range queries. A quick code snippet might help explain this further:

var L = [], N = 10, M = 2;
for (var i = 0; i < N; i++) {
  L.push([i, Math.floor(M * (N - i - 1) / N)]);
}
var c = crossfilter(L);
var d0 = c.dimension(function(x) { return x[0]; });
var g0 = d0.group();
var d1 = c.dimension(function(x) { return x[1]; });
var g1 = d1.group();
d0.filterRange([3, 8]);

At this point, we can inspect the dimensions and groups to understand the effect of filterRange():

> JSON.stringify(d1.top(Infinity))
'[[4,1],[3,1],[7,0],[6,0],[5,0]]'
> JSON.stringify(g1.all())
'[{"key":0,"value":3},{"key":1,"value":2}]'

Note that the range [3, 8] is actually interpreted as the semi-open interval $ [3, 8) $. Note also that the elements of g1.all() are of the form {key: k, value: v} where v is the number of elements x with 3 <= x[0] && x[0] < 8 && x[1] == k.

d3.js #

D3.js is a JavaScript library for manipulating documents based on data.

Using HTML, SVG, CSS, and JavaScript, you can build some pretty stunning visualizations. Again, check out the interactive example below. For more examples, the D3 Gallery is many kinds of awesome.

A Quick Demo #

If you're viewing this through an RSS reader, the above demo of crossfilter won't show correctly. You can view it on my blog.

Insights From My Data #

How To Use fitbit-crossfilter #

First, you will need a Fitbit app with Partner API access; see this page for more details on setting that up. Use the following application settings:

Now copy settings.py.nopasswd to create your settings file:

$ cp settings.py.nopasswd settings.py

Edit the bottom of settings.py:

SYNC_ENABLED = True
DEFAULT_USER = None
FITBIT_CONSUMER_KEY = <your app key>
FITBIT_CONSUMER_SECRET = <your app secret>

Start the server, login, and sync your data:

$ python manage.py runserver 9001
# visit localhost:9001/login in the browser to do the OAuth handshake
# visit localhost:9001/sync-user-data in the browser to sync data

When the syncing completes, you'll be redirected to your dashboard.