An EPIC Twitter hack to analyze tweets in real-time

The NBA is synonymous with players having clutch moments and being the talk of everyone, especially on Twitter.

With many talented athletes on NikeBasketball’s roster, EPIC visualizes those moments in real-time by ranking the NBA players according to each one’s tweets per hour calculation.

Coming up with an accurate tweets per hour calculation for each player required many calls to Twitter’s Search API, a huge database, and lots and lots of cron jobs running throughout the day, every day.

To start, a unique table was set up for each player. Each player also had a couple fields reserved for meta data to take into account when searching for tweets about them, such as aliases or their Twitter usernames. With so many athletes to track tweets for, and in order to keep in line with Twitter’s Search API rate limiting, it was necessary to break up the athletes into different groups, sorted by popularity: “A”-priority players, the most popular and active ones on Twitter, “B”-priority players who are less active on Twitter, and “C”-priority players who make up the bulk of the league and will be tweeted about least frequently at all. Then, throughout the course of the day, and with varying frequency according to each priority group, a script would run through all the players in that list. For each player, tweets would then get surfaced up by making queries to Twitter’s Search API, and then those tweets would get stored into the player’s respective table in the database.

During live games it was a little different. The script would run more frequently through all the athletes on each team currently playing, regardless of the priority group the player was in, and mine the tweets. Doing so allowed for even more accurate calculations based on the athlete’s performance during the game.

Meanwhile, on top of the tweet mining, various jobs would run at all times to accurately re-calculate tweets per hour and other figures based on the constant influx of Twitter data coming in.

In the end, one of the most rewarding things of seeing the system work was when a lesser-known athlete would have a clutch moment and catapult themselves into the top rankings of EPIC. San Antonio Spurs guard Gary Neal had his moment in Game 5 of the Spurs / Grizzlies series. With the Spurs down in the series 3-games to 1, Neal made a clutch game-tying three-pointer at the buzzer to force the critical game for the Spurs into overtime, and an eventual win to bring the series to 3-2. Gary Neal, once in the middle rankings of EPIC, went ahead of household names like Kevin Durant and Dirk Nowitzki at that moment from all people on Twitter talking about his epic clutch shot.