Free Day 2012 Recap
Let's go over the numbers for Free Day 2012
Another January, another Free Day come and gone. Win or lose, we hope everyone had fun, we sure had a lot of fun in the building here putting it on!
Numbers
Let's go over some numbers, then I'm going to write a bit about how we managed to pull it off this year.
- 6,437,592 Correct reCaptcha solves - This is the number of people who filled out a reCaptcha successfully. This does not count unsuccessful attempts, including those made by people, automated bots or just those pressing "GO" over and over again with a blank box.
- Supposing the average user takes 4 seconds to fill out a Captcha, there were 7188 hours spent solving Captchas, that's almost ten months!
- Every Captcha passed through the SparkFun servers - at a rate of just over 300 per second for the 355 minutes that we were actively giving money away.
- 32,137 unique visitors to /free_day/ as reported by Analytics. This loosely represents how many people took at shot at winning.
- Dividing those, we get that the average user filled out 208 Captchas each. That's a lot of garbled words!
- 704,617 Pageviews on the site over the course of the day.
- Of the 2,000 promotional codes given away, 473 codes were redeemed on free day. Almost a quarter of y'all just couldn't wait to spend your bounty! In fact, the first local pickup order from Free Day winnings was at 10:45, less than two hours after the event began.
- We gave out promo codes at a rate of 5.6 winners per minute.
- We had 7,776 new accounts created on Free Day.
Numbers for the Nerds
- At its peak, our MySQL server was serving up 13,380 queries per second.
- Throughput in our network peaked at 1,214.35Mb/s.
- Bandwidth from our data center peaked at just over 76Mb/s. (See note about cloudfront below)
- At its peak, we had 84,220 active TCP connections.
- We blocked 32 bots from actively participating in Free Day.
- We took down one external site by sending them too much traffic. (Sorry, Pachube!)
The actual setup - the cap was removed from the Geiger tube during Free Day
How We Did It - For the Nerds
If you have experience with the first two Free Days, you might have noticed that this one went quite a bit smoother than in years past. Aside from an initial hiccup while traffic ramped up a full order of magnitude over a span of minutes, the site was responsive and snappy for the next hours as we dealt with the insane amounts of traffic. So what changed from years past?
We put to use some new software and hardware in our web stack. We now have two beefy web servers each with 16 cores and 32 GB of RAM, and a database server with double the RAM. In front of the web servers, we have redundant hardware load balancers to divvy up traffic to the two machines.
In addition to optimizing our own code base to run more efficiently on the hardware we've got, we changed some of the open-source software we use to run things. We jumped ship from Apache in favor of Nginx to serve web pages, with PHP-FPM doing the heavy lifting behind it. In front of everything, we are now running Varnish to cache pages where we can - which proved invaluable to putting on a successful Free Day with the traffic we saw.
When tracking down bugs from last year, we noticed that MySQL was a culprit more often than not when it came to slow-downs. In some places, we've replaced the use of MySQL with the document store MongoDB. For some data, including some hierarchichal data, it just made sense. This is one reason we can keep the page generation time for pages like yesterdays post, with over a thousand comments, so quick. Unfortunately, due to a small bug in Mongo's PHP driver, we experienced an initial slowdown until we could isolate the problem.
In the past, we served all assets from our own servers. During normal load, this was fine - pages were snappy and our bandwidth wasn't too outrageous. Of course, during Free Day everything changes :) Last year we were pushing much more bandwidth because we were serving all static content ourselves. Since then, we've changed things around to use Amazon's Cloudfront. Cloudfront is a global CDN that sits in front of the ever-popular S3 storage service that can deliver content to users from edge servers all over the world - and keep the bandwidth on their infrastructure instead of our own. This does put the bandwidth number above in perspective, though. The 76Mb/s we were pushing through our data center was almost entirely just gzipped text! That's a lot of bits!
Geiger Counter?
Any computer scientist will tell you that random numbers that computers come up with are pseudorandom, that is they are still bound by their nature to be predictable. Even if there's a 1 in a million chance (or more), it's a chance nonetheless and can therefore be predicted beforehand. What, then, can we do to generate a completely random bidding system?
We decided on a Geiger tube to measure background radiation. The Geiger tube pulls a pin low every time it detects a piece of radiation, which for all intents and purposes is completely random and entirely unpredictable. We set up an Ethernet Pro to watch for these interrupts, and send a byte to a daemon running on one of our web servers to alert that the next participant was a winner.
Here's the Arduino code:
And for completeness, here's the bit server that Ben wrote in C to handle the extreme load of handling the hundreds of requests per second we saw on Free Day. He benchmarked it to something like 600,000 requests per second.
Conclusion
Love it or hate it, Free Day has come and gone another year. We in IT think it was success in that we were able to give away the money, and even under our own terms. We were aiming to stretch it out a bit more this year, rather than have it a flash in the pan and over with in just a few minutes. To that end, it was successful, and completely random.
I hope this gives some insight as to what goes on in the IT room at SparkFun during (our) busiest day of the year.
To all that won, congratulations! To those that didn't, thanks for trying and we all hope you had fun.
