Making Gravatar fast again

As Matt blogged, Automattic recently purchased Gravatar. The first thing we did was move the service onto the infrastructure. Since the application is very different from what this really means is using what we have learned from scaling to increase both speed and reliability of the service, as well as leveraging our existing hardware and network infrastructure to stabilize the service. The current infrastructure is laid out as follows:

  • 2 application servers (in 2 different data centers for redundancy). One of these servers primarily handles the main Gravatar website which is Ruby on Rails while the other serves the images themselves. If either of these servers or data centers were to fail, we could easily switch things around to work around the outage.
  • 2 cache servers (1 in each datacenter). These servers are running Varnish. They cache requested images for a period of 10 minutes, so frequently requested images are not repeatedly requested from the application servers. We are seeing about a 65% cache hit rate and about 1000 requests/second at peak times, although as adoption of the service increases, we expect this number to go up significantly. A single server running Varnish can serve many thousands of requests/sec. The amount of data we are caching is small enough to fit in RAM, so disk I/O is not currently an issue.

On the hardware side, for those of you who are curious, we are using HP DL365s for the application servers, and HP DL145s for the caching servers. 4GB of RAM and 2 x AMD Opteron 2218s all around. The application servers have 4 x 73GB 15k SAS drives in a RAID 5, while the caching servers are just single 80GB SATA drives. We use the same hardware configurations extensively for and they work well.

Previously, the service was using Apache2 + Mongrel to serve the main site and lighttpd + mod_magnet to serve the images. We decided to simplify this and we are currently using lighttpd to serve everything and it is working well for the most part. We seem to have a memory usage issue with lighttpd, which may be related to this long-standing bug.  For now, we are just monitoring memory usage of the application with monit, and restarting the service before memory usage gets too high.

Virgin America review

This weekend I flew to New York City and decided to try Virgin America. They received some press when they launched mostly around the cool lighting and seat-back computers. Here is my review:

Service: A
Online check-in was painless and everyone was super nice. The flight crews even have a sense of humor.

Comfort: B
The seats, although leather, aren’t more comfortable than any other airline seat I have been in. The leg room is comparable with airlines such as JetBlue and Frontier. I would say United and US Airways have a little less legroom. Having power outlets under the seat is great, although I didn’t use them since there isn’t WIFI yet.

Technology: C
Although the seat-back computer systems (called “Red”) are cool and full of potential, they are currently full of bugs. My experience included the following:

  • Most of the TV stations don’t work at all. On the flight back, the flight attendant said they are scheduled to fix them in an update next month.
  • Slowness — on screen display is very slow and lags behind what you are trying to do, making the touch screen hard to use. Most games are not playable because of the lag.
  • On the way to NYC, half of the computers on the plane (including mine) didn’t work at all for the first part of the flight. After a reboot, they started working again.
  • On the way back, my remote stopped working half way through the flight.
  • The TV is on a 4 hour loop so on a 6 hour flight (NYC –> San Francisco) you get to watch everything 1.5 times.

Food: B
No free food or snack provided in coach on the transcontinental flight. Both are available for purchase, although the selection is very sparse. On the “food” side, you can choose from a sandwich or a cheese and fruit plate. I think each was around $7 or $8. What is really cool is that you order your food and drinks via the computer in front of you. Simply navigate the touch screen to select what you want, add it to your shopping cart, swipe your credit card, and it is delivered to you within minutes. This is how you order both paid for (food/snacks) and free (soda) stuff. This seems WAY more efficient and I suspect most airlines will start doing this at some point.

Price: A-
Pricing is in-line with the other major carriers, but the routes are very limited. There is a surcharge if you want to sit in an exit row or in the first row behind 1st class because there is more legroom. On the flight from NYC to SFO, the surcharge was $25 which was almost 20% of the one-way fare. That seems steep to me.

The Verdict
I have already booked another flight on Virgin America, albeit a much shorter one (SFO to LAS), but I think for the longer flights I will stick with Jet Blue for now until Virgin America can work out the bugs with their TV system.

%d bloggers like this: