Running one of the largest websites on the internet with about 5 million unique sites hosted exposes you to all sorts of issues. There are constant events to deal with, some internal, some external. This morning, one of the more common external events, a Distributed Denial of Service Attack occurred. We experience these types of attacks rather frequently, but most are easily mitigated and have no user impact. One this morning, however, was rather large and thus impacted some users.
Here is a timeline and description of this morning’s events:
9:40 AM EST — Our internal monitoring systems alerted us to unusual activity in one of the four geographically diverse datacenters which serve WordPress.com traffic. Here is what that anomaly looks like in graphical terms:
10:00 AM EST — The target of the attack was identified and removed from our network. The attack, however continued. This is because the attacker had hijacked tens of thousands of computers (probably by installing a virus which was spread via email) and these computers had no idea the site was no longer there. A small log sample shows over 8 million requests for this one site from over 10,000 unique IP addresses.
10:20 AM EST — Since we have servers in multiple data centers throughout the United States which serve traffic for WordPress.com all the time, we were able to route all legitimate traffic out of the affected data center, and let the single affected data center deal with the attack.
11:30 AM EST — The IPs targeted in the attack were null routed at this point which allowed us to bring all datacenters back online to serve normal traffic.
We keep hourly traffic metrics and based on those numbers, it looks like during the attack there was about a 5% decrease in overall pageviews during the 40 minutes before traffic was re-routed. All things considered, not a bad outcome for an attack this size. Looking at bandwidth graphs, this attack was in the 500Mbit – 750Mbit/sec range.
Demitrious has a great post explaining how we are using S3, Varnish, and Pound to serve 60 million image requests per day on WordPress.com
UPDATE: Almost forgot, but Matt reminded me, he has a really super duper awesome post about WordPress.com and S3 too!
At 10:56:22PM PDT on 5/23/2007, the 1 millionth active blog was registered on WordPress.com. And the winner is…..
Not much there right now, but hopefully there will be soon. Maybe head over and leave a comment on their about page to let them know!
Predictions on how long it will take to get to 2 million?
So, I haven’t blogged much lately but there is a reason. Over the past month we have been hard at work expanding the infrastructure behind WordPress.com and Akismet. Here are some of the things that we have done over the past month or so:
- Migrated out of San Diego
- Brought online almost 100 new servers in 3 new datacenters — Dallas, TX, San Antonio, TX, and San Francisco, CA
- Tripled the database hardware behind WordPress.com
- Now serving WordPress.com blogs out of 3 datacenters in real-time
- Akismet is now served from 2 datacenters
Here are a couple pictures of some new hardware racked and powered on just before we put it into production last week.
From top to bottom (left):
- 21 x HP DL145
- 4 x HP DL365
From top to bottom (right):
- 18 x HP DL145
- 4 x HP DL365
- 1 x 3U HP Storage Array
- 1 x HP DL385
And the back….
Thanks to Evan League and Brian Maples of Layered Tech for doing the build-out pictured above and sending the photos over.
We are getting ready to place an order for an additional 37 servers in a new datacenter. This new point of presence will serve as the 3rd active node for WordPress.com. Over the past few weeks, I have been doing lots of testing and seemingly endless negotiation with various hosting companies.
The model we have adopted is to use commodity hardware to serve all the functions of the site. We do not rely on SANs or super-expensive multi-processor systems. Our web servers are usually either single or dual processor machines with 1-2GB of RAM and a small, inexpensive hard drive. Our database servers are single or dual processor machines with 4-8GB of RAM and 2-4 fast SCSI drives in a RAID array using a hardware RAID controller. Because there is redundancy built into the architecture that several of these machines can fail at once at the site is unaffected, the individual machines do not need to be extremely robust. Historically, CPU time has been the most precious resource on the web servers and disk I/O on the DBs.
- Provide the hardware outlined above
- Support Debian AMD64 or similar
- Provide a Gigabit private backend network for inter-server communication
- Ability to deploy additional servers quickly and painlessly
- Sales and support team that is competent and easy to work with
- US Datacenter and not Dallas or San Diego
Dedicated server providers seem to fall into 2 classes:
1) No-frills provider that offers server between $79 – $149/month. No phone support or advanced services. This would be fine, but they usually cannot provide the DB-class machines we need and do not provide Gigabit backend networks.
2) Full-service provider that offers servers beginning at $250/month and up. These providers are usually not the best fit for us because they justify their higher prices by saying they have superior support. We don’t really need “superior” support, just someone that can take care of hardware issues when needed. Seems like a waste to pay for something you are never going to use and don’t really need. Also, in my experience, the more expensive the hosting company, the more painful the sales process is. Sometimes I feel like I am buying a car….
Over the past month, we have narrowed the field from about 10 different possible providers down to the following 3. Each of these fit somewhere between no-frills and full-service as mentioned above, but I definitely get the feeling that they lean one way or the other.
Based in San Francisco, California (where Automattic is also based) they seem to lean towards the full-service side, offering 24×7 phone support, higher-end servers, load balancers, and firewalls. Their sales process has been pretty agonizing. It is now going on 45 days of back and forth, price changes, configuration changes, conference calls, and about 50 one-on-one calls with our sales guy. I had chance to visit their offices and tour their datacenter earlier this week so I got a feel of how things work there — it appears to be a well-run organization. One thing that struck me as a bit odd was that they do not deploy any rack-mounted servers in their datacenter. All of their servers, which are built in house, are in tower (white-box) chassis like you would see under an office desk. This is something I would expect in the lower-end market, but at $500+ per month and rather large RAID arrays (300GB SCSI x 6) I would expect to see more rack-mounted chassis to take advantage of the superior cooling.
- Server Beach
Based in San Antonio, Texas and now owned by Peer1, Server Beach is definitely more on the no-frills side. When we were looking at new datacenters about 6 months ago, Server Beach could not provide what we wanted — they did not offer SCSI RAID or Gigabit privatenet, but said it was coming. Well, those things are now available. Kudos to Server Beach on a super-simple and painless sales process. Once I contacted them with the configuration we wanted, they scheduled a conference call to discuss the details. They had members of their support, operations, and sales teams on the line. About 30 minutes later we were all done and a day later they had sent over a proposal. It was right the first time – had everything we asked for and at a very fair price.
Honestly, we probably will not go with TextDrive for this deployment, as it is a pretty radical departure from our current configuration, and they don’t technically meet the requirements laid out above. They are worth mentioning, however, because they are doing some pretty cool stuff with OpenSolaris, zones, and ZFS, and it sounds like it could be a good fit for our architecture model. Their storage is super-fast, the container model allows you to replicate existing containers with a single command — there is no need to provision and setup a new physical server. ZFS offers all sorts of cool stuff like snapshots, compression, and built-in data consistency checks. Utilizing this architecture, would require that we maintain 2 completely separate environments — Linux and Solaris — and there is a definite time investment in doing so. I think that we will probably look at moving some of our services to Solaris containers in the near future, but I am not sure it will be WordPress.com.
First, let me say that we still have yet to make the final decision, but hope to do so by the end of the week. Both ServePath and Server Beach seem like they will be great companies to work with. ServePath is local, so we can walk over to their offices and meet with them if needed – there is something to be said for working with local vendors. Server Beach has been a pleasure to work with thus far and I have some experience working with them in the past. Peer1’s VP of Marketing also blogs on WordPress.com. Pricing and server configurations are almost identical, so that really isn’t as much of a factor as one would think.
Anyone have experience working with either of these companies? Suggestions? Feedback?
WordPress.com has a new look this evening! Big ups to Matt Thomas who did an awesome job on the design.
As a follow up to Matt’s September wrap up post, I thought it would be fun to post some more technical stats about the current WordPress.com infrastructure:
- 80 physical processors
- 139GB of RAM
- 91 hard drives with a combined total of over 15 terabytes of storage space
- 2000 database queries per second spread over 40 MySQL instances
- Over 8 million objects stored in memcached serving over 8000 requests per second
The best part is that this is just the beginning! There are many more exciting things to come. Next up — expansion to 2 additional datacenters in the US.