So, I haven’t blogged much lately but there is a reason. Over the past month we have been hard at work expanding the infrastructure behind WordPress.com and Akismet. Here are some of the things that we have done over the past month or so:
- Migrated out of San Diego
- Brought online almost 100 new servers in 3 new datacenters — Dallas, TX, San Antonio, TX, and San Francisco, CA
- Tripled the database hardware behind WordPress.com
- Now serving WordPress.com blogs out of 3 datacenters in real-time
- Akismet is now served from 2 datacenters
Here are a couple pictures of some new hardware racked and powered on just before we put it into production last week.
From top to bottom (left):
- 21 x HP DL145
- 4 x HP DL365
From top to bottom (right):
- 18 x HP DL145
- 4 x HP DL365
- 1 x 3U HP Storage Array
- 1 x HP DL385
And the back….
Thanks to Evan League and Brian Maples of Layered Tech for doing the build-out pictured above and sending the photos over.
We are getting ready to place an order for an additional 37 servers in a new datacenter. This new point of presence will serve as the 3rd active node for WordPress.com. Over the past few weeks, I have been doing lots of testing and seemingly endless negotiation with various hosting companies.
The model we have adopted is to use commodity hardware to serve all the functions of the site. We do not rely on SANs or super-expensive multi-processor systems. Our web servers are usually either single or dual processor machines with 1-2GB of RAM and a small, inexpensive hard drive. Our database servers are single or dual processor machines with 4-8GB of RAM and 2-4 fast SCSI drives in a RAID array using a hardware RAID controller. Because there is redundancy built into the architecture that several of these machines can fail at once at the site is unaffected, the individual machines do not need to be extremely robust. Historically, CPU time has been the most precious resource on the web servers and disk I/O on the DBs.
- Provide the hardware outlined above
- Support Debian AMD64 or similar
- Provide a Gigabit private backend network for inter-server communication
- Ability to deploy additional servers quickly and painlessly
- Sales and support team that is competent and easy to work with
- US Datacenter and not Dallas or San Diego
Dedicated server providers seem to fall into 2 classes:
1) No-frills provider that offers server between $79 – $149/month. No phone support or advanced services. This would be fine, but they usually cannot provide the DB-class machines we need and do not provide Gigabit backend networks.
2) Full-service provider that offers servers beginning at $250/month and up. These providers are usually not the best fit for us because they justify their higher prices by saying they have superior support. We don’t really need “superior” support, just someone that can take care of hardware issues when needed. Seems like a waste to pay for something you are never going to use and don’t really need. Also, in my experience, the more expensive the hosting company, the more painful the sales process is. Sometimes I feel like I am buying a car….
Over the past month, we have narrowed the field from about 10 different possible providers down to the following 3. Each of these fit somewhere between no-frills and full-service as mentioned above, but I definitely get the feeling that they lean one way or the other.
Based in San Francisco, California (where Automattic is also based) they seem to lean towards the full-service side, offering 24×7 phone support, higher-end servers, load balancers, and firewalls. Their sales process has been pretty agonizing. It is now going on 45 days of back and forth, price changes, configuration changes, conference calls, and about 50 one-on-one calls with our sales guy. I had chance to visit their offices and tour their datacenter earlier this week so I got a feel of how things work there — it appears to be a well-run organization. One thing that struck me as a bit odd was that they do not deploy any rack-mounted servers in their datacenter. All of their servers, which are built in house, are in tower (white-box) chassis like you would see under an office desk. This is something I would expect in the lower-end market, but at $500+ per month and rather large RAID arrays (300GB SCSI x 6) I would expect to see more rack-mounted chassis to take advantage of the superior cooling.
- Server Beach
Based in San Antonio, Texas and now owned by Peer1, Server Beach is definitely more on the no-frills side. When we were looking at new datacenters about 6 months ago, Server Beach could not provide what we wanted — they did not offer SCSI RAID or Gigabit privatenet, but said it was coming. Well, those things are now available. Kudos to Server Beach on a super-simple and painless sales process. Once I contacted them with the configuration we wanted, they scheduled a conference call to discuss the details. They had members of their support, operations, and sales teams on the line. About 30 minutes later we were all done and a day later they had sent over a proposal. It was right the first time – had everything we asked for and at a very fair price.
Honestly, we probably will not go with TextDrive for this deployment, as it is a pretty radical departure from our current configuration, and they don’t technically meet the requirements laid out above. They are worth mentioning, however, because they are doing some pretty cool stuff with OpenSolaris, zones, and ZFS, and it sounds like it could be a good fit for our architecture model. Their storage is super-fast, the container model allows you to replicate existing containers with a single command — there is no need to provision and setup a new physical server. ZFS offers all sorts of cool stuff like snapshots, compression, and built-in data consistency checks. Utilizing this architecture, would require that we maintain 2 completely separate environments — Linux and Solaris — and there is a definite time investment in doing so. I think that we will probably look at moving some of our services to Solaris containers in the near future, but I am not sure it will be WordPress.com.
First, let me say that we still have yet to make the final decision, but hope to do so by the end of the week. Both ServePath and Server Beach seem like they will be great companies to work with. ServePath is local, so we can walk over to their offices and meet with them if needed – there is something to be said for working with local vendors. Server Beach has been a pleasure to work with thus far and I have some experience working with them in the past. Peer1’s VP of Marketing also blogs on WordPress.com. Pricing and server configurations are almost identical, so that really isn’t as much of a factor as one would think.
Anyone have experience working with either of these companies? Suggestions? Feedback?
WordPress.com has a new look this evening! Big ups to Matt Thomas who did an awesome job on the design.
As a follow up to Matt’s September wrap up post, I thought it would be fun to post some more technical stats about the current WordPress.com infrastructure:
- 80 physical processors
- 139GB of RAM
- 91 hard drives with a combined total of over 15 terabytes of storage space
- 2000 database queries per second spread over 40 MySQL instances
- Over 8 million objects stored in memcached serving over 8000 requests per second
The best part is that this is just the beginning! There are many more exciting things to come. Next up — expansion to 2 additional datacenters in the US.
Over the past week here at wordpress.com, we have been doing some experimentation with various software load balancers. In the latest test, we are using Pound with a weighted round-robin algorithm between all of the web servers in a given datacenter. The individual weight takes into account hardware differences and any other tasks that server may be doing. So, a dual processor server would receive more traffic than a single processor machine. The tests have been going really well so far. Once the testing is complete and we have decided on a solution, I will post about the various things we tried, pros and cons of each, etc. The graph below shows the load averages of the various web servers over a one week period. I’ll let you guess when the test began 🙂 Pretty dramatic difference, eh?