Barry on WordPress

We are getting ready to place an order for an additional 37 servers in a new datacenter. This new point of presence will serve as the 3rd active node for WordPress.com. Over the past few weeks, I have been doing lots of testing and seemingly endless negotiation with various hosting companies.

Background

The model we have adopted is to use commodity hardware to serve all the functions of the site. We do not rely on SANs or super-expensive multi-processor systems. Our web servers are usually either single or dual processor machines with 1-2GB of RAM and a small, inexpensive hard drive. Our database servers are single or dual processor machines with 4-8GB of RAM and 2-4 fast SCSI drives in a RAID array using a hardware RAID controller. Because there is redundancy built into the architecture that several of these machines can fail at once at the site is unaffected, the individual machines do not need to be extremely robust. Historically, CPU time has been the most precious resource on the web servers and disk I/O on the DBs.

Requirements

• Provide the hardware outlined above
• Support Debian AMD64 or similar
• Provide a Gigabit private backend network for inter-server communication
• Ability to deploy additional servers quickly and painlessly
• Sales and support team that is competent and easy to work with
• US Datacenter and not Dallas or San Diego

Initial Impressions

Dedicated server providers seem to fall into 2 classes:

1) No-frills provider that offers server between $79 – $149/month. No phone support or advanced services. This would be fine, but they usually cannot provide the DB-class machines we need and do not provide Gigabit backend networks.

2) Full-service provider that offers servers beginning at $250/month and up. These providers are usually not the best fit for us because they justify their higher prices by saying they have superior support. We don’t really need “superior” support, just someone that can take care of hardware issues when needed. Seems like a waste to pay for something you are never going to use and don’t really need. Also, in my experience, the more expensive the hosting company, the more painful the sales process is. Sometimes I feel like I am buying a car….

The Finalists

Over the past month, we have narrowed the field from about 10 different possible providers down to the following 3. Each of these fit somewhere between no-frills and full-service as mentioned above, but I definitely get the feeling that they lean one way or the other.

• ServePath
  Based in San Francisco, California (where Automattic is also based) they seem to lean towards the full-service side, offering 24×7 phone support, higher-end servers, load balancers, and firewalls. Their sales process has been pretty agonizing. It is now going on 45 days of back and forth, price changes, configuration changes, conference calls, and about 50 one-on-one calls with our sales guy. I had chance to visit their offices and tour their datacenter earlier this week so I got a feel of how things work there — it appears to be a well-run organization. One thing that struck me as a bit odd was that they do not deploy any rack-mounted servers in their datacenter. All of their servers, which are built in house, are in tower (white-box) chassis like you would see under an office desk. This is something I would expect in the lower-end market, but at $500+ per month and rather large RAID arrays (300GB SCSI x 6) I would expect to see more rack-mounted chassis to take advantage of the superior cooling.

• Server Beach
  Based in San Antonio, Texas and now owned by Peer1, Server Beach is definitely more on the no-frills side. When we were looking at new datacenters about 6 months ago, Server Beach could not provide what we wanted — they did not offer SCSI RAID or Gigabit privatenet, but said it was coming. Well, those things are now available. Kudos to Server Beach on a super-simple and painless sales process. Once I contacted them with the configuration we wanted, they scheduled a conference call to discuss the details. They had members of their support, operations, and sales teams on the line. About 30 minutes later we were all done and a day later they had sent over a proposal. It was right the first time – had everything we asked for and at a very fair price.

• Joyent/TextDrive
  Honestly, we probably will not go with TextDrive for this deployment, as it is a pretty radical departure from our current configuration, and they don’t technically meet the requirements laid out above. They are worth mentioning, however, because they are doing some pretty cool stuff with OpenSolaris, zones, and ZFS, and it sounds like it could be a good fit for our architecture model. Their storage is super-fast, the container model allows you to replicate existing containers with a single command — there is no need to provision and setup a new physical server. ZFS offers all sorts of cool stuff like snapshots, compression, and built-in data consistency checks. Utilizing this architecture, would require that we maintain 2 completely separate environments — Linux and Solaris — and there is a definite time investment in doing so. I think that we will probably look at moving some of our services to Solaris containers in the near future, but I am not sure it will be WordPress.com.

The Verdict

First, let me say that we still have yet to make the final decision, but hope to do so by the end of the week. Both ServePath and Server Beach seem like they will be great companies to work with. ServePath is local, so we can walk over to their offices and meet with them if needed – there is something to be said for working with local vendors. Server Beach has been a pleasure to work with thus far and I have some experience working with them in the past. Peer1’s VP of Marketing also blogs on WordPress.com. Pricing and server configurations are almost identical, so that really isn’t as much of a factor as one would think.

Anyone have experience working with either of these companies? Suggestions? Feedback?