Nginx, SPDY, and Automattic

Yesterday, Valentin Bartenev, a developer at Nginx, Inc., announced SPDY support for the Nginx web server. SPDY is a next-generation networking protocol developed by Google and focused on making the web faster. More information on SPDY can be found on Wikipedia.

At Automattic, we have used Nginx since 2008. Since then, it has made its way into almost every piece of our web infrastructure. We use it for load balancing, image serving (via MogileFS), serving static and dynamic web content, and caching. In fact, we have almost 1000 servers running Nginx today, serving over 100,000 requests per second.

I met Andrew and Igor at WordCamp San Fransicso in 2011.  For the next six months, we discussed the best way for Automattic and Nginx, Inc. to work together. In December 2011, we agreed that Automattic would sponsor the development and integration of SPDY into Nginx. The only real requirement from our end was that the resulting code be released under an open source license so that others could benefit from all the hard work.

For the past 6 months, Valentin and others have been implementing SPDY support in Nginx, and for the past month or so, we have been continually testing SPDY, fixing bugs, and improving stability. Things are almost ready for production and we hope to enable SPDY for all of in the next few weeks. Today, this site is SPDY-enabled if you are using a recent version of Chrome or Firefox and accessing this site over SSL. You can download the Chrome extension here and the one for FireFox here.

Thanks to the Nginx team for all their hard work implementing SPDY, and thanks to all of my Automattic co-workers who helped us test SPDY.  I hope to post some real-world performance numbers in the next few weeks as we complete our SPDY deployment and gather more data. We are also looking forward to SPDY support being part of the official Nginx source in the near future.

“We’d like to say big thanks to the team at Automattic and especially to Pyry Hakulinen who has been great in helping us test and debug this first public version of SPDY module for nginx. Automattic is a great partner, and we will continue to work with Barry and his team on improvements to nginx and to nginx/SPDY in particular.”

Andrew Alexeev – Nginx, Inc.

Predicting server hardware failure with mcelog

Have you ever wanted to predict that a piece of hardware in your server was failing before it actually caused the server to crash?

Sure! We all do.

Over the past few months, I have been tracking the correlation between errors logged to the Machine Check Event Log (MCElog) and the hard crash of a server or application running on that server (mostly MySQL). So far, the correlation is about 90%. That is to say, about 9 times out of 10, there will be an error logged to the MCElog before the server actually crashes. It may take days or even weeks between the time of the logged error and the crash, but it will happen. We are now actively monitoring this log and replacing hardware (RAM and CPUs) which show errors before they actually fail which I thought was pretty cool, so I thought I would share how we are doing it.

On Debian, there is a package for the mcelog utility which will allow you to decode and display the kernel messages logged to /dev/mcelog Part of this package is a cron job which outputs the decoded contents of /dev/mcelog to /var/log/mcelog every 5 minutes:

*/5 * * * * root test -x /usr/sbin/mcelog -a ! -e /etc/mcelog-disabled && /usr/sbin/mcelog --ignorenodev --filter >> /var/log/mcelog

We modify this a little bit and add another cron job which rotates this log file on reboot:

@reboot root test -f /var/log/mcelog && mv /var/log/mcelog /var/log/mcelog.0

The reason we do this is because after a reboot, which is most likely a result of the hardware repair, we want to clear the active logfile (monitored by the nagios plugin below), so the alert will clear.  In case, however, the reboot was not part of the hardware maintenance, we still want to have a record of the hardware errors so we move the log file to mcelog.0.

We then have a simple nagios plugin which monitors /var/log/mcelog for errors:



if [ ! -f "$LOGFILE" ]
	echo "No logfile exists"
	exit 3
	ERRORS=$( grep -c "HARDWARE ERROR" /var/log/mcelog )
	if [ $ERRORS -eq 0 ]
		echo "OK: $ERRORS hardware errors found"
		exit 0
	elif [ $ERRORS -gt 0 ]
		echo "WARNING: $ERRORS hardware errors found"
		exit 1

And thats pretty much it.  In just a few weeks we have caught about a dozen hardware faults before they led to server crashes.

Disclaimer: This only works when running a X86_64 kernel and YMMV.

WordPress Code Repository

WordPress Code

We have decided to consolidate all of the small projects we have released into a single subversion repository. Previously these were spread across multiple domains and not very well publicized. We have setup a Trac instance as well to facilitate bug reports. There are 5 projects currently in the repository all of which we have used or are currently using at Automattic. Some of the projects, like Servermattic, are also being used elsewhere. All of these projects are obviously open source and are released under the GPL. Patches and feedback are welcome! We hope to release more of these soon. Thanks to Nikolay and Demitrious who have both contributed to the projects in the repository.

mod_auth_mysql and phpass

With the release of WordPress 2.5, there were some significant changes to the way passwords were stored in the database.  Prior to 2.5, passwords were stored as MD5 hashes.  While simple and easy, there were some security implications, so since 2.5, passwords are now salted and hashed using the phpass encryption library.  At Automattic we like to keep things simple, so we use the WordPress and bbPress user system for external authentication for things such as Trac and Subversion.  This allows us an effective and simple single sign on (SSO) solution for almost everything we do.  Unfortunately, the existing mod_auth_mysql apache module did not have support for the new password format.

Thanks to Nikolay, we now have the best of both worlds.  He has patched mod_auth_mysql to support phpass.  This means you can now have plug and play authentication against your WordPress blog or bbPress forum almost anywhere you can think of.  The patch allows automatic fallback to MD5 in case the user has not yet logged into WordPress and their password is still stored in the old format.  

Once the new module is loaded, you will just need to replace the following line in your apache configuration file.

OLD:AuthMySQLPwEncryption md5
NEW:AuthMySQLPwEncryption phpass

You can download the patched version here. It has been tested with Apache 2.2.3 and MySQL 4.1/5.0

%d bloggers like this: