A while back, I posted about some testing we were doing of various software load balancers for WordPress.com. We chose to use Pound and have been using it past 2-ish years. We started to run into some issues, however, so we starting looking elsewhere. Some of these problems were:
- Lack of true configuration reload support made managing our 20+ load balancers cumbersome. We had a solution (hack) in place, but it was getting to be a pain.
- When something would break on the backend and cause 20-50k connections to pile up, the thread creation would cause huge load spikes and sometimes render the servers useless.
- As we started to push 700-1000 requests per second per load balancer, it seemed things started to slow down. Hard to get quantitative data on this because page load times are dependent on so many things.
So… A couple weeks ago we finished converting all our load balancers to Nginx. We have been using Nginx for Gravatar for a few months and have been impressed by its performance, so moving WordPress.com over was the obvious next step. Here is a graph that shows CPU usage before and after the switch. Pretty impressive!
Before choosing nginx, we looked at HAProxy, Perlbal, and LVS. Here are some of the reasons we chose Nginx:
- Easy and flexible configuration (true config “reload” support has made my life easier)
- Can also be used as a web server, which allows us to simplify our software stack (we are not using nginx as a web server currently, but may switch at some point).
- Only software we tested which could handle 8000 (live traffic, not benchmark) requests/second on a single server
We are currently using Nginx 0.6.29 with the upstream hash module which gives us the static hashing we need to proxy to varnish. We are regularly serving about 8-9k requests/second and about 1.2Gbit/sec through a few Nginx instances and have plenty of room to grow!
Barry –
One question: How are you guys doing failover with the nginx box? Nginx has been on my list of things to look at, but so far no time in the R&D bank.
Mike
Mike,
We are using Wackamole and Spread which is the same way we did it when we were running Pound. Works great!
Thanks for the rundown Barry. We’re about to go live with nginx in a similar role, it’s a really nice piece of software. We’ve got it behind ipvs / keepalived to handle simple layer 4 load balancing and failover, the combination works well.
Have you seen any issues with ssl or ssl+gzip? This seems to be an area where 0.5 and 0.6 have both had a few bugs recently — and something that seems not too easy to exercise without real traffic. Thanks!
James,
Awesome. We haven’t seen any issues with ssl or ssl+gzip. Our SSL traffic volume is pretty low, however.
We’ve been using nginx for a while now (for about 2-3 years) and the performance is just impressive.
What was the +/- versus HAProxy? We haven’t used either but am very curious!
Thanks.
From everything I’m reading, there’s not many reasons *not* to switch to nginx. I’m building my network with it starting out, so I can use its various capabilities in the future. What kind of load balancing does it do? It has built in round-robin, with a weight measurement, right? It doesn’t have anything to check the upstream servers’ health as far as I know. I’m esp interested in the static gzip module and passing things off to Varnish – can you explain more how those tie together? I assume Varnish is upstream from the nginx load balancer?
Thanks
Nginx offers weighed round robin and IP-hash based sticky sessions by default. It also does health checks.
See: http://wiki.codemongers.com/NginxHttpUpstreamModule
We do not serve text via varnish, just images and video, so I don’t have any input as to how the gzip functionality would work in that case.
What’s the max *safe* amount of traffic a single Nginx instance is load balancing for you and how much memory and cpu are being utilized?
🙂
We have tested it up to about 10k req/sec. Memory footprint is minimal, and Nginx doesn’t use much CPU time. Where you end up with problems is in the TCP overhead and the time spent handling software interrupts. It gets much worse with iptables and connection tracking. Performance here is probably better on FreeBSD than Linux (we run Linux), but I haven’t tested it.
How did you handle the iptables and connection tracking limitations? Did you disable connection tracking?
Yes, no more connection tracking for us. We got around the limits but the CPU usage is too high.
[…] Vor kurzem hat auch WordPress mitgeteilt, nun benutzt WP Nginx in eigenen Proxi-Server. Er ist der Einzige, der fähig ist 8000 Anfragen in die Sekunde zu bearbeiten […]
Hi. Nice article – thanks. Not sure if I understand where varnish fits in. Does it work like this?
Internet Client –> Varnish(s) –> NGINX(s) –> Webservers
Our setup is:
Client –> Nginx –> (Varnish|Webserver) –> [Webserver]
Depending on the request type some requests are then passed to Varnish and others are sent directly to the web servers. We currently use Varnish only to serve on static images and video content (reverse caching proxy to Amazon’s S3).
Barry,
This is a follow up to Mike’s question on 4/28 about the failover configuration of nginx. We are specifically interested in understanding if and how nginx can be configured for a traditional active/active failover pair. We want to know if nginx supports state sharing between the failover pair so as to maintain continuation of service for such features as server affinity.
Any light and/or guidance you can share is greatly appreciated.
Matthew
Hi, you said ..
“Only software we tested which could handle 8000… ”
so you mean LVS kind of kernel level load balancing is even slower than ngnix?
Memory footprint is minimal, and Nginx doesn’t use much CPU time in me too…
Wackamole would be a decent replacement for heartbeat for managing IPs it sounds like.
So in theory you could use Wackamole+nginx for Active/Passive(+more) nginx instances and Wackamole would handle all the IP switching and skip using LVS/ldirectord|keepalived/heartbeat, right?
Barry – I appreciate your answers, having real-world examples of nginx and varnish give us the answers we need – this is a great resource.
Ok, here’s a silly question. Are you using nginx on the backend web servers too?
hi,
we plan a website with around 10000-50000 online users.
we plan to use nginx as a loadbalancer and will have the webservers within an internal ip-network.
my question is: if the nginx LB has to route+NAT all the users to the internal webservers, how much load will that make on the nginx server? Is it possible?
Thank u very much for your help!
Hi,
Really interesting post. I also like to know like Matthew Porter +/- against HAProxy. HAProxy as i know also supports hot-reconfiguration and can take pretty heavy load.
And another thing thats really interesting is how many servers you need to server that kind of traffic. Especially how many you need as proxy servers?
Thanks for sharing!!
I just read the post again and found an answer to my own question…
You have 20+ loadbalancers…
That raises another question.. how to you spread the load over the load balancers 😉
/Mathias
@Mathias – reading this document, looks like nginx is only able to do simple round-robin.
.
Nothing fancy yet such as like intelligent request queueing with HAproxy.
.
Also I noticed that HAproxy can handle 34000+ connections per second, as shown in this page. This is well beyond WP.com’s 10000 conn/sec.
.
Could the lack of performance of HAproxy in this post be explained by the request queue bug? Which has since been fixed after the publication of this post. The test on the new version of HAproxy shows it beating nginx, cpu-load wise.
No bench on connection/sec though, so it may be completely irrelevant, but still, it might be of interest.
.
Hopefully we’ll be able to find out even more on these great pieces of software.
.
Thanks.
how to you spread the load over the load balancers
I’m curious about that, too… seems like the only real option at that layer is RR DNS?
I cant believe your comment that nginx was the only solution that could reach 8000 cons/sec. HaProxy (latest) I’ve had doing full cookie inserts at 27,000 cons/sec. A graph here compares connections/sec on the Kemp 1500 and Loadbalancer.org R16 which are both based on LVS here http://www.loadbalancer.org/whyr16.html (we also use Pound & Haproxy). Blatant commercial link but still relevant.
and what are you using to route and switch 1.2Gbit/sec?
i have same question with you. what are you using to route and switch 1.2Gbit/sec?
I’m just learning about load balancing WordPress. I was wondering how the load balancing deal with mysql and how data would replicate between the different servers.
The idea is to have 2 data centers, each data center would have 1 load balancers, 2 web servers and 1 mysql server.
Scott.
Very impressive, will nginx work with Joomla??
industryfinest nginx work with Joomla!
Barry,
Thanks, though I found this post a bit late, it saved my job. We have decided to port our latest word press news site to nginx. We are already getting 10K hits per day, and expect around 50K once new features and channels are added..
I’m definately going to download it now.. I’ve been looking for a small load-balancing solution myself for along time, I just with it was’t all russian documentation…
Also give crossroads a try (crossroads.e-tunity.com). It has a very small footprint but still a lot of powerful features like access control and dos prevention.
Красивый блог у тебя! Особенно падающий снег!
Тут уже столько напостили до меня. Остается только присоединиться.
What was the +/- versus HAProxy? We haven’t used either but am very curious!
nice thanx you
Script for measurte cpu usage write by hand or buy?
They are generated using Munin which is open source software.
The link to NginxHttpUpstreamModule from codemongers.com has turned into a 404. To help locate the proper documentation, here is the updated link:
http://wiki.nginx.org/NginxHttpUpstreamModule
Thanks, I updated the link in the post.
Good info! I’ve been reading more and more about nginx lately. I have to check it out.
Apache is best…
Great post!
You indicated that you might want to use nginx for content serving, too. What has come of it, and why/why not, please?
Оформление зачет 🙂 Сами рисовали или тему стандартную ставили?
I’m just learning about load balancing WordPress. I was wondering how the load balancing deal with mysql and how data would replicate between the different servers.
industryfinest nginx work with Joomla!
Gr, http://taart-bestellen.blogspot.com
I came to this post from nginx official website, your post is like a case study. You should make a small pdf and publish it as a white paper or case study. I’m sure many people would like to learn more. Mikrowelle Edelstahl
Nginx + WordPress is the fast and stable. I use it for my blog.
I had to comment because I just wrote an article http://realtechtalk.com/nginx_the_ultimate_load_balancer-951-articles about nginx. When I saw this BLOG post I was sure that nginx is the best load balancer and has significant advantages over others such as pound.
I think WordPress really proved it to the world that nothing beats nginx. I plan to use nginx myself for future projects.
Great post!
You indicated that you might want to use nginx for content serving, too. What has come of it, and why/why not, please?
slm
So how does nginx handling massive 1000PPS+ DDOS attacks? Especially the http ones. In that case you would put a filtering device before it which stops the “bad” packets but im curious by itself how does it deal with it.
Interesting write-up. I understand that you uses quite a few Nginx LBs, and even more backends. How do you spread the load across the Nginx instances?
DNS.
I know this item is very old, but I have a question.
I have 2 NginX load balancers which DNS spreads the load between them, but if one of load balancer servers stops, what will happen? half of users will get 404?
nope, half of users will get an error page from their browser telling that the connection is not possible, because your nginx does not answer anymore.
that’s the problem with DNS…
In this case you should set a very low TTL on your DNS records, in order to switch quickly if neededd 😉
Thank barry for sharing how you handle clients of wordpress.com. I am also an nginx user but I haven’t tried using varnish. I’ll try to use it in some of my web applications. More power!
Vor diesen Hintergründen ist es tatsächlich nachvollziehbar, warum Informationsmagazine oder Nachrichten absolut die professionellen Schädlingsbekämpfer favorisieren.
Thank you for your share, ı like this.
Jakuzi modelleri oldukça çeşitlilik göstermekle beraber jakuzilerin faydaları da saymakla bitmez. Sağlık üzerine etkileri ve sağladığı hem fiziksel hem de psikolojik rahatlama ile insan vücudu üzerinde etkisi oldukça fazladır.