Load Balancer Update

A while back, I posted about some testing we were doing of various software load balancers for WordPress.com.  We chose to use Pound and have been using it past 2-ish years.  We started to run into some issues, however, so we starting looking elsewhere.  Some of these problems were:

  • Lack of true configuration reload support made managing our 20+ load balancers cumbersome.  We had a solution (hack) in place, but it was getting to be a pain.
  • When something would break on the backend and cause 20-50k connections to pile up, the thread creation would cause huge load spikes and sometimes render the servers useless.
  • As we started to push 700-1000 requests per second per load balancer, it seemed things started to slow down.  Hard to get quantitative data on this because page load times are dependent on so many things.

So…  A couple weeks ago we finished converting all our load balancers to Nginx.  We have been using Nginx for Gravatar for a few months and have been impressed by its performance, so moving WordPress.com over was the obvious next step.  Here is a graph that shows CPU usage before and after the switch.  Pretty impressive!

Before choosing nginx, we looked at HAProxy, Perlbal, and LVS. Here are some of the reasons we chose Nginx:

  • Easy and flexible configuration (true config “reload” support has made my life easier)
  • Can also be used as a web server, which allows us to simplify our software stack (we are not using nginx as a web server currently, but may switch at some point).
  • Only software we tested which could handle 8000 (live traffic, not benchmark) requests/second on a single server
We are currently using Nginx 0.6.29 with the upstream hash module which gives us the static hashing we need to proxy to varnish.  We are regularly serving about 8-9k requests/second  and about 1.2Gbit/sec through a few Nginx instances and have plenty of room to grow!

135 responses to “Load Balancer Update”

  1. Barry –

    One question: How are you guys doing failover with the nginx box? Nginx has been on my list of things to look at, but so far no time in the R&D bank.

    Mike

  2. Mike,

    We are using Wackamole and Spread which is the same way we did it when we were running Pound. Works great!

  3. Thanks for the rundown Barry. We’re about to go live with nginx in a similar role, it’s a really nice piece of software. We’ve got it behind ipvs / keepalived to handle simple layer 4 load balancing and failover, the combination works well.

    Have you seen any issues with ssl or ssl+gzip? This seems to be an area where 0.5 and 0.6 have both had a few bugs recently — and something that seems not too easy to exercise without real traffic. Thanks!

  4. James,

    Awesome. We haven’t seen any issues with ssl or ssl+gzip. Our SSL traffic volume is pretty low, however.

  5. […] глобально в Алексе, и 26-ой сайт в США, что больше LiveJournal) перевел все load-balancer’ы на nginx на прошлой неделе. « Microsoft передумала в плане […]

  6. We’ve been using nginx for a while now (for about 2-3 years) and the performance is just impressive.

  7. wordpress.com использует nginx…

    WordPress.com перешел на использование nginx на своих load-balancer-а…

  8. […] перевел свои балансировщики нагрузки на nginx – очень серьезный […]

  9. Matthew Porter Avatar
    Matthew Porter

    What was the +/- versus HAProxy? We haven’t used either but am very curious!

    Thanks.

  10. […] After two years of using Pound, WordPress decided to switch to Nginx as software load balancers for WordPress.com. Read the full story on Barry’s Blog. […]

  11. From everything I’m reading, there’s not many reasons *not* to switch to nginx. I’m building my network with it starting out, so I can use its various capabilities in the future. What kind of load balancing does it do? It has built in round-robin, with a weight measurement, right? It doesn’t have anything to check the upstream servers’ health as far as I know. I’m esp interested in the static gzip module and passing things off to Varnish – can you explain more how those tie together? I assume Varnish is upstream from the nginx load balancer?

    Thanks

  12. Nginx offers weighed round robin and IP-hash based sticky sessions by default. It also does health checks.
    See: http://wiki.codemongers.com/NginxHttpUpstreamModule

    We do not serve text via varnish, just images and video, so I don’t have any input as to how the gzip functionality would work in that case.

  13. What’s the max *safe* amount of traffic a single Nginx instance is load balancing for you and how much memory and cpu are being utilized?

    🙂

  14. We have tested it up to about 10k req/sec. Memory footprint is minimal, and Nginx doesn’t use much CPU time. Where you end up with problems is in the TCP overhead and the time spent handling software interrupts. It gets much worse with iptables and connection tracking. Performance here is probably better on FreeBSD than Linux (we run Linux), but I haven’t tested it.

    1. How did you handle the iptables and connection tracking limitations? Did you disable connection tracking?

      1. Yes, no more connection tracking for us. We got around the limits but the CPU usage is too high.

  15. […] Vor kurzem hat auch WordPress mitgeteilt, nun benutzt WP Nginx in eigenen Proxi-Server. Er ist der Einzige, der fähig ist 8000 Anfragen in die Sekunde zu bearbeiten […]

  16. […] from WrdPress is explaining how they setup nginx into their 300 servers cluster, for their 2 main sites (wordpress.com and […]

  17. Hi. Nice article – thanks. Not sure if I understand where varnish fits in. Does it work like this?

    Internet Client –> Varnish(s) –> NGINX(s) –> Webservers

  18. Our setup is:

    Client –> Nginx –> (Varnish|Webserver) –> [Webserver]

    Depending on the request type some requests are then passed to Varnish and others are sent directly to the web servers. We currently use Varnish only to serve on static images and video content (reverse caching proxy to Amazon’s S3).

  19. […] From 0:00 CET orion.janforman.com is running nginx russian webserver with apache backend 🙂 this result in much faster transfers and better memory usage. Some testimonials from wordpress […]

  20. Barry,

    This is a follow up to Mike’s question on 4/28 about the failover configuration of nginx. We are specifically interested in understanding if and how nginx can be configured for a traditional active/active failover pair. We want to know if nginx supports state sharing between the failover pair so as to maintain continuation of service for such features as server affinity.
    Any light and/or guidance you can share is greatly appreciated.

    Matthew

  21. Hi, you said ..

    “Only software we tested which could handle 8000… ”

    so you mean LVS kind of kernel level load balancing is even slower than ngnix?

  22. Memory footprint is minimal, and Nginx doesn’t use much CPU time in me too…

  23. Wackamole would be a decent replacement for heartbeat for managing IPs it sounds like.

    So in theory you could use Wackamole+nginx for Active/Passive(+more) nginx instances and Wackamole would handle all the IP switching and skip using LVS/ldirectord|keepalived/heartbeat, right?

  24. […] more than a million sites; more than doubling in numbers. The WordPress blogging system recently converted all of its load balancers to nginx, using the upstream hash module to serve 8-9 thousand requests […]

  25. Barry – I appreciate your answers, having real-world examples of nginx and varnish give us the answers we need – this is a great resource.

  26. […] from Jamie’s pointer, in doing the initial research, what got me excited was reading that WordPress.com had switched to nginx for their load balancing (and might eventually switch for their web serving as well), and that Fastmail is using nginx for […]

  27. […] more than a million sites; more than doubling in numbers. The WordPress blogging system recently converted all of its load balancers to nginx, using the upstream hash module to serve 8-9 thousand requests […]

  28. […] more than a million sites; more than doubling in numbers. The WordPress blogging system recently converted all of its load balancers to nginx, using the upstream hash module to serve 8-9 thousand requests […]

  29. Ok, here’s a silly question. Are you using nginx on the backend web servers too?

  30. hi,
    we plan a website with around 10000-50000 online users.
    we plan to use nginx as a loadbalancer and will have the webservers within an internal ip-network.
    my question is: if the nginx LB has to route+NAT all the users to the internal webservers, how much load will that make on the nginx server? Is it possible?
    Thank u very much for your help!

  31. […] baca postingan Barry tentang migrasi wordpress ke nginx ( jadi balancernya kl gak salah) saya jadi penasaran untuk coba mencicip engine x (nginx). […]

  32. […] nginx, a lightweight HTTP server. This is the softwareWordpress.com uses as a load balancer: We have been using Nginx for Gravatar for a few months and have been impressed by its […]

  33. Hi,
    Really interesting post. I also like to know like Matthew Porter +/- against HAProxy. HAProxy as i know also supports hot-reconfiguration and can take pretty heavy load.

    And another thing thats really interesting is how many servers you need to server that kind of traffic. Especially how many you need as proxy servers?

    Thanks for sharing!!

  34. I just read the post again and found an answer to my own question…

    You have 20+ loadbalancers…

    That raises another question.. how to you spread the load over the load balancers 😉

    /Mathias

  35. […] WordPress (eh già credevate che usassero Apache), YouPorn, FastMail.FM, Kongregate, Hulu, provate infatti a vedere le risposte http da questi siti. […]

  36. @Mathias – reading this document, looks like nginx is only able to do simple round-robin.
    .
    Nothing fancy yet such as like intelligent request queueing with HAproxy.
    .
    Also I noticed that HAproxy can handle 34000+ connections per second, as shown in this page. This is well beyond WP.com’s 10000 conn/sec.
    .
    Could the lack of performance of HAproxy in this post be explained by the request queue bug? Which has since been fixed after the publication of this post. The test on the new version of HAproxy shows it beating nginx, cpu-load wise.
    No bench on connection/sec though, so it may be completely irrelevant, but still, it might be of interest.
    .
    Hopefully we’ll be able to find out even more on these great pieces of software.
    .
    Thanks.

  37. how to you spread the load over the load balancers

    I’m curious about that, too… seems like the only real option at that layer is RR DNS?

  38. I cant believe your comment that nginx was the only solution that could reach 8000 cons/sec. HaProxy (latest) I’ve had doing full cookie inserts at 27,000 cons/sec. A graph here compares connections/sec on the Kemp 1500 and Loadbalancer.org R16 which are both based on LVS here http://www.loadbalancer.org/whyr16.html (we also use Pound & Haproxy). Blatant commercial link but still relevant.

  39. and what are you using to route and switch 1.2Gbit/sec?

    1. i have same question with you. what are you using to route and switch 1.2Gbit/sec?

  40. […] which were delivering static assets like CSS, JavaScript and (some) image files. Recently the WordPress.com load balancers were upgraded to nginx and since then nginx has been proving to be a very high performance piece of software, with some […]

  41. I’m just learning about load balancing WordPress. I was wondering how the load balancing deal with mysql and how data would replicate between the different servers.

    The idea is to have 2 data centers, each data center would have 1 load balancers, 2 web servers and 1 mysql server.

    Scott.

  42. Very impressive, will nginx work with Joomla??

  43. […] what I have heard, it’s capable to run quite high load on very low requirements. Regarding to this article, WordPress.com is using Nginx as load balancer, handling over 8000 requests / sec. […]

  44. industryfinest nginx work with Joomla!

  45. Barry,

    Thanks, though I found this post a bit late, it saved my job. We have decided to port our latest word press news site to nginx. We are already getting 10K hits per day, and expect around 50K once new features and channels are added..

  46. I’m definately going to download it now.. I’ve been looking for a small load-balancing solution myself for along time, I just with it was’t all russian documentation…

  47. Also give crossroads a try (crossroads.e-tunity.com). It has a very small footprint but still a lot of powerful features like access control and dos prevention.

  48. […] what you’re probably thinking of is WordPress.com recently switching to using Nginx as a frontend load-balancing HTTP proxy or to serve static images and files instead of Lighttpd. Those are both excellent use cases for […]

  49. […] oh yeah, forgot to mention wordpress uses it: barry.wordpress.com/2008/04/28/load-balancer-update/ […]

  50. Красивый блог у тебя! Особенно падающий снег!

  51. […] Load Balancer Update « Barry on WordPress (tags: architecture wordpress varnish nginx) This entry was written by bairos, posted on December 23, 2008 at 1:30 am, filed under delicious-daily. Bookmark the permalink. Follow any comments here with the RSS feed for this post. Post a comment or leave a trackback: Trackback URL. « links for 2008-12-21 […]

  52. Тут уже столько напостили до меня. Остается только присоединиться.

  53. What was the +/- versus HAProxy? We haven’t used either but am very curious!

    nice thanx you

  54. […] sobre o mesmo assunto, me lembro de ter lido no blog do Barry, engenheiro do wordpress.com, um artigo sobre a atualização do load-balancer que usam para distribuir a carga de acesso ao site entre os diversos […]

  55. […] de 2009 ganhou mais 1270000 sites, o nginx é usado por exemplo no excelente site de blogging, o wordpress, do qual faz parte este blog, bem como no FastMailFM, só em Dezembro de 2008 estava a ser usado em […]

  56. […] beban yang sesuai, kami ada juga consider tentang LVS, Pound dan seangkatan dengannya. Tetapi pos dari wordpress tentang nginx ini membuatkan kami ingin mencuba nginx. Dah cuba, kami mula menyukainya, hohoho […]

  57. […] nginx has been running for more than four years on many heavily loaded Russian sites including Rambler (RamblerMedia.com). In March 2007 about 20% of all Russian virtual hosts were served or proxied by nginx. According to Google Online Security Blog year ago nginx served or proxied about 4% of all Internet virtual hosts, although Netcraft showed much less percent. According to Netcraft in March 2008 nginx served or proxied 1 million virtual hosts. The growing in picture and colour! According to Netcraft in December 2008 nginx served or proxied 3.5 millions virtual hosts. And now it is on 3rd place and ahead of lighttpd. According to Netcraft in March 2009 nginx served or proxied 3.06% busiest sites. 2 of Alexa US Top100 sites use nginx. Here are some of success stories: FastMail.FM, Wordpress.com. […]

  58. Script for measurte cpu usage write by hand or buy?

    1. They are generated using Munin which is open source software.

  59. […] nginx, it’s got a fair few success stories to its name and powers some major sites including WordPress.com and […]

  60. […] – 8000 koneksi secara LIVE! / sec ( WordPress mengatakan ) – Kaskus menggunakan Nginx ( Dapat dilihat menggunakan Addon firefox , sebelumnya Kaskus […]

  61. […] of *.wordpress.com is behind Nginx/Apache/Varnish and that’s pretty cool; Looks like they are saving a bunch of money by switching to Gei^H^H^H, […]

  62. The link to NginxHttpUpstreamModule from codemongers.com has turned into a 404. To help locate the proper documentation, here is the updated link:

    http://wiki.nginx.org/NginxHttpUpstreamModule

    1. Thanks, I updated the link in the post.

  63. Good info! I’ve been reading more and more about nginx lately. I have to check it out.

  64. […] having to go through that painful process again. So this time I searched the interweb and turned up this blog post which sparked the idea of using Nginx to solve this […]

  65. […] of *.wordpress.com is behind Nginx/Apache/Varnish and that’s pretty cool; Looks like they are saving a bunch of money by switching to Gei^H^H^H, […]

  66. Apache is best…

  67. Great post!
    You indicated that you might want to use nginx for content serving, too. What has come of it, and why/why not, please?

  68. Оформление зачет 🙂 Сами рисовали или тему стандартную ставили?

  69. […] Load Balancer Update « Barry on WordPress – January 20th ( tags: nginx loadbalancing performance wordpress scalability scaling web pound ) […]

  70. […] balancer, but still apache / mod_php on the webservers. The folks at WordPress.com also use it: https://barry.wordpress.com/2008/04/2…lancer-update/ Joe Taiabjee work: b5media.com – blog: joetek.ca – twitter: […]

  71. I’m just learning about load balancing WordPress. I was wondering how the load balancing deal with mysql and how data would replicate between the different servers.

  72. industryfinest nginx work with Joomla!

    Gr, http://taart-bestellen.blogspot.com

  73. I came to this post from nginx official website, your post is like a case study. You should make a small pdf and publish it as a white paper or case study. I’m sure many people would like to learn more. Mikrowelle Edelstahl

  74. […] more than a million sites; more than doubling in numbers. The WordPress blogging system recently converted all of its load balancers to nginx, using the upstream hash module to serve 8-9 thousand requests […]

  75. Nginx + WordPress is the fast and stable. I use it for my blog.

  76. […] nginx [engine x] is a HTTP and reverse proxy server, as well as a mail proxy server written by Igor Sysoev. It has been running for more than five years on many heavily loaded Russian sites including Rambler (RamblerMedia.com). According to Netcraft nginx erved or proxied 4.24% busiest sites in January 2010. Here are some of success stories: FastMail.FM, WordPress.com. […]

  77. I had to comment because I just wrote an article http://realtechtalk.com/nginx_the_ultimate_load_balancer-951-articles about nginx. When I saw this BLOG post I was sure that nginx is the best load balancer and has significant advantages over others such as pound.

    I think WordPress really proved it to the world that nothing beats nginx. I plan to use nginx myself for future projects.

  78. […] nginx [engine x], Igor Sysoev tarafından yazılan bir HTTP, reverse proxy ve mail proxy sunucusudur. 5 yıldır, özellikle Rus sitelerinde yoğun bir şekilde kullanılmaktadır. Örneğin; Rambler (RamblerMedia.com). Netcraft’a göre, nginx, Nisan 2010 itibari ile %4.70 oranında kullanılmaktadır. Bazı başarı hikayeleri (İngilizce): FastMail.FM, WordPress.com.   […]

  79. Great post!
    You indicated that you might want to use nginx for content serving, too. What has come of it, and why/why not, please?

    1. So how does nginx handling massive 1000PPS+ DDOS attacks? Especially the http ones. In that case you would put a filtering device before it which stops the “bad” packets but im curious by itself how does it deal with it.

  80. […] nginx [engine x] is a HTTP and reverse proxy server, as well as a mail proxy server written by Igor Sysoev. It has been running for more than five years on many heavily loaded Russian sites includingRambler (RamblerMedia.com). According to Netcraft nginx served or proxied 4.70% busiest sites in April 2010. Here are some of success stories: FastMail.FM, WordPress.com. […]

  81. Interesting write-up. I understand that you uses quite a few Nginx LBs, and even more backends. How do you spread the load across the Nginx instances?

      1. I know this item is very old, but I have a question.
        I have 2 NginX load balancers which DNS spreads the load between them, but if one of load balancer servers stops, what will happen? half of users will get 404?

        1. nope, half of users will get an error page from their browser telling that the connection is not possible, because your nginx does not answer anymore.
          that’s the problem with DNS…
          In this case you should set a very low TTL on your DNS records, in order to switch quickly if neededd 😉

  82. […] tend to modify the LAMP stack, for instance replacing Apache with the faster and newer nginx (eg wordpress.com) which is becoming increasingly popular and can substantially exceed Apache performance with more […]

  83. […] WordPress.com Publicidade ComentáriosMageia Linux – A primeira avaliação detalhada do Appunix no […]

  84. […] or proxied 4.70% of the busiest sites in April 2010. Here are some of success stories: FastMail.FM, WordPress.com.The sources are licensed under 2-clause BSD-like license. The source can be downloaded at […]

  85. […] are plenty of examples of HTTP reverse-proxying in big websites, so I’d say it has no problems in terms of performance, scaling or ease of […]

  86. […] nginx [engine x] is an HTTP and reverse proxy server, as well as a mail proxy server, written by Igor Sysoev. For a long time, it has been running on many heavily loaded Russian sites including Yandex, Mail.Ru, VKontakte, and Rambler. According to Netcraft nginx served or proxied 7.84% busiest sites in October 2011. Here are some of the success stories: FastMail.FM, WordPress.com. […]

  87. […] all, WordPress.com is using nginx! Load Balancer UpdateA while back, I posted about some testing we were doing of various software load balancers for […]

  88. […] Nginx est utilisé par des sites très connus et à fort trafic comme WordPress ( pour la repartition de charge ) , Github et SourceForge. Cette entrée a été publiée dans appsthru, blog, feedtweet, post. […]

  89. […] nginx [engine x], Igor Sysoev tarafından yazılan bir HTTP, reverse proxy ve mail proxy sunucusudur. 5 yıldır, özellikle Rus sitelerinde yoğun bir şekilde kullanılmaktadır. Örneğin; Rambler (RamblerMedia.com). Netcraft’a göre, nginx, Nisan 2010 itibari ile %4.70 oranında kullanılmaktadır. Bazı başarı hikayeleri (İngilizce): FastMail.FM, WordPress.com. […]

  90. Thank barry for sharing how you handle clients of wordpress.com. I am also an nginx user but I haven’t tried using varnish. I’ll try to use it in some of my web applications. More power!

  91. […] nginx [えんじんえっくす] は Igor Sysoev によって作られた HTTP とリバースプロキシのサーバで、メールプロキシサーバでもあります。Rambler (RamblerMedia.com) を含むロシアの多くの高負荷サイトで5年以上も動いています。Netcraft によると、nginx は 2010 年 4 月時点で 4.70% の人気サイトでサーバーとして、もしくはプロキシとして利用されています。成功例としては FastMail.FM やWordPress.com があります。 […]

  92. […] nginx [えんじんえっくす] は Igor Sysoev によって作られた HTTP とリバースプロキシのサーバで、メールプロキシサーバでもあります。Rambler (RamblerMedia.com) を含むロシアの多くの高負荷サイトで5年以上も動いています。Netcraft によると、nginx は 2010 年 4 月時点で 4.70% の人気サイトでサーバーとして、もしくはプロキシとして利用されています。成功例としては FastMail.FM やWordPress.com があります。 […]

  93. […] busiest sites in April 2012. Here are some of the success stories: FastMail.FM, WordPress.com. 读法,engine x […]

  94. […] WordPress.com has found nginx to be the only load balancer able to handle 8000 live traffic requests per second. […]

  95. […] Here are some of the success stories: FastMail.FM, WordPress.com. […]

  96. […] are plenty of examples of HTTP reverse-proxying in big websites, so I’d say it has no problems in terms of performance, scaling or ease of […]

  97. […] event driven. Nginx is faster at serving static files and consume less memory than Apache. As per wordpress.com, nginx is able to do load balancing for 8000 live traffic requests per […]

  98. […] Nginx ist viel performanter als der gute alte Apache Webserver. Auch Golem.de, ComputerBase oder WordPress.com setzen auf Nginx.Durch das neue Hosting hat Bitpage jetzt auch eine durchschnittliche Ladezeit von […]

  99. […] Nginx ist viel performanter als der gute alte Apache Webserver. Auch Golem.de, ComputerBase oder WordPress.com setzen auf Nginx.Durch das neue Hosting hat Bitpage jetzt auch eine durchschnittliche Ladezeit von […]

  100. […] để đánh giá việc này, NghinX hoạt động tuyệt vời với 8000 request / giây ( https://barry.wordpress.com/2008/04/28/load-balancer-update/ […]

  101. […] to your question, but nginx and pound both have good reputations as load-balancers. WordPress just switched to nginx with good […]

  102. […] The Only software That they tested, which could handle 8000 (live traffic, not benchmark) requests/second on a single server […]

  103. Vor diesen Hintergründen ist es tatsächlich nachvollziehbar, warum Informationsmagazine oder Nachrichten absolut die professionellen Schädlingsbekämpfer favorisieren.

  104. […] Barry on WordPress. Load Balancer Update […]

  105. […] Barry on WordPress. Load Balancer Update […]

  106. […] converted all their load-balancers to Nginx for WordPress.com in 2008 (you can read about it here) and migrated their server stack completely to […]

  107. […] converted all their load-balancers to Nginx for WordPress.com in 2008 (you can read about it here) and migrated their server stack completely to […]

  108. […] converted all their load-balancers to Nginx for WordPress.com in 2008 (you can read about it here) and migrated their server stack completely to […]

  109. […] convirtió todos sus balanceadores de carga a Nginx para WordPress.com en el 2008 (puede leer sobre esto aquí) y migraron el stack de su servidor completamente a […]

  110. […] 即时WordPress.com旗下的Automattic在2008年将所有负载均衡器都转换为Nginx(相关阅读),并将其服务器堆栈完全迁移到Nginx。 […]

  111. […] converted all their load-balancers to Nginx for WordPress.com in 2008 (you can read about it here) and migrated their server stack completely to […]

  112. […] converted all their load-balancers to Nginx for WordPress.com in 2008 (you can read about it here) and migrated their server stack completely to […]

  113. Thank you for your share, ı like this.

    Jakuzi modelleri oldukça çeşitlilik göstermekle beraber jakuzilerin faydaları da saymakla bitmez. Sağlık üzerine etkileri ve sağladığı hem fiziksel hem de psikolojik rahatlama ile insan vücudu üzerinde etkisi oldukça fazladır.

Leave a Reply to james holden | Locations and conditional rewrites in Nginx Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: