Anatomy of a Denial of Service Attack

Running one of the largest websites on the internet with about 5 million unique sites hosted exposes you to all sorts of issues.  There are constant events to deal with, some internal, some external.  This morning, one of the more common external events, a Distributed Denial of Service Attack occurred.  We experience these types of attacks rather frequently, but most are easily mitigated and have no user impact.  One this morning, however, was rather large and thus impacted some users.

Here is a timeline and description of this morning’s events:

9:40 AM EST — Our internal monitoring systems alerted us to unusual activity in one of the four geographically diverse datacenters which serve WordPress.com traffic.  Here is what that anomaly looks like in graphical terms:

10:00 AM EST — The target of the attack was identified and removed from our network.  The attack, however continued.  This is because the attacker had hijacked tens of thousands of computers (probably by installing a virus which was spread via email) and these computers had no idea the site was no longer there.  A small log sample shows over 8 million requests for this one site from over 10,000 unique IP addresses.

10:20 AM EST — Since we have servers in multiple data centers throughout the United States which serve traffic for WordPress.com all the time, we were able to route all legitimate traffic out of the affected data center, and let the single affected data center deal with the attack.   

11:30 AM EST — The IPs targeted in the attack were null routed at this point which allowed us to bring all datacenters back online to serve normal traffic.

We keep hourly traffic metrics and based on those numbers, it looks like during the attack there was about a 5% decrease in overall pageviews during the 40 minutes before traffic was re-routed.  All things considered, not a bad outcome for an attack this size.  Looking at bandwidth graphs, this attack was in the 500Mbit – 750Mbit/sec range.  

27 responses to “Anatomy of a Denial of Service Attack”

  1. I hope it wasn’t just me suddenly becoming popular. Sorry, guys!

  2. Nice job, what were they attacking?

  3. Nice – I’ve never seen it graphically like that before but looks like it was handled well.

  4. What software do you use to monitor the traffic and get the graphs, etc?

  5. Well handled Barry 🙂 Crap I’m always humbled by the amount of traffic that goes through wordpress.com!

  6. Barry is my hero. The 5% decrease is pretty amazing (that it was that little). Dang.

  7. […] network. The only obvious communication that I could find about this matter was a tweet linking to a blog post about DOS attacks in general, and the actual one in particular, also written by a WordPress.com team representative. […]

  8. […] Non ne è stata data nessuna comunicazione ufficiale e quello che si sa, come spesso accade, arriva per vie traverse. I fatti si riferiscono ad appena un paio di giorni fa e l’attacco ha interessato tutti i blog ospitati sulla piattaforma di WordPress.com, inclusi alcuni ospiti vip come il network GigaOM. Questa volta però non si è trattato del solito attacco sporadico, debole e senza particolare impatto sulla piattaforma, ma di un attacco DDoS davvero massiccio e ben architettato, portato avanti con una potenza di fuoco tra i 500 ed i 750 Mbit/s con più di 10000 macchine coinvolte. Gli amanti dei dettagli tecnici e dei grafici di carico del server e traffico, possono trovare maggiori informazioni sul blog di Barry. […]

  9. […] no official word on any of the WordPress/Automattic blogs, only a tweet from @wordpressdotcom and a post entitled “Anatomy of a Denial of Service […]

  10. Excellent work Barry! That is a pretty sizable attack for sure! It always impresses me at how fast traffic can be rerouted with a well planned out setup like you guys have in place!

  11. […] Blog Herald picks up the story, pointing to Barry’s post with the stats. It looks like some hackers had a beef with one of the blogs they were hosting and directed .5/Gig […]

  12. I hate the internet.

  13. […] comunicación disponible acerca de este asunto es un tweet enlazando al post de un blog acerca de ataques DOS, pero en general, sin precisar, también comentado por un representante del equipo de […]

  14. Wow! Thanks for sharing that 🙂 It’s good to be able to see just how ably WordPress keeps itself protected, and how easily you can manouver around DoS attacks! Nothing like this to instill ever more faith in the great service you guys offer! Thanks 🙂

  15. […] is the case with any year, there were a couple rough days in 2008, but we survived a DOS attack or two with very minimal downtime, and learned a lot in the process that will have us better prepared in […]

  16. Very interesting. Thanks. It’s interesting to see a more detailed ‘timeline’ of events and to see the impact on graphs.

  17. […] is the case with any year, there were a couple rough days in 2008, but we survived a DOS attack or two with very minimal downtime, and learned a lot in the process that will have us better prepared in […]

  18. Omg…..
    for example my site Блесенка.ру have 3 mb traff per day +)

  19. Thanks for sharing this info! Although smaller in scale obviously, we’re running into similar problems too.

    Could you share in a new post what kind of tools you use to prevent ddos?

    Cheers,

  20. […] is the case with any year, there were a couple rough days in 2008, but we survived a DOS attack or two with very minimal downtime, and learned a lot in the process that will have us better prepared in […]

  21. […] looks like across about 700 CPU cores.  As you can see there is plenty of idle CPU for those big spikes or in case one of the other 2 data centers fail and we have to route more traffic to this […]

  22. прикольно, нигде не видал

  23. […] looks like across about 700 CPU cores.  As you can see there is plenty of idle CPU for those big spikes or in case one of the other 2 data centers fail and we have to route more traffic to this […]

  24. […] has said it receives DDOS attacks frequently, but is usually able to contain them from affecting users. (The […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: