[Surviving Attack] What’s doesn’t kill you makes you stronger

[Surviving Attack] What’s doesn’t kill you makes you stronger

Recently my website was down due to attacks to my Amazon server. Now that my website is back online, I want to share my experience so hopefully people can get something useful from my mistakes.

2016-10-02-21-54-17

My site’s traffic. It increases A LOT starting from 9/27…

The Timeline:

  • 9/27: Staying in the dorm when typhoon Megi attacked. Found my website became so slow that I couldn’t write new posts. Thought it was a bad connection due to the typhoon.
  • 9/28: Started feeling something was wrong. Checked Amazon instance traffic monitoring and saw a huge spike starting from 7/29. Website started to fail with message “Error establishing a database connection.” Reboot and website continued to fail.
  • 9/28: Started the debug process. Look at apache log /var/log/apache2/access.log and found a lot of entries (over 1GB). Realized I was attacked.
  • Isolated my server by Amazon security group. Limit access to only my ip.
  • Looked at MySQL error log /var/log/mysql/error.log and found the innodb/memory full error. Googled and realized it was due to xmlrpc.php attack. Confirmed it in apache access.log
  • 9/29: Added codes denying accessing xmlrpc.php in apache site config file. Found many entries with CONNECT http tag. Realized it was due to an insecure setting that turned my server in a forward proxy. Fixed that in apache config file.
  • Did some standard password changes preventive steps.
  • 9/29 11pm: Got a letter from Amazon Abuse team. Have to explain to them detailed steps I took to mitigate the intrusion.
  • 9/30: Started reopen port 80 to the world and still saw wordpress being crashed. Tried adding swapping but still no luck. Decided I need something more sophisticated and found an All-in-one WP Security & Firewall. Isolated my site again.
  • 10/1: Setting up the plugin, reopen port 80 and monitor the website. Started blocking ips.
  • 10/2: website seems to stabilized. Did some tweaking on apache performance.

Lesson Learned:

Never assume. Always check:

Every web server user should have the habit of monitoring web traffic metrics and logs.

Be REALLY careful about proxy settings:

Inappropriate proxy setting enables hackers to hide their identities while they attack other websites through your server. You don’t want to take the blame of something you didn’t do.

Honor Murphy’s Law

Most web tutorials include none or very little security, and many people like me would think that a simple wordpress blog site won’t interest hackers to attack.

This is not true with modern auto-attack tools. These auto-attack tools restlessly scanning the Internet and auto-attack any vulnerable site.

Thus, once your site is live you should really consider adding security tools. The question is not whether your site will be attacked, but WHEN.