High server load

During the last few days, I experienced a high load on my server (sometimes up to 15). Each time it happens, I observed that apache was unable to serve pages. Restarting it regularly seemed to fix the problem.

Yesterday, I started to investigate the problem. Actually it was "referer spam". The stats of my blog are generated with webdruid and are available on http://blog.aurel32.net/stats/ . Some spammers tried to increase their website's page rank by submitting spoofed referers. It seems that they use zombie hosts, as the requests come from many IPs. The bad thing is that the hosts don't close the TCP connections, causing a lot of apache processes to be unable to serve pages. It's like a DoS, though this was not the aim.

A search on Google gave me a way to stop that. I added the following lines to /etc/wordpress/htaccess:

RewriteEngine On
RewriteCond %{HTTP_REFERER} ^http://www.spammersite1.com [OR]
RewriteCond %{HTTP_REFERER} ^http://www.spammersite2.com
RewriteRule .* - [F,L]

The load started to go down. Good! I also added a robots.txt, so that the stats pages are not indexed anymore by the search engines (note to the wordpress maintainer: it would be nice to have a /etc/wordpress/robots.txt).

After a day, I grepped the apache logs to find all the zombies IPs, and I blacklisted all of them on my firewall with iptables, ie. 217 IPs!

This event reminds me that my server doesn't have enough RAM and that I should add some more.