I operate a number of websites, some quite large as far as total “number of pages.” Over the years, I’ve noticed an increasing number of poorly behaved web crawler applications – some completely ignore the robots.txt file, and more and more submit dozens of requests per minute. (eek!) For a site with thousands and thousands of pages, this can negatively impact server performance. Severely.
While I do have a few methods for blocking these guys in place, just this morning I noticed that one particular crawler, identifying itself as “INA dlweb; http://ina.fr/” and coming from 195.221.138.170, was really killing my machine. The only solution was to reconfigure my firewall to drop all packets coming from this IP address.
If you’ve got a server with the very popular LAMP setup (Linux/Apache/Mysql/Php), you can likely do the same.
Here’s how: As root, enter:
iptables -A INPUT -s a.b.c.d -j DROP
, where a.b.c.d is the IP address of the offending web bot. In my case, I entered “iptables -A INPUT -s 195.221.138.170 -j DROP”.
Worked like a charm – my server load dropped drastically:
Before the Change:
10:37:01 up 241 days, 18:20, 3 users, load average: 1.26, 0.70, 0.47
10:38:01 up 241 days, 18:21, 3 users, load average: 1.13, 0.73, 0.49
After the Change:
11:21:01 up 241 days, 19:04, 3 users, load average: 0.40, 0.55, 0.64
11:22:01 up 241 days, 19:05, 3 users, load average: 0.31, 0.52, 0.62
11:23:01 up 241 days, 19:06, 3 users, load average: 0.11, 0.42, 0.58
This method should work on most flavors of Linux, including Ubuntu, CentOS, Red Hat, Debian, etc.