I have a few unadvertised services exposed to the internet on my personal server, so I was surprised when I learned that I get several hundred invalid access attempts every day. Here are a few examples from two log files:
/var/log/auth.log - someone tried to login as every user from 'a' to 'z' and more.
Jul 12 11:32:28 frankie sshd: Failed password for invalid user a from 188.8.131.52 port 50291 ssh2 Jul 12 11:32:37 frankie sshd: Failed password for invalid user b from 184.108.40.206 port 50735 ssh2 Jul 12 11:32:47 frankie sshd: Failed password for invalid user c from 220.127.116.11 port 51167 ssh2 ...
/var/log/apache2/error.log - someone was scanning my machine for commonly-exploited software
[Mon Jun 18 20:39:09 2007] [error] [client 18.104.22.168] File does not exist: /var/www/xmlrpc [Mon Jun 18 20:39:09 2007] [error] [client 22.214.171.124] File does not exist: /var/www/xmlsrv [Mon Jun 18 20:39:09 2007] [error] [client 126.96.36.199] File does not exist: /var/www/blog [Mon Jun 18 20:39:09 2007] [error] [client 188.8.131.52] File does not exist: /var/www/drupal ...
So what did I do about it? I found Fail2Ban. Its a python script that watches your logs. It looks for patterns and takes action after a threshold is met. Everything is user-configurable in the /etc/fail2ban directory, so its easy to setup. It works [by default] by dropping the offender's IP address from your routing tables for several minutes, which makes you seem invisible. They won't get any response from your machine (not even from a ping). Best of all, now I see things like this in my fail2ban.log file:
2007-07-15 13:10:58,773 fail2ban.actions: WARNING [ssh] Ban 184.108.40.206 2007-07-15 13:20:58,908 fail2ban.actions: WARNING [ssh] Unban 220.127.116.11 2007-07-15 15:09:53,393 fail2ban.actions: WARNING [ssh] Ban 18.104.22.168 2007-07-15 15:19:53,822 fail2ban.actions: WARNING [ssh] Unban 22.214.171.124
and my log files are much shorter.
Unfortunately, the text in my auth.log file was different than what fail2ban was looking for, so I had to change it. I modified the /etc/fail2ban/filter.d/sshd.conf file as follows:
[Definition] failregex = (?:Authentication failure|Failed [-/\w+]+) for(?: [iI](?:llegal|nvalid))? user .*(?: from|FROM) <HOST> ROOT LOGIN REFUSED .* FROM <HOST> [iI](?:llegal|nvalid) user .* from <HOST> Failed password for .* from <HOST> #2007-05-02 CF above line added ("Failed password") ignoreregex =
Sometime after upgrading to Ubuntu 9.04, I browsed thru my logs and found this set of statements in the
2009-05-11 21:08:42,110 fail2ban.server : ERROR Unexpected communication error 2009-05-11 21:08:42,124 fail2ban.server : ERROR Unexpected communication error 2009-05-11 21:08:42,125 fail2ban.server : ERROR Unexpected communication error 2009-05-11 21:08:42,127 fail2ban.server : ERROR Unexpected communication error
A quick search on Ubuntu's forums led me to this defect on launchpad, where I discovered that a new release of Python might be the culprit. So I changed the first line of
/usr/bin/fail2ban-server from this:
and restarted the service. Within seconds, it was working again.
It concerns me that Ubuntu didn't let me know that fail2ban was messing up. Ubuntu 9.04 was released on April 23, 2009, and I upgraded a week or two later. I didn't discover the malfunction until May 11th, so I was “exposed” for a week or so. Fortunately the basic banning feature was still working. I saw ban/unban messages in my logs even while the errors were happening, and I haven't seen any evidence of unauthorized activity on my server. So I guess I can't really complain.
By now Fail2Ban has been updated and its package dependency has been changed in the Ubuntu repositories. I guess thats the price I pay for being on the leading edge.