Tuesday, August 5, 2014

UFW with Fail2ban – Quick Secure Setup

Welcome to the first part of my Quick Secure Setup series.  This series serves to accomplish a primary goal.  To help one quickly and efficiently configure their ubuntu server in such a way that it is very secure, WHILE still providing great usability and maintainability.  A lot of people have great intentions to secure their server, SELinux, detailed ip-tables configurations, log audits, etcetera.  I’ve found that after a short while because of the maintenance involved or the complexity of the configurations, your average user will not keep up with such a configuration and their server ends up exposed in one way or another.  The first part in this series will give you the initial, basic things you should configure or customize to get you on the road to an extremely secure server.  Future parts will go into more depth with varying applications to help you with their configurations.  Each will be straightforward and something you can implement very quickly and maintain quite easily.  Finally I’ll be referencing all this using the Lucid Lynx LTS 10.04 Ubuntu server, however much of this will apply to any distribution or Ubuntu version with minor tweaks.
Congratulations, you’ve built your Ubuntu Server.  First things first.  Don’t “enable” root.  Simply edit the /etc/group file and add your account(for this article’s sake your account = ‘myuser’) to the admin group. This will allow you to run any command with sudo or just sudo -i and you will drop into root’s shell as if you logged in as root. If you had set a password for the root account I suggest you fix it by doing the following:
  • sudo usermod -p '!' root
  • Then check the /var/log/auth.log over the next couple days for  entries like: CRON[14357]: pam_unix(cron:account): account root has expired (account expired)
  • If you see those then run: sudo passwd --unlock root followed by sudo usermod --lock root
Ok so we have a sudo enabled user, you need to securely login to this box without exposing/advertising your server to the internet so let’s lock up ssh a bit. First create a sshlogin group: groupadd sshlogin and add your 'myuser' account to that group with: usermod -a -G sshlogin myuser. Now edit the/etc/ssh/sshd_config file.
Starting near the top you’ll see a line with Port 22. I strongly suggest changing this to another port. Changing this port alone will stop a lot of attacks on your server (you’ll learn more of that in Part II).  For the sake of being easy to remember set that to Port 2222 now.  Most of the defaults are pretty good in the rest of the file but there are a few others we should change so ensure the following are set:
  • PermitRootLogin no
  • X11Forwarding no
  • UseDNS no
  • AllowGroups sshlogin
This last line ensures that only the users in the sshlogin group are allowed to even attempt to login over SSH to your server. This is a good extra bit of security as there are a lot of other default accounts on ubuntu and you don’t want script kiddies trying to utilize an exploit with known user names to get into a system.  Check this ubuntu community page, StricterDefaults for a few other nice tidbits in increasing your default security and apply what fits for your server.  At this point you now have one user, whose user name is not a known default, that is allowed ssh access to your system, on a non-standard ssh port.  Very nice, you are already more secure than most servers out there!  I strongly recommend leveraging this ssh for file transfers (scp) as the data will be encrypted automatically for you.  Just remember using the non-standard port you’d have to specify scp -P 2222 ....first.
In Part II In another article, I’ll introduce a secure way to configure FTP as most people find it easier than scp.
Now let’s address something that is frequently overlooked.  Keeping the system or user installed packages updated. There is a concern that automatically updating packages could break something on your system that was previously working.  Although this is a valid concern, one nice feature of Ubuntu’s package system is the concept of “safe-upgrades” and “full-upgrades”.  Safe-upgrades are meant to be just that, rather safe to do at any time.  Now that doesn’t mean there is zero chance of a safe-upgrade package causing an issue but I’d say that the scale of risk is tipped in favor of applying these versus NOT and leaving your server at risk to needed patches, especially security based ones.  So what we should do is configure a system such that every night it will check for those security updates and apply them to your server.  To get started:
aptitude install unattended-upgrades update-notifier-common
To configure the settings for unattended-upgrades vi /etc/apt/apt.conf.d/50unattended-upgrades
Pay close attention to the first two sections of that file:

// Automatically upgrade packages from these (origin, archive) pairs
Unattended-Upgrade::Allowed-Origins {
"Ubuntu lucid-security";
// "Ubuntu lucid-updates";

// List of packages to not update
Unattended-Upgrade::Package-Blacklist {
// "vim";
// "libc6";
// "libc6-dev";
// "libc6-i686";

What you can see is that I am only allowing updates from lucid-security which is quite safe. You could also add lucid-updates by removing the // from the front of the line and then in the second section enter the packages you do not want to be automatically upgraded. My issue with this is that you need to get the exact package name for the blacklist to work. The blacklist doesn’t support regex’s although unattended-upgrades is written in python and I have seen a simple edit to allow it to support regular expressions, but we won’t be doing that here. I see a lot of forum posts online where people thought they were blocking kernel upgrades but it didn’t match and so their server upgraded its kernel. I’d recommend just automatically doing the lucid-security updates and logging into your server to manually do any others.
Now before any of this will run automatically we need to create/edit one more file: vi /etc/apt/apt.conf.d/10periodic and add this to it:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
Finally we are on our last main section for Part I. You don’t want to be blind to what is going on with your server, but realistically you won’t log in everyday and scour through the logs (linux logs can be insanely detailed!). There is a good program I’ve used for years called logwatch that can easily help you out with this by giving you filtered logs all in one easy to “human brain parse” output. To install simply aptitude install logwatch. Now to tweak its settings a bit go to vi /usr/share/logwatch/default.conf/logwatch.conf. The main thing to decide is how you want to check this. If you’ve configured your server to be able to send outbound emails which is very handy then I recommend having the output of logwatch emailed to you. Otherwise you can set it to output to a file and just check that everyday which is still easier than checking multiple log files and scouring through their content. I find the html output harder to read than the text output so I just have it set to email me once a day. I also like the detail level set to Med. It provides much more valuable information than the default while not giving so much that you end up not reading it.
Finally, before we end Part I you need to make sure some sort of firewall is running. We’ll want to use iptables but one main complaint regarding it is that it can be difficult to maintain and setup if you aren’t accustomed to the way the rules need to work. I find people start with good intentions and configure it, then somewhere down the line they can’t get something working just right, they turn off iptables and the program works. Instead of figuring it out they leave iptables off. So for Part II we’ll go over an easy way to get iptables under your control but in the meantime you should apply some basic rules. You can copy and paste this as a script on your system and run it. You can even easily add a few other ports you have to the script and re-run it. It will always flush all rules and then re-apply based on what’s in the script. Currently it will allow our customized ssh port of 2222, and the two web ports of 80 and 443 which you’ll want open if you are running a web site.
So now create a script like vi iptables_config and paste in the following:

# iptables example configuration script
# Flush all current rules from iptables
iptables -F
# Set default policies for INPUT, FORWARD and OUTPUT chains
iptables -P INPUT ACCEPT
iptables -N MY-Firewall-1-INPUT
# Setup new chain
iptables -A INPUT -j MY-Firewall-1-INPUT
iptables -A FORWARD -j MY-Firewall-1-INPUT
# Set access for localhost
iptables -A MY-Firewall-1-INPUT -i lo -j ACCEPT
# Accept packets belonging to established and related connections
iptables -A MY-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Allow SSH connections on tcp port 2222
# This is essential when working on remote servers via SSH to prevent locking yourself out of the system
iptables -A MY-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 2222 -j ACCEPT
# Allow web
iptables -A MY-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
iptables -A MY-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
# Reject everything else
iptables -A MY-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
# Save settings
# List rules
iptables -L -v

To install iptables if you don’t have it aptitude install iptables iptables-persistent
Run the script to set the rules.
That’s it! This is a tremendous start. As a quick security review here is what you now have:
  • non-default ssh port to stop script kiddies from hammering user login/password guesses
  • per sshlogin group only one non-default user is even allowed to ssh in even if someone were to target you and find your open ssh port
  • daily security updates being applied to minimize risk of an exploitable package causing any harm
  • daily log reports being emailed to you for review
  • nice, concise iptables script to ensure only the ports you intend are open to the outside world
Look out for the Quick Secure Setup Part II, where we will protect our server more by blocking attacks and managing our firewall easier.
Welcome to Part II of the Quick Secure Setup Series.  Be sure to check out Quick, Secure Setup Part I first, although this can be taken on its own if you’d just like to configure UFW with Fail2ban correctly.
At the end of Part I we quickly setup a basic iptables config just to get the firewall up and doing its job.  The problem with iptables, isn’t actually a problem with iptables itself, but rather the administrator running it.  Iptables is a great firewall and like any great firewall there is a lot you can configure it to do.  The more configuration options open to the user, the more complicated a piece of software can get.  What I’ve witnessed is a well intentioned user will configure and run iptables on Day 1 of their server, just as we did in Part 1, but as time moves on and they need to run more applications or find themselves with something not working just right that seems to behave fine once iptables is stopped, then iptables either gets turned off or mis-configured with larger holes than what is needed.  Unless you are a linux administrator of some sort you probably are going to learn just enough of iptables to get it running on that initial setup.  After that you don’t really touch a firewall on a day to day basis so by the time you have this new application installed that isn’t playing nice with your current iptables you don’t want to take the steep learning curve plunge to figure out the correct  configuration you would need.  Therein would lie your chink in the security chain.
Starting with UFW for the first time check the UFW Ubuntu Wiki.  The introduction on this page explains perfectly why one would want to use UFW over iptables.
The Linux kernel in Ubuntu provides a packet filtering system called netfilter, and the traditional interface for manipulating netfilter are the iptables suite of commands. iptables provide a complete firewall solution that is both highly configurable and highly flexible.
Becoming proficient in iptables takes time, and getting started with netfilter firewalling using only iptables can be a daunting task. As a result, many frontends for iptables have been created over the years, each trying to achieve a different result and targeting a different audience.
The Uncomplicated Firewall (ufw) is a frontend for iptables and is particularly well-suited for host-based firewalls. ufw provides a framework for managing netfilter, as well as a command-line interface for manipulating the firewall. ufw aims to provide an easy to use interface for people unfamiliar with firewall concepts, while at the same time simplifies complicated iptables commands to help an adminstrator who knows what he or she is doing.
You may also find useful the Ubuntu Community page on UFW.  There are helpful links at the bottom of that page to continue reading.  Some quick google searches will get you moved to UFW quite easily.  Remember you are using the same backend as iptables just using a less complicated front-end to get the rules going.  So if you flush your current iptables and put in some basic ufw rules for your ssh, apache, and in my case NTP your ufw status output could look like this:

activeTo      Action   From
--------      ------   ----
OpenSSH       LIMIT    Anywhere
Apache Full   ALLOW    Anywhere
123           ALLOW    Anywhere

Now recall we changed our OpenSSH port in Part I, so to keep UFW simple I’ve edited the openssh-server file in /etc/ufw/applications.d to reflect our custom port. For any custom ports you find yourself opening or blocking consider creating an application profile for it, it’ll be easier to read your rules and if you don’t touch them for months, easier to remember when you have to re-visit them. You can easily see your apps and their configuration with ufw app list and ufw app info OpenSSH. Once you have your basic UFW configuration in place you should install fail2ban: aptitude install fail2ban.
Fail2ban is a very simple yet very useful application that simply looks at the log files you tell it about, parses them for certain errors or failures and then inserts a firewall rule to block the IP that caused that error or failure.  Trust me when I tell you that you want this.  Every server I’ve ever put on the internet gets scanned by scripts looking for open ports, trying ssh or ftp logins, attempting urls for various mysql, php, remote access URL’s, etcetera etcetera etcetera….  Here is a small example of ports that were scanned on my server:

Service: ms-sql-s (tcp/1433) ([UFW BLOCK])
Service: ssh (tcp/22) ([UFW BLOCK])
Service: sip (udp/5060) ([UFW BLOCK])
Service: 3389 (tcp/3389) ([UFW BLOCK])
Service: 27977 (tcp/27977) ([UFW BLOCK])
Service: radmin-port (tcp/4899) ([UFW BLOCK])
Service: 5900 (tcp/5900) ([UFW BLOCK])
Service: http-alt (tcp/8080) ([UFW BLOCK])
Service: loc-srv (tcp/135) ([UFW BLOCK])
Service: mysql (tcp/3306) ([UFW BLOCK])
Service: ms-sql-m (udp/1434) ([UFW BLOCK])
Service: 49153 (udp/49153) ([UFW BLOCK])
Service: 1022 (tcp/1022) ([UFW BLOCK])
Service: socks (tcp/1080) ([UFW BLOCK])

On the web server side various URL’s for web administration are always attempted like: /phpMyAdmin, /myadmin, /mysql, etcetera. Without Fail2ban in place these scripts can run until they’ve exhausted every login attempt they want, or every URL in their list. WITH Fail2ban we can give them 3-5 attempts and then realize they are a script kiddie and ban their IP from the server for X amount of time.
Out of the box Fail2ban works with iptables rules, however these don’t play nice with our simpler UFW commands so we need to make a couple edits to have Fail2ban block the IP’s with UFW.
First lets go into /etc/fail2ban/jail.conf and change a few default ban actions for ssh and apache to use ufw actions we will create:

enabled = true
banaction = ufw-ssh
port = 2992
filter = sshd
logpath = /var/log/auth.log
maxretry = 3

enabled = true
port = http,https
banaction = ufw-apache
filter = apache-auth
logpath = /var/log/apache*/error*.log
maxretry = 4

enabled = true
port = http,https
banaction = ufw-apache
filter = apache-nohome
logpath = /var/log/apache*/error*.log
maxretry = 3

enabled = true
port = http,https
banaction = ufw-apache
filter = apache-noscript
logpath = /var/log/apache*/error*.log
maxretry = 6

enabled = true
port = http,https
banaction = ufw-apache
filter = apache-overflows
logpath = /var/log/apache*/error*.log
maxretry = 2

In this file we are enabling the sections we want fail2ban to monitor and take action on. You’ll want to make sure your logpath points to your apache error logs and take note of the filter names as each of those corresponds to a file within the filter.d directory. All the filters are simply a regular expression to pattern match some error condition in the logfile, once matched fail2ban will execute the banaction. So if you look at the apache-auth filter, it will match for any user authentication failures to your websites. The only filter I’ve modified is the apache-nohome I’ve edited to match for any file not found error, not just checking for home directory attacks as the default.
The original regex was:
failregex = [[]client []] File does not exist: .*/~.*
and my modified version for any file not found errors is:
failregex = [[]client []] File does not exist: *
BE CAREFUL if you chose to also make this change. There are many things that will cause file not found errors that may not be attacks at all. Search bots looking for robots.txt, normal users can trigger on favicon.ico if you don’t have, etc. So if you make that change check your logs frequently and fix any valid file not found errors. The reason I turned this on is the constant attempts at bogus URL’s as I mentioned above where the scripts look for web GUI admin pages.
Now we simply need to create the valid banaction files we specified in our jail.conf. First is /etc/fail2ban/action.d/ufw-ssh.conf:

actionstart =
actionstop =
actioncheck =
actionban = ufw insert 1 deny from to any app OpenSSH
actionunban = ufw delete deny from to any app OpenSSH

and /etc/fail2ban/action.d/ufw-apache.conf:

actionstart =
actionstop =
actioncheck =
actionban = ufw insert 2 deny from to any app "Apache Full"
actionunban = ufw delete deny from to any app "Apache Full"

As you may see the ufw command here is quite simple. The actionban says to deny the offending IP for the specified application. The only gotcha here is we have to specify the line the rule is being inserted into as the order matters. Our original rules allow these apps so we must ensure that any denies to these apps come BEFORE the allow rule. As rules are processed in order if we have the allow first the offender will continue to hit our server as it will never hit the deny rule. So we make sure the denies get inserted before the allow lines and all is well. The great thing about UFW rules is that you can almost read them and understand what they are doing, as opposed to the standard iptables banaction which could look like this:

actionban = iptables -I fail2ban- 1 -s -j DROP
actionunban = iptables -D fail2ban- -s -j DROP

As you can see UFW provides for much more readability without that learning curve hit you’d have to go through to get a good grip on the iptables rules.
If you have logwatch configured as in Part I then you’ll see the bans that took place in the logwatch email for the day prior. For example the fail2ban section in one of my logwatch emails had this:

--------------------- fail2ban-messages Begin ------------------------
Banned services with Fail2Ban:                 Bans:Unbans
apache-filenotfound:                            [ 3:3 ] (37-141.80-90.static-ip.oleane.fr)   1:1 (adsl92-82-225-197.romtelecom.net)  1:1                                    1:1
---------------------- fail2ban-messages End -------------------------

As I mentioned earlier, for the first week with this configuration you should check your apache error log and make sure those file not found errors were scripts looking for /phpmyadmin or some other page that truly doesn’t exist and not a normal user getting file not found errors because of favicon.ico or something else.
That’s it! ufw status will show you any of the rules in effect on your system. After these first two parts are executed you’ll have a server configured securely with nothing unnecessary open to the internet, and those ports that are open now are blocking some bad guys from messing with them too much. For Quick Secure Setup Part III, the last in the series, we’ll tighten everything up even more and end up with server security that is second to none.

No comments:

Post a Comment