A Prelude to better things – Open Source and IDS

A recent number of attempted break-ins to a few machines I manage has had me thinking again about the overall security of the machines, and how to get a better handle on what’s going on. This isn’t something new – anyone managing internet-facing systems ought to be aware of the dangers, and how to mitigate them. As with many things related to open source, there are a plethora of tools out there aimed at providing alerting based on network activity.

I’d wager that many people have heard of Snort, and what it does. For those who aren’t familiar with it, it’s an open source intrusion detection system (IDS)/intrusion prevention system (IPS). In a normal configuration, Snort monitors traffic and alerts based on predefined rules for such things as port scans and maliciously-crafted HTTP requests. It’s an extremely powerful tool that is also highly configurable, and with an excellent community that provide custom rules for a wide variety of situations. But alerting is one thing – being able to make sense of those alerts is something else.

Prelude

Prelude is a security information management (or SIM) system – that is, it’s designed to aggregate and correlate events from tools like Snort and provide a centralised place to manage those events. On its own, this is useful, but coupled with a few additional tools it really becomes something else.

The main prerequisite for our setup is to MySQL, so if you don’t already have it installed, go ahead and do so. Most Linux distributions include Prelude in their repositories, so installing it should be pretty straightforward. I use Debian, so once MySQL is installed and working, installing the manager is a case of running the following:-

root@dev-vm-lnxd-01:~# apt-get install prelude-manager

Follow the on-screen debconf prompts regarding the database – one thing to note that I discovered on squeeze is that if I let debconf generate a password for the prelude database user, that password wasn’t written to the prelude configuration. Since debconf doesn’t output the password it’s generated, this means you’ll have no idea of the password. Therefore, I recommend picking a password yourself.

Next, it will generate a 2048-bit RSA key for the manager. This can take a while, and on a quiet server will take a few minutes at least. Generating disk I/O is how I usually increase the amount of entropy for the key generation – running find / >/dev/null works well in this situation I’ve found.

By default, prelude-manager will refuse to start, because it’s disabled in /etc/default/prelude-manager. Edit it, and change RUN=no to RUN=yes. Next make sure that the database configuration in /etc/prelude-manager/prelude-manager.conf is correct – especially regarding my note earlier about auto-generated passwords.

When you’re happy the configuration is correct, start prelude-manager:-

root@dev-vm-lnxd-01:~# /etc/init.d/prelude-manager start

If everything is okay, prelude-manager will start.

Putting it to work

We now have a running prelude-manager instance. But on its own, this is pretty useless – no events are being sent to it, so there’s nothing for it to do. Let’s fix that by installing a sensor – Snort.

Again, installing Snort should be straightforward:-

root@dev-vm-lnxd-01:~# apt-get install snort

As before, follow the debconf prompts. Unlike prelude-manager, snort will be started automatically after install. A quick glance at /var/log/snort will reveal a file named alert and possibly a number of tcpdump.log files. This is how Snort by default saves alerts, with the tcpdump.log files being captures associated with those alerts. But since we’re talking about Prelude, let’s get the alerts sent there.

The first step is to register Snort with prelude-manager, so that prelude-manager knows about it. We can do this with the prelude-admin command:-

root@dev-vm-lnxd-01:~# prelude-admin register "snort" "idmef:w" 127.0.0.1 --uid 106 --gid 106

You’ll notice that I passed the UID and GID of the snort to prelude-admin. By default, the snort has both a UID and GID of 106, but do check this before running it. The other options are the name of the agent (“snort”), the permissions (“idmef:w”) and the IP address of the server running prelude-manager. The name can be anything, as long as this matches the name used in the Snort configuration, and the ‘idmef:w’ refers to the Intrusion Detection Message Exchange Format, and that we want to give Snort write permissions. Finally, because we’re running Snort on the same server as the one running prelude-manager, we use 127.0.0.1 as the host.

root@dev-vm-lnxd-01:~# prelude-admin register "snort" "idmef:w" 127.0.0.1 --uid 106 --gid 106
Generating 2048 bits RSA private key... This might take a very long time.
[Increasing system activity will speed-up the process].
Generation in progress... X..+++++O.+++++O

You now need to start "prelude-admin" registration-server on 127.0.0.1:
example: "prelude-admin registration-server prelude-manager"

Enter the one-shot password provided on 127.0.0.1:

Once again, an RSA key is generated, but this time for the sensor, so now might be a good time to go and make a cuppa. Once the key has been generated, you’ll be prompted (as above) for a ‘one-shot password’ to authenticate it to prelude-manager. At this point, open a new session to the server running prelude-manager and run prelude-manager as directed by the above:-

root@dev-vm-lnxd-01:~# prelude-admin registration-server prelude-manager
The "8qovmgff" password will be requested by "prelude-admin register"
in order to connect. Please remove the quotes before using it.

Generating 1024 bits Diffie-Hellman key for anonymous authentication...
Waiting for peers install request on 0.0.0.0:5553...
Waiting for peers install request on :::5553...

Make a note of the password (in our example, 8qovmgff), switch back to our first session, and provide it at the prompt:-

Enter the one-shot password provided on 127.0.0.1: 8qovmgff
Confirm the one-shot password provided on 127.0.0.1: 8qovmgff

Connecting to registration server (127.0.0.1:5553)... Authentication succeeded.

Switch again to the second session, and accept the registration:-

Connection from 127.0.0.1:40123...
Registration request for analyzerID="2783582516549275" permission="idmef:w".
Approve registration? [y/n]: y
127.0.0.1:40123 successfully registered.

Snort is now registered as an sensor with prelude-manager.

The next step is to tell Snort to send its output to Prelude. Edit /etc/snort/snort.conf, and look for the following line:-

# output alert_prelude: profile=snort-profile-name

Uncomment it, and change the profile name to snort:-

output alert_prelude: profile=snort

Save the config, and restart snort:-

root@dev-vm-lnxd-01:~# /etc/init.d/snort restart
Stopping Network Intrusion Detection System : snort (eth0 ...done).
Starting Network Intrusion Detection System : snort (eth0 using /etc/snort/snort.conf ...done).

Making sense of it all

All being well, Snort should now be sending events to Prelude. But we still don’t have any visibility of these alerts… which is where Prewikka comes in.

Prewikka is a graphical front-end to Prelude – more specifically, to the database that prelude-manager uses. Again, there’s a package in Debian for this, so go ahead and install it:-

root@dev-vm-lnxd-01:~# apt-get install prewikka

As with prelude-manager, follow the debconf prompts regarding database setup. Once completed, Prewikka will have its own database, but you’ll also need to give it the details for prelude-manager’s database. Edit /etc/prewikka/prewikka.conf, and edit the settings under the [idmef_database] section. Save and exit the config, and start Prewikka:-

root@dev-vm-lnxd-01:~# prewikka-httpd &

By default, Prewikka listens on port 8000, so point your browser at http://<server>:8000/. The default username and password is admin and admin, so go ahead and log in. You’ll then be presented with the main event viewer, which should be similar to the screenshot to the right.

Running Prewikka in this way uses its own built-in webserver, but within the package a prewikka.cgi file is provided, which can be served by your favourite webserver of choice as a traditional CGI executable.

Now, before taking the screenshot to the right I cheated a little. I ran a quick nmap against the server to generate some events as an example, but in the absence of any events you can still check that snort is communicating with prelude-manager. Click on the Agents link in the menu, and there should be an entry for the server. Click on it once to expand it, and then on the Total box to expand the sensors. If everything is configured correctly, Snort should appear in the listing:-

Prewikka allows you to filter and sort the events displayed, so now would be a good time to have a play about with it. Clicking on an alert will let you view the details of that alert, and because the events at the moment are from Snort, all the relevant alert information that Snort would have logged to /var/log/snort/alert should be available.

Add logs, and sprinkle in a bit of correlation

So now we have working Prelude and Snort installs, with the two working together and a nice front-end to view them through. While nice to look at, we’re missing one of the things that all good IDS tools have, which is correlation.

Before going any further, let’s install two more Prelude packages – prelude-lml (Prelude Log Agent) and prelude-correlator:-

root@dev-vm-lnxd-01:~# apt-get install prelude-lml prelude-correlator

On my squeeze install, the post-install script for prelude-lml exits with an error because the service is initially disabled. This is fixed in a later version, but it’s something to look out for. For now, ignore the error – after we’ve configured it we can pacify dpkg…

Because prelude-lml and prelude-correlator are sensors themselves, we’ll need to register them with prelude-manager in the same way that we did for Snort with the prelude-admin command:-

root@dev-vm-lnxd-01:~# prelude-admin register "prelude-lml" "idmef:w" 127.0.0.1 --uid 0 --gid 0
...output from prelude-admin...
root@dev-vm-lnxd-01:~# prelude-admin register "prelude-correlator" "idmef:rw" 127.0.0.1 --uid 0 --gid 0
...output from prelude-admin...

As before, you’ll need to run prelude-admin registration-server prelude-manager in a second session for both sensors. Also, make sure that you correctly give prelude-correlator the ‘idmef:rw’ permissions – this is because it needs to both read and write events. Then, enable prelude-correlator in the same way as prelude-manager by changing RUN=no to RUN=yes in /etc/default/prelude-correlator. Finally, start prelude-correlator:-

root@dev-vm-lnxd-01:/etc/prelude/profile# /etc/init.d/prelude-correlator start
Starting prelude-correlator : prelude-correlator21 Jan 03:06:57 prelude-correlator (process:22339) INFO: [FirewallPlugin]: disabled on user request
21 Jan 03:06:57 prelude-correlator (process:22339) WARNING: SpamhausDropPlugin = PreludeCorrelator.plugins.spamhausdrop:SpamhausDropPlugin: No module named netaddr
21 Jan 03:06:57 prelude-correlator (process:22339) INFO: [BusinessHourPlugin]: disabled on user request
21 Jan 03:06:57 prelude-correlator (process:22339) INFO: 7 plugin have been loaded.
.

A word of warning at this point – initially, prelude-correlator failed to start for me. If this happens, make sure that /etc/prelude/prelude-correlator, /var/spool/prelude/prelude-correlator and /var/lib/prelude-correlator (plus any subdirectories) are owned by the prelude-correlator user, and then again try to start prelude-correlator.

If you had problems earlier with the dpkg post-install script for prelude-lml, running apt-get -f install here will tidy that up and start prelude-lml. If you didn’t, then start prelude-lml manually:-

root@dev-vm-lnxd-01:/etc/prelude/profile# /etc/init.d/prelude-lml start
Starting Prelude LML: prelude-lml.

Going back to our Prewikka browser window, if we click again on Agents and expand the nodes and sensors, we should see entries for prelude-lml and prelude-correlator.

To demonstrate what prelude-lml and prelude-correlator can do, let’s first add an iptables entry to log all TCP connection attempts on port 22:-

root@dev-vm-lnxd-01:~# iptables -I INPUT -p tcp --dport 22 -m state --state NEW -j LOG

Opening another SSH connection to the server should result in a log message in /var/log/messages similar to the following:-

Jan 21 03:17:08 dev-vm-lnxd-01 kernel: [ 7981.313741] IN=eth0 OUT= MAC=xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx SRC=192.168.xxx.aaa DST=192.168.xxx.yyy LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=1313 DF PROTO=TCP SPT=56168 DPT=22 WINDOW=14600 RES=0x00 SYN URGP=0

Next, purposefully fail an attempt to log into the server – for example, with an incorrect password. Then, check the events again:-

There’s a few things here, but the main thing to notice is that there are a number of new sensors listed in the Analyzer column. PAM, sshd, and netfilter have all been picked up by prelude-lml from the logfiles it monitors, and in the case of the first event, prelude-correlator has correctly identified – from the prelude-lml events – that a brute-force attack has occurred. In my example above, it’s also picked up that there was a successful login – this was me logging in with the correct password, but if this was a production system this may well be indicative of a brute-force attack resulting in a correctly-guessed password!

Where now?

This is just a quick overview of what’s possible with open-source IDS software. All the tools I’ve written about in this post are extremely configurable – prelude-lml for example can monitor many different types of logfile and can be configured with custom regular expressions to look for specific things. Many similar tools can be configured to send events to Prelude, which prelude-correlator can correlate as above. One thing I’ve not covered here is auditd, which has (via the audispd multiplexor) the ability to send to Prelude – something which I’ll cover in a future post.