Better netflow visualization, part II

I’ve been looking into netflow visualization tools since my little experiment last week and have seen a couple of interesting visualization tools. I checked out an article on IT World on visualizing netflow, which pointed to AfterGlow and GraphViz; I also looked at Plixer Labs’ blog; they offer commercial netflow visualization tools. Finally, I’ve been browsing through resources mentioned in Raffael Marty’s secviz.org site. Though these sites offer up impressive information on visualization tools (both commercial and free), I was unable to find anything about time-stepped visualization of traffic. This may in fact be a testament to my laziness and/or lack of trying, in which case I do apologize.

I figured that I should give time-stepped traffic visualization a shot on my own; so I dusted off my old college book on OpenGL and went to work figuring out how I might code a tool similar to codeswarm’s but intended for large volumes of traffic.

A quick shout-out to my wife (KRED on Research Salad) is in order, I believe. Kay, if you’re reading this: happy birthday, babe. Thanks for the ten years of laughs, sending me my favorite comic books from across the pond, and regularly assaulting me with a gazillion infosec links – don’t know how you can read all of them, I certainly can’t! Finally, thanks for putting up with the late nights and early mornings working, supporting this crabby ol’ geek through thick and thin!

Better netflow visualization with code_swarm coolness!

Howdy all,

In my last post, I may have mentioned codeswarm, a nifty tool for visualizing how frequently a software project gets updated throughout time. Since it’s an open-source project, I figured that it was worth having a look at the code and seeing if there are other uses for it.

If you check out the Google Code page, you’ll notice that the project isn’t terribly active – the last upload dates back to May 2009. But hey, it does what it’s supposed to do and it’s pretty straightforward.

Reading through the source files, in fact, use of the tool is super simple: you set up an XML file that contains the data to be used, you run Ant, and you let the program do the rest. The format of the sample data is very simple, frankly: a file name, a date, and an author.

So let’s see what other uses we could come up with. Here are a few ideas I thought might be cool:

  • What about adapting it to track your social media messages? First, if you’re following a lot of people, it would look wicked cool. Second, if you’re trying to prune your Follow list, that could be really practical for figuring out who’s the noisiest out there.
  • Sometimes when you’re trying to figure out bottlenecks in your traffic, it’s useful to have a decent visualization tool. Maybe this could be helpful!
  • Finally, you sometimes need a good way to track employee activities. Would this not be a kickass way to see who’s active on your network?

I decided to work on the second idea. I’m not looking to rework the code at this point, just to reuse it with a different purpose.

Prerequisites

To pull this off, you’re going to need the following:

  • The codeswarm source code and Java, so that you can run the code on your system
  • Some netflow log files to test out
  • flow-tools, so that you can process said netflow log files
  • A scripting language so that you can process and parse the netflow traffic into XML. My language of choice was ruby, but it could be as simple as bash.

 The netflow filter

Before we can parse the netflow statistics into the appropriate format, we need to know what we’ll be using and how to extract it. Here’s what I used: each IP endpoint should have its own line; the IP address maps to the “author” field (because that’s what is visible). The protocol and port will map to the filename field, and the octets in the flow will map to the weight field.

The following is the netflow report config file. You should save this in the codeswarm directory as netflow_report.config:

stat-report t1
 type ip-source/destination-address/ip-protocol/ip-tos/ip-source/destination-p$
 scale 100
 output
  format ascii
  fields +first
stat-definition swarm
 report t1
 time-series 600

If you save some netflow data in data/input, you can test out your report by running this line:

flow-merge data/input/* | flow-report -s netflow_report.config -S swarm

Parsing the netflow

If the report worked out correctly for you, the next logical step is to write the code to create the .XML file that will be parsed by codeswarm. You’ll want to set your input directory (which we’d said would be data/input) and your output file (for instance, data/output.xml).

Here’s the source code for my grabData.rb file:

#!/usr/local/bin/ruby
# Prepare netflow data for codeswarm.
$outputFilePath = "data/output.xml"
$outputFile = File.new($outputFilePath, "w")
$outputFile << "<?xml version="1.0"?>n"
$outputFile << "<file_events>n"
# Grab the netflow information using flow-tools
$inputDirectory = "data/input"
$input = `flow-merge data/input/* | flow-report -s netflow_report.config -S swarm`
# This is the part that gets a bit dicey. I believe that in order to properly visualize
# the traffic, we should add an entry for each party of the flow. That's exactly what we're
# Going to do. The "author" in this case is going to be the IP address. The "filename" will
# be the protocol and port. The weight will be the octets.
$input_array = $input.split("n")
$input_array.grep(/recn/).each do |deleteme|
 $input_array.delete(deleteme)
end
$input_array.each do |line|
 fields = line.split(",")
 last = fields[0]
 source = fields[1]
 dest = fields[2]
 srcport = fields[3]
 dstport = fields[4]
 proto = fields[5]
 octets = fields[8].to_i / 1000
$outputFile << " <event filename="#{proto}_#{srcport}" date="#{last}" author="#{source}" weight="#{octets}"/>n"
 $outputFile << " <event filename="#{proto}_#{dstport}" date="#{last}" author="#{dest}" weight="#{octets}"/>n"
end
$outputFile << "</file_events>"
$outputFile.flush
$outputFile.close

And we’re done! This should generate a file called data/output.xml, which you can then use in your code swarm. You can either edit your data/sample.config file or copy it to a new file, then run ./run.sh.

Reality Check

I was really excited when running my first doctored code swarm; unfortunately, though the code did work as expected, the performance was terrible. This was because the sample file that I used was rather large (over 10K entries). Probably considerably more than what the authors had expected for code repository checkins. Also, I suspect that my somewhat flimsy graphic card is unable to handle realtime rendering of the animation, so I set up the config file to save each frame to a PNG so I could reconstitute the animation later. Syntax for this is:

ffmpeg -r 10 -b 1800 -i %03d.jpg test1800.mp4

Moreover, I believe my scale was off; I changed the number of milliseconds per frame to 1000 (1 frame, 1 second).

The second rendering was much more interesting, but it did yield a heck of a lot of noise; let’s not forget that we’re working with hundreds, if not thousands, of IP addresses. However, if we do a little filtering we can probably make the animation significantly more readable.

All in all, this was a rather fun experience but a bit of a letdown. Codeswarm wasn’t meant to handle this high a volume of data, so it makes things tricky, and less readable than what I expected; if you play with your filters, you will definitely be able to see some interesting things but if you’re looking for a means to visually suss out what’s happening on your entire network, you are bound to be disappointed. By next time, I hope to talk a bit about more appropriate real-time visualization tools for netflow and pcap files, maybe even cut some code of my own.

>”Inventory fun” follow-up… AKA Kace’s built-in service tag and warranty report

>Back in September, I wrote a script to grab Dell machines’ warranty information based on their service tags, which I had retrieved from Kaseya or LANSweeper. A pretty nifty trick, or so I thought…

Since then, I’ve been introduced to Kace — a suite of Dell tools for inventory management, scripting, software and patch deployment, and ghosting. Think of it as a solution that offers the functionality of your FOG server, LANSweeper and Kaseya.

The Kace solution is actually divided in two parts: one component handles inventory, application and update deployment, and scripting, while the other component handles ghosting and driver management. A “component”, in this context, can be a piece of hardware (a physical server that you connect to your LAN – the O/S is a custom Unix distro) or simply a virtual machine (a VMWare application which can run on your existing ESX or ESXi box). The config is rather light — the only thing these devices need is storage space. For those of you that are looking for a free / SOHO-level solution for all your sysadmin  needs: stick to OSCInventory, Zabbix, and FOG… These things have a price tag.  However, though not free, these are well worth it in my opinion.

I digress. I’ve been porting a lot of my existing stuff to Kace recently; this week, I’m working on the inventory report from September. It turns out that my work is pretty much done: perhaps unsurprisingly, Kace already has a report for extracting machine names, service tags and expiry dates. They have two boiler-plate reports, dubbed “Dell Warranty Expired” and “Dell Warranty Expires in the next 60 days”, which dishes out all the information you may need in HTML, CSV or TXT format.

In point of fact, I needed something a bit more customized; I don’t actually need the full warranty information but rather the date the warranty expires. This is because with our clients, the machines are amortized from the moment we get them to the moment they’re no longer covered under warranty.

The nice thing about Kace is that you can take a boiler-plate report, duplicate it and change the SQL request directly, like with LANSweeper. This makes reporting a cinch. Here’s my final report for all machines on my campus:

SELECT DISTINCT M.NAME AS MACHINE_NAME,M.CS_MODEL AS MODEL, DA.SERVICE_TAG, DA.SHIP_DATE, M.USER_LOGGED AS LAST_LOGGED_IN_USER,
DW.END_DATE AS EXPIRATION_DATE
FROM KBSYS.DELL_WARRANTY DW
LEFT JOIN KBSYS.DELL_ASSET DA ON (DW.SERVICE_TAG = DA.SERVICE_TAG)
LEFT JOIN MACHINE M ON (M.BIOS_SERIAL_NUMBER = DA.SERVICE_TAG OR M.BIOS_SERIAL_NUMBER = DA.PARENT_SERVICE_TAG)
WHERE M.CS_MANUFACTURER LIKE ‘%dell%’
AND M.BIOS_SERIAL_NUMBER!=”
AND DA.DISABLED != 1
AND DW.END_DATE = (SELECT MAX(END_DATE) FROM KBSYS.DELL_WARRANTY DW2 WHERE DW2.SERVICE_TAG=DW.SERVICE_TAG AND DW2.SERVICE_LEVEL_CODE=DW.SERVICE_LEVEL_CODE);

It gives you a nice simple list with the machine name, model, service tag, shipment date, warranty expiry date, and the user that last logged on to the system.  Cool eh? Or maybe I’m just easily impressed. Regardless, it saves me time… Yay!

>A honeypot solution from start to finish

>
Operating System and tools
Pick an operating system with which you’re comfortable. A lot of *nix junkies out there will heckle you about which distro is best, especially when it comes to running security tools; and whilst I agree with the principle that a good solid distro will improve your machine’s robustness and prevent a malicious attacker from turning your security tools against you, let’s be realistic: there isn’t a single distro, operating system or device out there that can’t be exploited. This is not always due to the shortcomings of the developer, or administrator, or what have you: it is the result of a complex balance between security, functionality, communication and logistics. So what I say is, pick *one* distro and get to know it very well. Make sure it can patched on a regular basis and that any remote communication you set up with it is secured (encrypted, with a long password or certificate for authentication). For this example, I’m going to use an Ubuntu box, honeyd, swatch and ruby to set up the honeypot and monitoring systems.

OS setup
I would strongly recommend starting with a fresh install of your distro on the system. You don’t want to leave any unexpected services running on your box as it will increase its vulnerability. Install the bare minimum — if you’re comfortable with a barebones system with a simple shell and no xwin, go for it; just make sure that you can handle the config under stress.

Once the OS is installed, I’d test it right away to make sure no superfluous services are running on it. Start with several nmap scans; begin with a simple scan, use the -P0 flag, scan all ports consecutively, use the –send-ip flag… The works. I’d check out all the processes on your machine as well, to see if anything could be removed. Then use nessusd as a final test.

Tools setup
Next, install honeyd, farpd, rrdtool, rails, swatch and, optionally, snort. If you’re using APT, then the startup scripts and config files for honeyd and farpd will be setup (in /etc/init.d and /etc/default). These will not run automatically on startup – amongst other things, you’ll need to specify the interface and IP addresses on which these tools will listen.

To give you a bit of background, farpd was developed by Dug Song as part of dsniff (a collection of tools for network auditing and pentesting). What it does is reply to ARP requests, effectively directing traffic to your host. Whilst it can be used for nefarious purposes, in this case it allows your honeypot to capture traffic for multiple IP addresses. Rails is the tool I’ll be using for scripting — it’s just a preference. One could use anything from a simple bash script to a compiled mono exec; just remember that you want to be able to modify it fairly quickly and painlessly. The scripts in question will be for emulating services (such as IIS) but also for monitoring purposes (such as sending mail). As for swatch, it’s a tool that allows you to quickly set up monitoring of log files; it’s nifty because you can set all sorts of thresholds, filter the log by keyword, and send out alerts via e-mail or script. Finally — the pièce de résistance — honeyd is the tool you’ll be using to simulate other systems. It’s easily configurable and simulates machine profiles and network topologies. It uses rrdtool for this, so make sure that’s on your machine.

Configuring honeyd and swatch
The first thing to do is to make a copy of /etc/honeypot/honeyd.conf, which you’ll rename to honeyd.conf.orig. This will serve as a reference for your config. The configuration is super straight-forward and gives examples for a virtual network and several hosts. For each host, you’ll need to define the OS profile as labelled in the nmap.prints file, set up the services and bind an IP address. For each service, you indicate the port, transport protocol and action; the latter can be a command, such as “echo You’ve been 0wned get outta Dodge” or something a bit more complex like “ruby /opt/scripts/my_iis_emulator.rb” — check out honeyd’s site for a useful list of service scripts. Alternatively, you can set up the port to proxy a service running on another machine (including the source IP’s, using $ipsrc). If you’re feeling particularly nasty, you could fathomably proxy a malicious server running exploits — but I wouldn’t recommend that as it could seriously backfire on you…

In order to get logging going, you’ll have to change ownership of the /var/log/honeypot directory, using a command like ‘sudo chown -R honeyd.honeyd /var/log/honeypot’. You’ll need to perform a chmod on . and honeyd.log, something like ‘sudo chmod 777 /var/log/honeypot/.;sudo chmod 777 /var/log/honeypot/honeyd’ (I would say 766 is better). If honeyd.log does not exist, create it using touch. If this doesn’t work, you could chown it to nogroup.nobody.

Swatch is also fairly easy to set up. It needs a config file, which you’ll create from scratch. Place it somewhere logical, i.e. /etc/swatchrc. You’ll want to read the man file very carefully, but in essence you can set up the alerts very quickly using syntax like:

watchfor /tcp|udp|icmp/

exec=/path/to/ruby/script

threshold type=limit,count=1,seconds=90

The example above watches for the values ‘tcp’, ‘udp’ and ‘icmp’ as honeyd is wont to output, limits swatch to trigger a maximum of once every 90 seconds and sets the action to run a ruby script of your choice.

Setting up the startup scripts
The scripts are pretty much set up; if you’re using ubuntu even the honeyd user is created for you. You’ll need to edit the /etc/default/farpd and /etc/default/honeyd files so that they know which interface and ip addresses to use. Note that to specify IP addresses, enter them in separated by spaces.

The swatch script will need to be created by making a copy of the template init.d script (/etc/init.d/skeleton) and modifying it to execute /usr/bin/swatch with the -c (config file path) -t (honeyd.log path) flags set. Sounds more complicated than it actually is, but if you’re having trouble, google ‘init.d skeleton startup script’

Running the services for the first time
Easy: start up a terminal session and type ‘sudo /etc/init.d/farpd start;sudo /etc/init.d/honeyd start;sudo /etc/init.d/swatch start’. You should get no errors and if you run ‘ps aux’ you should see all processes running.

Testing your machineFirst thing to do is ping the fake machines; if that works, the next step is running nmap with the –send-ip flag (note that from the LAN this is absolutely necessary). Finally, try a few telnet sessions to open ports to make sure that honeyd works and swatch triggers correctly.