Analyzing SiLK Netflow data visually

Deep statistical visibility into your network traffic is a fundamental requirement for any serious security conscious enterprise.  While direct packet capture  will remain the best primary source of deriving these statistical insights, Netflow and cousins (JFlow,IPFIX,sFlow)  are the easiest to deploy and give you the biggest bang for the buck.

SiLK (acronym for System for Internet Level Knowledge) is a suite of open source  Linux tools for collection, storage, and analysis of flow data.  SiLK is created by the NetSA group at US-CERT

The two categories of SiLK tools are storage and analysis. The main tool for receiving compressing and storing flow records, called packing in SiLK terminology is rwflowpack  The main tool for querying is called rwfilter These are very flexible command line tools that follow the typical Unix idioms of composability. You can pipe outputs of various commands and build your own query tools. In the hands of a skilled analyst these can be  incredibly powerful.

However I found it quite hard to do some things in SiLK

  1. Analyze multiple things at once.  It is a query-response model that goes back to the primary data source for each analysis item.
  2. Visualize time-series metrics. The query results are usually in terms of flows or aggregates or top-N lists.   You can use rwcount to generate some basic time binned stats.

Trisul is a free real time streaming analytics platform which can supply some of the missing pieces to SiLK such as

  1. Time-series. From the flow data extracts hundreds of metrics and stores in a time-series. See some examples below
  2. Single pass analysis. Like all streaming platforms, Trisul needs to look at the flow data just once and various algorithms extract and store all data in a compact format. The raw flows are also packed, indexed, and stored for ad-hoc querying.
  3. Power User Interface. This is probably the most important addition. You have access to dozens of dashboards, Trisul’s second order metrics like “Number of Active Flows” , “Flow creation rate”, “Cardinality counters such as Unique Hosts per Port” etc.

Without disturbing an existing live SiLK deployment and toolchain, we can use a SiLK tool  rwcat to stream binary flow records to Trisul.  Since Trisul has a fully customizable inputfilter LUA API, we attach it to the output from rwcat like so

SiLK rwcat to export records

Up and running

Here are quick steps to get it working.

  1. Install Trisul using apt-get or yum. It is free no signups required. The limitation of the free version is you can only store max 3 day window of data.
  2. Create a new analysis context to hold the data. To do that type trisulctl_probe create context  silk11
  3. Download the following two LUA files from Github ( trisul-scripts ) into a directory say “/tmp”
    1. flowinput.lua  :  The helper library to process flow like data into Trisul
    2. silk.lua :  Reads binary SiLK records from the named pipe
  4. Create a named pipe mkfifo /tmp/silkpipe  This is the connector pipe.
  5. Run rwcat over your packed files and write to the pipe rwcat --ipv4-output --compression=none file1.17 -o /tmp/silkpipe  Currently the script only handles IPv4 so we specify the –ipv4-output flag.  Replace the file1.17 with your own list of SiLK dump files.  At this time rwcat will appear to hang because there is no one at the other end of the pipe yet.
  6. Run trisul trisulctl_probe importlua /tmp/silk.lua /tmp/silkpipe context=silk111
  7. Wait for the process to complete you can tail the log to check progress. Type trisulctl_probe ; then when you are inside the CLI tail the log from the probe log  log silk111@probe0 log=ns tail

So the whole command line looks like this

On 1 terminal

On Terminal 2

 

When the process completes. You can log on to the Web Interface and view the various dashboards, access the results, query flows and conduct further analysis.

Dashboards inside Trisul

Give it a shot and let us know how it works for you . We should be able to support IPv6 flows too via rwcat  currently we dont have a use for it. Let us know in the comments section if you need that support.

How silk.lua works

Trisul has a full featured LUA API that allows both the packet pipeline and the analytics pipeline to be programmed.  There are about 16 different script types which let you do everything from handle packets, reassembled flows, reconstructed HTTP files, process metrics streams, etc etc.  Check out the Trisul LUA API Docs for an overview. One of the script types is the inputfilter.  The inputfilter script allows you to drive the Trisul input from a LUA script. So we arrange to read from the namedpipe.

We use the incredibly cool LuaJIT FFI interface to process the binary records from rwcut, extract the required flow fields, and push it into Trisul. Once we extract the flow fields we load it into a LUA table and use  the helper library flowinput.lua to push the metrics into Trisul.

Here is the relevant FFI snippet from silk.lua where we use the C Struct defined in rwrec.h and then read it on the LuaJIT side.

 

Conclusion

If you are running SiLK and would like to try out a new way of analyzing data you already have in your dump files – try this script and see how Trisul can help you get a very different perspective.

 

Download Trisul Network Analytics for free today . Installing it is as easy as apt-get or yum.

Happy SiLK ing

Author: Vivek Rajagopalan

Vivek Rajagopalan is the a lead developer for Trisul Network Analytics. Prior products were Unsniff Network Analyzer and Unbrowse SNMP. Loves working with packets , very high speed networks, and helping track down the bad guys on the internet.