December 30th, 2009 | Tags:

Logo

Lately, we are working with Netflow quite a bit for our upcoming release of Trisul Network Metering. Here is a tiny script we find invaluable while looking at network captures containing Netflow traffic.

Often while troubleshooting issues we need to look at the raw Netflow records.

For example : You may want to see all the Netflow records sent for IP = 10.22.1.29

Display filters wont get you far because you will still be left with individual packets. These packets themselves can contain dozens of records of which only one or two match.

We turned to Unsniff Scripting and wrote a simple Ruby script that allows you to query for netflow records matching any field value.  We use this script heavily for our internal testing and wish to share it on the blog.

# ------------------------------------------------------------------
# nfquery    Print matching netflow records
# ------------------------------------------------------------------
require 'win32ole'
 
USAGE =   "nfquery <capture-filename> <fieldname>=<value>"
EXAMPLE = "nfquery c:\temp\nfcap.usnf SrcAddress=209.209.1.200"
 
if ARGV.length != 2
 puts USAGE
 exit 1
end
 
# Set up input
InputFile = ARGV[0]
UnsniffDB = WIN32OLE.new("Unsniff.Database")
QueryFieldName = ARGV[1].split('=')[0]
QueryFieldValue = ARGV[1].split('=')[1]
 
# Open capfile and get packet index
 
UnsniffDB.OpenForRead(InputFile)
PacketIndex = UnsniffDB.PacketIndex
 
# Scan each packet looking for netflow protocol id
(0..PacketIndex.Count-1).each do |idx|
 pkt = PacketIndex.Item(idx)
 
 nf_layer = pkt.FindLayerByGUID("{DF2428E1-4843-48CF-B7DD-CCC9E5AE4BC1}")
 next if nf_layer.nil?
 
 pdu_count=nf_layer.FindField("Count").Value.to_i
 (0..pdu_count-1).each do |pduidx|
 nf_pdu = nf_layer.FindField(">NetflowRecord>pdu[#{pduidx}]")
 next if nf_pdu.nil?
 
 target_field=nf_pdu.FindField(QueryFieldName)
 if target_field.Value == QueryFieldValue
 duration_ms = nf_pdu.FindField("End Time").Value.to_i - nf_pdu.FindField("Start Time").Value.to_i
 s_ip = nf_pdu.FindField("SrcAddress").Value
 d_ip = nf_pdu.FindField("DstAddress").Value
 s_port = nf_pdu.FindField("SrcPort").Value
 d_port = nf_pdu.FindField("DstPort").Value
 octets = nf_pdu.FindField("Octets").Value
 proto = nf_pdu.FindField("Protocol").Value
 
 print pkt.ID.to_s.ljust(6) + "  "
 [s_ip,d_ip,s_port,d_port,octets,proto,duration_ms].each do |item|
 print item.to_s.ljust(16) + " "
 end
 print "\n"
 
 end
 end
 
end
 
UnsniffDB.Close()

To use this script.

1. Import the capture file in tcpdump format into Unsniff (or capture off a live interface)

2. Save the file as USNF format

3. Start querying using the script

More :

Unsniff Scripting Home

Download Unsniff

November 23rd, 2009 | Tags: , ,

snmp_mib_pack.JPGWe just updated the Juniper SNMP MIBS to the latest releases.  Sourced from Junipers public website.

The packages available are

  • JUNOS_10.0R1.8.zip (5MB)
  • ERX_Enterprise_MIBs_10.3.0.zip (7MB)
  • ScreenOS-6.3.zip (780KB)

Get them here

Get Unbrowse SNMP here

Install them via Repository > Import Package

November 22nd, 2009 | Tags:

We just enhanced the SSL/TLS capabilities of Unsniff Network Analyzer substantially in our newest release (1.8.0.1420)

  1. Support for TLS extensions – RFC 4366
  2. Support for TLS extensions – RFC 4492 (ec_point_formats and elliptic_curves)
  3. Validates if specified key file is in unencrypted PKCS#8 format
  4. Support for the latest TLS extension Renegotiation_Info with the tentative extension number of 0xFF01. This is the fix for the TLS MITM Renegotiation Flaw that has been making the rounds the past couple of weeks. See the Internet Draft at http://tools.ietf.org/html/draft-rescorla-tls-renegotiation-00

In this release :

Verify if the specified key file is unencrypted PKCS#8

This is the number one problem people face when using Unsniff for decrypting SSL/TLS. The private key needs to be in unencrypted PKCS#8 format. Prior versions of Unsniff happily allowed you to specify a key in any format, but would log an error message “Invalid Key Material …..” when the time comes to use it.

Any format other than unecrypted PKCS#8 will give this error

Any format other than unecrypted PKCS#8 will give this error

TLS Extensions

Here is a screenshot of Unsniff’s support for TLS extensions. Most but not all extensions are completely decoded (not just shown as TLV blobs Type-Length-Value).

TLS extensions completely decoded

TLS extensions completely decoded

But we already have Wireshark

We all use and love Wireshark. But if you work with SSL/TLS a LOT then you need to give Unsniff Network Analyzer a try. It could be useful to have it around in your toolbox along with Wireshark. Specifically, Unsniff could save you bunch of time because (1) it can produce bounce diagrams that you otherwise need to draw by hand (2) it can reassemble upper layer content like web pages (3) it tracks entire SSL records not just ethernet link layer packets (4) scriptable using Ruby (5) share decrypted packet captures without sharing the private keys.

Download Unsniff


The big security news this past week was the SSL Renegotiation Flaw found by researchers Marsh Ray and Steve Dispensa at Phone Factor. Here is what they say :

The SSL authentication gap allows an attacker to mount a man-in-the-middle attack, and affects the majority of SSL-protected servers on the Internet. Specifically, the vulnerability allows the attacker to inject malicious data and commands into the authenticated SSL communications path. This can often be done without either the client or server (e.g. web server and browser) being able to detect the attack.

Read more at their site

Listen to an interview featuring Marsh Ray

What is really cool with this vulnerability is that the researchers have released packet captures of it in action.

Grab the packet captures, a whitepaper, and protocol diagrams from Phone Factor’s site.

Analyzing the packet captures

In this post,let us see how we can analyze these and similar packet captures. There is plenty of heavy lifting to be done, so we will be breaking open a tool called Unsniff. Laura Chappell over at Chappell Seminars has a similar post on analyzing the traces using Wireshark.

Among all the captures released, client_init_renego_bis.pcap seems to demonstrate the attack the best. To make this exercise easy we first decrypt the capture file using Unsniff’s TLS decryption capabilities and save it in Unsniff’s native USNF format.

The USNF format

The USNF format is a great way to share decrypted SSL/TLS packet captures without supplying the private key. No, the private key is not secretly tucked away in the file. It is simply not required after the initial decryption.

Step 1 : Download the latest version of Unsniff

Step 2 : Download the capture file in USNF format from here

Step 3 : Open the file and play with the Packets, PDU, and Sessions tabs.

  • Packets : A list of link layer packets like you would see with Wireshark.
  • Sessions : A list of TCP streams.
  • PDUs : A list of reassembled and decrypted SSL records.

The Packets tab is not very interesting here because SSL records bear no relationship to link layer packet boundaries. So lets focus on the PDU and Sessions tabs.

Analyzing the streams

Switch to the sessions tab and note that the Sessions tab shows 6 TCP streams. But the original pcap file contained only 2 streams (from victim to MITM and from MITM to server). Unsniff creates two extra streams for every SSL/TLS stream. This is an great way to strip and analyze encrypted or tunneled streams.

The two extra streams are :

  • The decrypted SSL stream (with all the handshakes in it. Decrypted but with signature and block cipher padding intact)
  • The decrypted upper layer HTTP stream (with only the application data)

View HTTP data from the perspective of Server and MITM

Lets first look at the HTTP data from the HTTP server’s perspective.

Move to the Sessions tab and make sure the Stream sheet is activated. You may also have to rt-click and switch to UTF-8.

  • Click on Stream 5 ( HTTP data as seen by the server)
What the web server sees (behold the evil)

What the web server sees (behold the evil)

  • Click on Stream 6 (HTTP data sent by the victim)
What the victim sent to the webserver

What the victim sent to the webserver

Going deeper with the Bounce View

SSL Records Bounce (click for bigger)

SSL Records Bounce Diagram (Click to view generated PDF)

The real juicy part is ofcourse the protocol exchanges between the victim, the MITM, and the server. The ideal tool for this is called the “PDU Bouncer’. Activate it by Right-Clicking on a PDU and selecting Bounce View. This creates a bounce diagram containing all the PDUs that went back and forth on the same TCP stream in time order. For example : Right click on PDU #2 to bring up the diagram shown on the right.

Three-way Bounce

What is really cool about this bounce diagram is that you can select two PDUs from different streams and you will then get a three-way (or even 4 way diagrams).

This is exactly what we need, the victim, the MITM, and the server PDUs all laid out in a protocol diagram. Switch to the PDU view, right click and select PDUs 2 and 14. Now bring up the PDU bouncer.

Lets see what we get.

Stage 1

Shows the MITM (192.168.80.17) intercepting the Client Hello from the victim and then establishing a separate connection to the server with the intention of patching on the client later. The session ID can be seen to be 1C144B..

Showing MITM intercepting victim

Showing MITM intercepting victim

Stage 2

This shows the MITM sending the evil payload (PDU#28). Note the evil payload is an incomplete HTTP request sent with the hope that the victim will complete it. Once the payload is sent, you can see a new session being renegotiated. This new session is passed on to the client, from which point on the MITM stays away. The damage having been done ! Pure evil.

Step 2 : Renegotiating a new session and patching the client in

Step 2 : Renegotiating a new session and patching the client in

You can generate a PDF for any bounce diagram via Right Click + Print + Select CutePDF.

The PDF for this diagram is available here.

Tailpiece

The authors themselves have a pretty comprehensive whitepaper including a protocol diagram of their own in the package (http://www.phonefactor.com/sslgap/). The purpose of this post is to introduce a new tool you may wish to consider adding to your security toolbox in addition to the ubiquitous and lovable Wireshark.

Unsniff Network Analyzer 2.0 also features remote analysis in line with the NSM principles advocated in the Tao of Network Security Monitoring book. You can download the latest beta builds as they become available from http://www.unleashnetworks.com/downloads. Please support the product by testing out the beta builds.

November 14th, 2009 | Tags: , , ,

Who says people who look at raw packets cant have fun ? Let’s see how we can reconstruct YouTube videos from raw packets and play them back directly from Unsniff. We also show you a nifty Ruby script to automate this.

Playback

1. Fire up Unsniff Network Analyzer and start a packet capture

2. Go watch some YouTube video(s)  to completion. We will explain how to deal with half played videos later.

3. Once you are done,  switch over to the User Objects tab where all the action is. You will see a list of “user objects” extracted by Unsniff from the stream. Click on the Type column to sort by Type and locate the FLV objects. Right click to Save or Play (To playback you need the free VLC Player installed and FLV files associated with VLC).

Right click the user object of type FLV and select Play or Save

Right click the user object of type FLV and select Play or Save

Feeling adventurous ?  Lets try some more tricks.

Pruning the packet capture

When you interact with a site like YouTube there is a ton of content exchanged, not all of it is the video bits. You have packets containing images, HTML, Flash, and Javascript. The actual FLV is exchanged in a single TCP stream. If you can save these streams alone, you will be able to reconstruct the videos.

Here is how you do it.

1. Switch over to the “Sessions”, here is where you will see a list of TCP sessions updated in real time (every packet).

Only the TCP Stream containing the media is required ! Cut and paste it

Only the TCP Stream containing the media is required ! Cut and paste it

2. Locate the TCP Stream carrying FLV traffic. These are typically the really large ones. Select the stream, copy the stream (Edit->Copy) and Paste it as a new file (Edit -> Paste as New). Voila ! Switch to the User Objects in the new file and you will find the FLV without all the other clutter.

Script the whole thing using Ruby

The right clicking novelty quickly wears out after a while. You really want to automate this type of forensic stuff.

Here is a sample which shows you how you can use Ruby to script this stuff.

Task : Extract all the videos in a capture file as separate playable files. (including the ones that were aborted in the middle due to the use losing interest)

1. Save the capture as USNF format (this is the scriptable format used by Unsniff).

#
# extract FLVs from a capture file (USNF format) into a directory
#
require 'win32ole'
 
raise "Usage : xu2be <capture-file>" unless ARGV.length==1  
 
UnsniffDB = WIN32OLE.new("Unsniff.Database")
UnsniffDB.OpenForRead(ARGV[0])
UOIndex = UnsniffDB.UserObjectsIndex
 
(UOIndex.Count-1).times  do |idx|
 uo = UOIndex.Item(idx)
 if  uo.Type == 'FLV'
 uo.SaveToFile("Video_#{idx}.flv")
 print "Saved file Video_#{idx}.flv\n"
 end
end
 
UnsniffDB.Close

2. Run this script like so

C:\vboxshare\CAPS>ruby xu2be.rb incomplete-youtube.usnf
Saved file Video_91.flv
Saved file Video_192.flv
Saved file Video_193.flv

We are still working on tweaking the experience for our upcoming Unsniff 2.0 release. In the meantime, we invite you to  try out the Unsniff Beta 1.8 and give us feedback on what you’d like to really see.

November 14th, 2009 | Tags: , , , ,

A common problem while analyzing SNMP traffic is resolving OIDs to names.

We don’t want to see this :

Showing raw SNMP OIDs in the packet list

Showing raw SNMP OIDs in the packet list

We want to see this :

Showing human readable names in the packet list

Showing human readable names in the packet list

The venerable Wireshark’s own resolution capabilities work fine for many simple cases. With Wireshark, you can list the modules you need and have it load them upon startup. But what if you want to load thousands of MIBs ? What if you want to deal with badly written MIBs, MIBs with incorrect module names, MIBs with dependencies ?  We might be able to help you.

We make two products, Unbrowse SNMP and Unsniff Network Analyzer. Unbrowse SNMP is a full fledged SNMP tool that can compile almost anything you throw at it. It then persists the properties of each OID in a very efficient format on disk. The Unbrowse Scripting API provides a number of ways to get at this data. The other product, Unsniff is the actual SNMP packet analyzer. We have integrated both these products in such a way that Unsniff will use the OID information already available via Unbrowse.

To use this feature : (Requires latest versions of Unbrowse SNMP and Unsniff Network Analyzer)

  1. Download and Install Unbrowse SNMP
  2. Press Crtl+M and select all the MIB files you want to add
  3. Alternately, Download a precompiled package (we have one containing all the Cisco MIBs)
  4. Done

Unsniff will automatically detect if the Unbrowse SNMP name resolution facility is installed and will then proceed to resolve all OIDs to the maximum extent it can.

Resolving OIDs where ever they are found

Resolving OIDs where ever they are found

The advantages :

  • Leverage Unbrowse SNMP’s very flexible compiler
  • OIDs of thousands of modules are instantly available for resolution
  • Has no impact on Unsniff’s startup time
  • High speed resolution with low memory overhead
  • Scriptable via Ruby
April 3rd, 2009 | Tags:

A new version of the Cisco MIB Package is now available for download. This monster package contains all the latest MIBs published by Cisco on its web site at http://www.cisco.com/public/sw-center/netmgmt/cmtk/mibs.shtml as of April 2 2009.

Modules : 1137 Objects : 72,000+

Get it from here  (Click on the first item. Download size = 35.1 MB)

NOTE : The MIB Package takes less than 5 minutes to install on Windows XP and about 15 minutes on Windows Vista (dont ask why !) Users of Vista please be patient while the package is installed.

What is MIB Package ?

It is a ZIP file specially created for use with Unbrowse SNMP. You can install the package by selected Repository -> Import Package from the menu.

What is  Unbrowse SNMP ?

It is a easy to use SNMP utility which allows you to easily perform all SNMP operations in a rich graphical environment. Read here for features and screenshots. It is free for basic use but some premium feature require a license after 30 days.

April 3rd, 2009 | Tags:

We have a new release of Unbrowse SNMP available for immediate download.

This is a free upgrade for all current users of Unbrowse SNMP Power Features.

Get it from here

So whats new in this release !

  1. Contains a lot of tweaks and fixes
  2. Much improved support for IPv6
  3. Fixed bugs related to printing
  4. Fixed occasional problems with verifying authentication of traps

We would also like to welcome the following recent customers to the growing family of Unbrowse SNMP Power Features users.

  • SAIC Corporation
  • United States Coast Guard
  • Oracle Corporation
  • Fortress Technologies
  • Dell Computer
  • Accelenet
  • Network Integrity Systems
  • and other individual users

Thanks all for your support.

November 18th, 2008 | Tags:

In this post, I just want to share a great resource I found on the web.

If you are a techie entrepreneur like myself, you will sometimes find it hard to grasp critical business aspects such as pricing and sales strategy. Unfortunately, if you get some of these wrong – no amount of cool code is going to save your venture. So, how does a techie get the inside information about product marketing in plain English.

I stumbled upon a very good set of articles written by Daniel Shefer on exactly this topic.

Check it out at http://www.shefer.net/articles.html

September 18th, 2008 | Tags:

I just uploaded a new build of Unsniff 1.8 Beta that supports Google’s new protocol buffers scheme. Basically, you can stick your proto files in a particular folder and decode files and network streams on the fly.
Click here for step by steps on how to use this feature.

This is Beta software. Please report problems and suggestions – either as comments to this post or to the forum.

In the rest of the post, I will explain why we worked on this feature and how it works.
—–

If you have not yet heard, Protocol Buffers (protobuf) is a serialization mechanism for structured data.

From Googles Open Source Blog,

Protocol Buffers allow you to define simple data structures in a special definition language, then compile them to produce classes to represent those structures in the language of your choice. These classes come complete with heavily-optimized code to parse and serialize your message in an extremely compact format.

Blog post by Kenton Varda, Software Engineering Team at Google

You can visit the project page for more detail.

When the project was first announced in July 08, I was immediately attracted to it. It sounded like a perfect test case for Unsniff 2.0’s dynamic plugin framework.

A little background first, Unsniff Network Analyzer is a multi layer, scriptable, and content aware network analyzer. One of the cool things about Unsniff is its API. You can write a variety of plugins using the Unsniff API, but protocol plugins are the most common.

The types of protocol plugins you could write are.

  • A native plugin. A protocol plugin written as a C++ ATL COM Object using the framework provided. It is packaged as a DLL.
  • A dynamic plugin. Written using XML which describes the protocol in detail.
  • A mix. The XML handles the field dissection and the ATL handles other things like reassembly, custom descriptions, etc.

In Unsniff 2.0, we are introducing a new concept called “Custom Dynamic Plugin”. Instead of XML, the user can create plugins in any “IDL like” language they could parse. The API provides hooks so and they can be integrated into the Unsniff framework. This approach has great advantages because frequently a user has hundreds of in-house protocol messages in a custom format. They cannot be expected to write “XML documents” and certainly not “C functions”.
So, we decided to try supporting Protocol Buffers in the Beta (Unsniff 1.8) as a way to test out the concept. The way it works is.

1. You stick all your proto files in a special folder

2. You write a small XML stub describing each protocol and how they integrate into the Unsniff framework (eg, which ports they operate on, the name of the protocol, the ID etc)

Thats it !

When required, Unsniff will compile each proto on the fly and create a dynamic custom decoder. This supports decoding network packets as well as files containing protobuf encoded data.

You get all of Unsniff’s larger network features for free. This includes handling many link layer protocols, TCP segmentation, IP defragmentation, TLS decryption for debugging, etc. Each message is shown as a separate PDU in the PDU sheet. These messages could span multiple packets or several could be contained in a single link layer packet.

You can download the latest builds from the Beta Page

Enjoy !

—-

Postscript

I wrote a custom parser and lexer for proto files. It handles pretty much everything including groups, extensions, import files, package names, etc. I could have just used the library’s methods for compiling it, but I was already too far down the road of YACC. I also wanted to extract the comments in the proto file, which the grammar does.

If anyone is interested I can post the YACC and LEX files as public domain. I will post this offer in the discussion group.