|
|
|
Analyze Google Protocol Buffer streams
Protocol Buffers (protobuf)
is a serialization scheme recently open sourced by Google.
This article is a simple tutorial on how to analyze protobuf
messages using Unsniff 1.8 Beta. You just have to drop the proto files in a
specific folder. Unsniff then creates a decoder on the fly using the
information in the file.
We built support for protocol buffers as a project to test the dynamic
decoder framework for Unsniff 2.0. It is now in a shape that many might
be of practical use to a lot of people.
What are Protocol Buffers ?
Protocol Buffers (protobuf) are a way of serializing structured data
for storage or transmission. The basic idea is that you write a
so-called 'proto' file, then run the 'proto compiler' to produce code
in a variety of languages to read and write the messages you defined in
the proto file. Google uses protocol buffers for most of its internal
file formats and network protocols. It was open sourced under
the Apache License in July 08.
For more details, visit the project site at http://code.google.com/p/protobuf/
Feature Overview
- No coding or compiling needed
- Parses proto files on the fly as and when required
- Can read binary files as well as network captures
- Handles IP defragmentation, TCP segmentation, tunneling,
TLS decryption
- Presents entire messages as complete PDUs instead of
ethernet packets
- Handles length prefixed messages
- Points out missing fields and groups
- Supports all protobuf standard types including groups
The rest of this article describes how to use the tool.
Step by step guide
We figured it would
be easier to just show you how to use it by example.
Step 1 : Required downloads
You need the following :
Step 2 : Install Unsniff 1.8 Beta
If you are installing on Vista, you need to allow UAC to
proceed with the install.
Note : You
may also want to install Winpcap if you wish to capture packets
from the network.
Run Unsniff by double clicking on the desktop icon. The first run creates the required folder structure in the current users data folders. You will put your proto files there in Step 4.
Step 3 : Unzip the test data
Unzip the unsniff_protobuf_test.zip file from Step 1 to a temporary
folder.
You show see two sub folders.
- proto_binary_data
: Containing protobuf encoded files and a tcpdump capture
- proto_plugins
: Containing 4 sample protos (addressbook, northwind, unittest,
unittest_extensions)
Step 4 : Install the proto files
Unsniff will automatically detect protocols when they are
placed in
the user's Application Data folder. This folder is typically at
C:\Users\<Username>\AppData\Roaming, it is represented by
the variable %APPDATA%. We need to use the Unsniff\XMLPlugs subfolder.
- For Vista
: Press the Start Button and type %APPDATA% in
the search box. Locate the Unleash
Networks > Unsniff > XMLPlugs directory. If
the XMLPlugs directory does not exist create it.
- For XP :
Press Start > Run and type %APPDATA%.
Locate the Unleash
Networks > Unsniff > XMLPlugs directory. If
the XMLPlugs directory does not exist create it.
- Copy all the files in the proto_plugins folder into the
XMLPlugs directory.
Thats it ! You are ready to test.
Step 5 : Start analyzing
This section walks you through analyzing a binary file and then a
network capture over TCP.
Analyze from a protobuf
binary file
This
section shows you how to analyze a protobuf encoded binary
file.
- Start Unsniff Network Analyzer
- Click on the "G" icon on the toolbar or select Tools
> Google Protocol Buffers > Import File
 - Select the addressbook_in_a_file.bin file from the test
data folder (from Step 3)
- You will be shown a list of protocols, select the
PB_ADDRESSBOOK
- Thats it ! You will be shown a list of Person
records.
- You may view the tree on the left side and raw data on the
right. (Click on Raw to see bytes instead of the picture)
- The screenshot
below shows how the nwind.proto.bin is displayed. Each "Order" message in
the nwind.proto is
shown separately as a PDU.
Sample data and their corresponding proto files are described in the
table below.
| Sample Data File |
Proto File |
Source |
| addressbook_in_a_file.bin |
addressbook.proto |
Generated |
| addressbook_over_tcp_stream |
addressbook.proto (attached to port 20000) |
Generated |
| golden_message |
unittest.proto (TestAllTypes message) |
From the Google Protocol Buffers distribution package |
| unittest_alltypes.dat |
unittest.proto (TestAllTypes message) |
Generated |
| unittest_some_extensions |
unittest.proto (TestAllExtensions message) |
Generated |
| nwind.proto.bin |
nwind.proto |
From the protobuf-net project page. See here |
Analyze from a
network capture
You can capture live packets or import files in tcpdump (libpcap)
format. Note that you need to install Winpcap for this
purpose.
We have a tcpdump file for you to try called addressbook_over_tcp_stream.tcpd
- Start Unsniff Network Analyzer
- Select "File
> Import > From Tcpdump"
- Select the addressbook_over_tcp_stream.tcpd file
- Observe the results
Unsniff is a multi-layer packet analyzer. Observe the sheets tab at the
bottom.
- The "Packets" sheet will show you the individual ethernet
packets. You can see the protobuf message in packet #4 for example.
- The "PDU" sheet will show you individual PDUs. Each of
these
PDUs can span multiple packets or many could fit in a single packet.
- The "Sessions" sheet will show you each TCP Session. You
can
expand each session or Right-Click on a session for more options.
Adding your own proto files
We said that you can put any proto file in the %APPDATA%\Unleash
Networks\Unsniff\XMLPlugs folder and it automagically
turns into a dynamic plugin. Well, almost ! There is an extra thing you
have to do to hook up a proto to the Unsniff dynamic plugin framework.
You need to write a tiny XML describing the protocol. You can of course
simply copy an existing XML file and change some fields.
Lets take an example. When we added the addressbook.proto file to the
%APPDATA% folder, we also added an addressbook.proto.xml file which
looks like this.
<?xml version="1.0" encoding="utf-8"?>
<USNFProtocol id="PB_ADDRESSBOOK"
shortname="PBADDR" name="PBUF (addressbook.proto)"
protid="{A11F24EC-599C-486f-9E7E-56F5026F3A7A}">
<vendor>Google Protocol Buffer
Sample</vendor>
<version>1.0</version>
<color>#FF0000</color>
<rootfield>AddressBook</rootfield>
<SurrogateInfo>
<CLSID>{FAB19DF6-581E-4F46-89B4-4574A3F2DCD2}</CLSID>
<Description>Google
Protocol Buffer Surrogate Object (do not change
CLSID)</Description>
<CustomURI>addressbook.proto</CustomURI>
</SurrogateInfo>
<AccessPoints>
<AccessPoint
hostid="{14D7AB53-CC51-47e9-8814-9C06AAE60189}" apvalue="20000"
hostdesc="UDP Protocol"/>
<AccessPoint
hostid="{77E462AB-2E42-42ec-9A58-C1A6821D6B31}" apvalue="20000"
hostdesc="TCP Protocol"/>
</AccessPoints>
</USNFProtocol>
The key fields are shown in bold. The purpose of these fields is
explained below.
- The id, shortname, name can be any string to help you
identify the protocol
- The protid is a GUID. You can create this using Visual
Studio, or just type in some random hex digits to uniquely identify
your protocol.
- The rootfield is the name of the container message in your
proto file. If you have multiple container files create a separate XML
for each.
- The CustomURI is the name of your proto file.
Connecting your protobuf
message to TCP port number(s)
You
can either specify the connections in the XML file in the AccessPoint
section (see above)
or you can manually associate them via the "Manage
> Access Points" menu. In the above example, the
GUIDs represent
the
protocol ID of TCP and UDP protocols. If you wish to add more than one
port, simply add an extra entry. To find out the protocol ID of any
protocol use the "Plugins
> Manage Protocols" menu. Then click on
"Details"
to find out the protocol ID.
Further
Click on "Plugins
> Configure" to access other options for this
feature.
Conclusion
What
started off as an exercise to develop our API to support custom dynamic
dissectors evolved to a good support for protocol buffers. This is part
of Unsniff 1.8 Beta, prior to Unsniff 2.0. We have not finalized our
pricing model for Unsniff 2.0. We may make it free to process upto a
thousand packets or stay with the current model. We can
provide a
free license to involved beta testers for this feature.
We can enhance this feature based on feedback.
Email us at info dottt unleashnetworks
|
|
