How to detect SHA1 signed certificates on your network

A few colleagues at work today upgraded their Google Chrome browser to Chrome 57 and some of them were surprised they could no longer open many websites including gmail.

Blocked !

If you notice the error message it says NET:ERR_CERT_WEAK_SIGNATURE_ALGORITHM.  This is a clear indicator that we are dealing the use of SHA1 algorithm used to sign certificates.  If you click on the text that says WEAK_SIGNATURE you will get a dump of the certificate chain. You can then copy paste that into a file and then run openssl x509 -in err.pem -inform pem -text to print out the cert. When we did this we found that in each failure case atleast one of the certificates in the chain has the following line  Signature Algorithm: sha1WithRSAEncryption

Google has been trying very hard to phase out use of SHA1 and move to SHA2 (aka SHA256) for a long time now. With the latest release of Google Chrome (Rel 57) they seem to have really tightened the noose. It appears with Chrome 57 even intermediate CA certs that chain directly to a local “trust anchor” will not be honored. See the following explanation from Google Security.

Starting with Chrome 54 we provide the EnableSha1ForLocalAnchorspolicy that allows certificates which chain to a locally installed trust anchor to be used after support has otherwise been removed from Chrome. Features which require a secure origin, such as geolocation, will continue to work, however pages will be displayed as “neutral, lacking security”. Without this policy set, SHA-1 certificates that chain to locally installed roots will not be trusted starting with Chrome 57, which will be released to the stable channel in March 2017.

Google Security Blog

 

Since we have Trisul running 24×7 in our office network,  I decided to look into the issue from a network viewpoint.  Trisul saves all SSL Certs as Full Text Documents (FTS) in the same format as the command “openssl x509..” . This makes it easy to search old traffic or more importantly to script detection and analysis as we will see shortly.

From the SSL FTS we found that for the failing Chrome 57 browsers the cert chain looked like this.

The fail cases have 3 certs

The first line is the Subject name, the second line is the Issuer, and the last line is the Signature Algorithm.

and for browsers  that worked fine we found 2 in chain.

 

In the fail cases, GeoTrust Global CA is an intermediate CA and Equifax is the root CA. Equifax signed GeoTrust using SHA1 and therefore Chrome rejects it. In the success cases, GeoTrust Global CA happens to be the root CA and it uses SHA2.

The SHA1 rules for Chrome are :

  • SHA1 can be used by the root CA for self signing ; the signatures of the Root CA dont matter because the browsers trust them by the public key.
  • When any signature in the entire certificate chain except the Root is signed by SHA1, it appears Chrome 57 and above are going to block it.

While I am yet to get my Chrome 57 to open any Google site, I tried adding the Chrome profile “EnableSha1ForLocalAnchorsPolicy” as mentioned in the Google Security Blog without luck.  I have a support ticket open so will update this post when I figure out how to solve this.

UPDATE: SOLVED – see below

Alert when you see usage of SHA1 signing on your network

After this I decided to write a little script to alert whenever a SHA1 signed certificate was seen in our network traffic. Using the Trisul LUA API this kind of detection is super easy.  The script is available on Github at detect_sha1.lua

Once we got the script installed on our office perimeter Trisul has been generating alerts whenever SHA1 is detected. Surprisingly most websites from Facebook,Twitter,Github,many Banking sites seem to have moved to SHA2 so the alert volume is low.

The following is a screenshot you can see the highlighted sha1WithRSAEncryption  signing algorithm in the alert

We developed Trisul Network Analytics 6.0 to be a platform where you can build various applications on top of it using a general purpose language like LUA and loosely coupled from the actual protocol fields. For scripts like these the flow is :

  1. you decide what you want to work off – in this case SSL Certificates
  2. you plug in to that stream
  3. use standard text manipulations to get the data because Trisul documents are in standardized canonical format
  4. feed something back into the Trisul Analytics stream as a result of your script

In this script the real action takes place in only one line

 

What to do with this alert ?

One of the concerns of this  deep Network Security Monitoring (NSM)  action that generates an alert is — now what? out analysts now have extra alerts to take care of !!

After turning on SHA1 detection we have about 190+ alerts of this kind. These alerts are tagged Medium/Low priority and gives us an idea of exactly how much of SHA1 is going on.  It could be an annoyance if  each alert to be handled in some way.  In those cases, you can modify the  same Trisul LUA Script a bit  to turn the alert into a metric.  When you turn alerts into a metric you will be able to answer :

  • How many SHA1 signed certs did I see over time
  • You will not be able to see the individual alerts

Hope these techniques are useful to security pros particularly those obsessed with visibility and metrics.

Get Trisul , the LUA API Documentation , and the Github samples at trisulnsm  Trisul is totally free for a rolling 3-day window and works on Ubuntu 14.04, 16.04, RHEL/CentOS 7.x

 

— UPDATE 26-Mar-2017 —

SOLVED : 

I was able to solve the issue and get Chrome 57 working again by “Untrusting Equifax Secure CA”. See my answer at Chrome Help Forum

Building ZeroMQ with libsodium : No package libsodium found

Are you trying to build ZeroMQ with libsodium ? You may run into the following issue while trying to configure ZeroMQ where it doesnt detect that you have already installed libsodium.

The real issue is that the default libsodium install from source installs the package in /usr/local/ and ZeroMQ doesnt pick itup.

There are a lot of answers on the internet about using --with-libsodium=/usr/local. This will not work because the ZeroMQ configure script uses pkg-config (a tool to get information about installed packages to detect libs). That tool depends on finding a file called libsodium.pc ( <package-name>.pc) in a number of directories.

Do this to fix the issue.

Check if pkg-config is indeed picking libsodium

It is not picking it up. Try adding the /usr/local/lib/libsodium

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

Now try finding libsodium

Now ZeroMQ should find it and you are on your way to CURVE heaven.

Hope this helps a few people. We spend quite some time trying to hack the ZeroMQ autoconf process !!

New self monitoring performance stats in Trisul Network Analytics

The latest release of Trisul Network Analytics [ release details ]  has a really nifty self-monitoring feature. There is no need for outside tools like atop or nmon to check on Trisul’s performance any more.

To use the feature :

  • Login as admin
  • Select Tools > Perf Stats

The Perf stats dashboard is shown similar to the one below.

Self Performance Stats Module in Trisul
Self Performance Stats Module in Trisul

The modules shown in the Perf Stats dashboard are

Global Flush Time

Trisul is a streaming network analytics engine. Since there is a continuous stream of high volume data, one key requirement is that aggregated results are flushed out within a small amount of time.  Global Flush-0, 1 are the two default backend threads of Trisul flushing out in parallel. You can have upto 8 backend threads in that case all 8 will be shown on the graph.

Key thing to look for : See if there is spikiness in the chart, all data should ideally be less than 20 seconds.

CPU Usage

Shows Total and Trisul CPU Usage over time.

Lookout for : CPU being pegged. Ideally Trisul CPU usage must rise and fall with load.  Unusual spikes in low traffic times have led us to find scans and DoS attacks as the Trisul TCP reassembly engine is stressed during these times.

 

Memory Usage

Total and Trisul Memory Usage

Lookout for : Memory climbing , this is very very rare and could indicate a memory leak.   Trisul has a lot of hi-water and lo-water marks to restrict memory usage. If your server has a lot of memory you could consider increasing these numbers so Trisul can make more judicious use of RAM.

 

Disk Usage

Total and Trisul Usage.  Shows the amount of disk used by the Trisul backend database.

Lookout for : If data climbs steadily and threatens to it the maximum, you may need to adjust the SlicePolicy to reduce the number of slices kept in the system. This way the DB size is kept smaller than the available disk space.

 

With this new feature, your Trisul monitors itself.

 

Have you tried Trisul Network Analytics yet ? It is a free download and can immediately start providing value to you !