Unleash Networks Forum :: Topic: Trisul segmentation fault error (1/3)

Welcome, Guest

TOPIC: Trisul segmentation fault error

Trisul segmentation fault error 12 years 1 month ago #659

Chris

I'm experiencing a problem with the Trisul server process stopping randomly which appears to be as a result of a segmentation fault. Running trisul in nodaemon mode shows the following errors:

root@trisul:/# /usr/local/bin/trisul -nodemon /usr/local/etc/trisul/trisulConfig.xml -mode onlinerxring
Segmentation fault

or

root@trisul:/# /usr/local/bin/trisul -nodemon /usr/local/etc/trisul/trisulConfig.xml -mode onlinerxring
*** stack smashing detected ***: /usr/local/bin/trisul terminated
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6(__fortify_fail+0x50)[0xa1f390]
/lib/tls/i686/cmov/libc.so.6(+0xe233a)[0xa1f33a]
/usr/local/lib/trisul/plugins/libdnswatcher.so(+0x77a4)[0xe967a4]
/usr/local/lib/trisul/plugins/libdnswatcher.so(_ZN11CPIDNSWatch18SkipQuestionRecordEPhPm+0xfb)[0xe94a2b]
[0x2e313632]
======= Memory map: ========
00110000-00117000 r-xp 00000000 fb:00 655646     /lib/tls/i686/cmov/librt-2.11.1.so
00117000-00118000 r--p 00006000 fb:00 655646     /lib/tls/i686/cmov/librt-2.11.1.so
00118000-00119000 rw-p 00007000 fb:00 655646     /lib/tls/i686/cmov/librt-2.11.1.so
00119000-0011b000 r-xp 00000000 fb:00 655633     /lib/tls/i686/cmov/libdl-2.11.1.so
0011b000-0011c000 r--p 00001000 fb:00 655633     /lib/tls/i686/cmov/libdl-2.11.1.so
0011c000-0011d000 rw-p 00002000 fb:00 655633     /lib/tls/i686/cmov/libdl-2.11.1.so
0011d000-00130000 r-xp 00000000 fb:00 655606     /lib/libz.so.1.2.3.3
00130000-00131000 r--p 00012000 fb:00 655606     /lib/libz.so.1.2.3.3
00131000-00132000 rw-p 00013000 fb:00 655606     /lib/libz.so.1.2.3.3
00132000-0021b000 r-xp 00000000 fb:00 3278344    /usr/lib/libstdc++.so.6.0.13
0021b000-0021c000 ---p 000e9000 fb:00 3278344    /usr/lib/libstdc++.so.6.0.13
0021c000-00220000 r--p 000e9000 fb:00 3278344    /usr/lib/libstdc++.so.6.0.13
00220000-00221000 rw-p 000ed000 fb:00 3278344    /usr/lib/libstdc++.so.6.0.13
00221000-00228000 rw-p 00000000 00:00 0
00228000-00245000 r-xp 00000000 fb:00 655415     /lib/libgcc_s.so.1
00245000-00246000 r--p 0001c000 fb:00 655415     /lib/libgcc_s.so.1
00246000-00247000 rw-p 0001d000 fb:00 655415     /lib/libgcc_s.so.1
00247000-002e1000 r-xp 00000000 fb:00 3288288    /usr/local/lib/trisul_badfellas/plugins/libbadfellas.so
002e1000-002e2000 r--p 00099000 fb:00 3288288    /usr/local/lib/trisul_badfellas/plugins/libbadfellas.so
002e2000-002e3000 rw-p 0009a000 fb:00 3288288    /usr/local/lib/trisul_badfellas/plugins/libbadfellas.so
002e3000-002ef000 r-xp 00000000 fb:00 3288150    /usr/local/lib/trisul/plugins/libalertmeters.so
002ef000-002f0000 ---p 0000c000 fb:00 3288150    /usr/local/lib/trisul/plugins/libalertmeters.so
002f0000-002f1000 r--p 0000c000 fb:00 3288150    /usr/local/lib/trisul/plugins/libalertmeters.so
002f1000-002f2000 rw-p 0000d000 fb:00 3288150    /usr/local/lib/trisul/plugins/libalertmeters.so
002f2000-002fc000 r-xp 00000000 fb:00 3288145    /usr/local/lib/trisul/plugins/libsessmeter.so
002fc000-002fd000 ---p 0000a000 fb:00 3288145    /usr/local/lib/trisul/plugins/libsessmeter.so
002fd000-002fe000 r--p 0000a000 fb:00 3288145    /usr/local/lib/trisul/plugins/libsessmeter.so
002fe000-002ff000 rw-p 0000b000 fb:00 3288145    /usr/local/lib/trisul/plugins/libsessmeter.so
002ff000-00309000 r-xp 00000000 fb:00 655639     /lib/tls/i686/cmov/libnss_files-2.11.1.so
00309000-0030a000 r--p 00009000 fb:00 655639     /lib/tls/i686/cmov/libnss_files-2.11.1.so
0030a000-0030b000 rw-p 0000a000 fb:00 655639     /lib/tls/i686/cmov/libnss_files-2.11.1.so
0030b000-0032f000 r-xp 00000000 fb:00 655634     /lib/tls/i686/cmov/libm-2.11.1.so
0032f000-00330000 r--p 00023000 fb:00 655634     /lib/tls/i686/cmov/libm-2.11.1.so
00330000-00331000 rw-p 00024000 fb:00 655634     /lib/tls/i686/cmov/libm-2.11.1.so
00331000-00344000 r-xp 00000000 fb:00 655636     /lib/tls/i686/cmov/libnsl-2.11.1.so
00344000-00345000 r--p 00012000 fb:00 655636     /lib/tls/i686/cmov/libnsl-2.11.1.so
00345000-00346000 rw-p 00013000 fb:00 655636     /lib/tls/i686/cmov/libnsl-2.11.1.so
00346000-00348000 rw-p 00000000 00:00 0
0035d000-00495000 r-xp 00000000 fb:00 655513     /lib/i686/cmov/libcrypto.so.0.9.8
00495000-0049d000 r--p 00137000 fb:00 655513     /lib/i686/cmov/libcrypto.so.0.9.8
0049d000-004ab000 rw-p 0013f000 fb:00 655513     /lib/i686/cmov/libcrypto.so.0.9.8
004ab000-004af000 rw-p 00000000 00:00 0
004c0000-004d5000 r-xp 00000000 fb:00 655644     /lib/tls/i686/cmov/libpthread-2.11.1.so
004d5000-004d6000 r--p 00014000 fb:00 655644     /lib/tls/i686/cmov/libpthread-2.11.1.so
004d6000-004d7000 rw-p 00015000 fb:00 655644     /lib/tls/i686/cmov/libpthread-2.11.1.so
004d7000-004d9000 rw-p 00000000 00:00 0
004d9000-0051a000 r-xp 00000000 fb:00 3288146    /usr/local/lib/trisul/plugins/libcounters.so
0051a000-0051b000 ---p 00041000 fb:00 3288146    /usr/local/lib/trisul/plugins/libcounters.so
0051b000-0051e000 r--p 00041000 fb:00 3288146    /usr/local/lib/trisul/plugins/libcounters.so
0051e000-0051f000 rw-p 00044000 fb:00 3288146    /usr/local/lib/trisul/plugins/libcounters.so
00527000-0052d000 r-xp 00000000 fb:00 655637     /lib/tls/i686/cmov/libnss_compat-2.11.1.so
0052d000-0052e000 r--p 00006000 fb:00 655637     /lib/tls/i686/cmov/libnss_compat-2.11.1.so
0052e000-0052f000 rw-p 00007000 fb:00 655637     /lib/tls/i686/cmov/libnss_compat-2.11.1.so
0053a000-0057e000 r-xp 00000000 fb:00 655515     /lib/i686/cmov/libssl.so.0.9.8
0057e000-0057f000 r--p 00044000 fb:00 655515     /lib/i686/cmov/libssl.so.0.9.8
0057f000-00582000 rw-p 00045000 fb:00 655515     /lib/i686/cmov/libssl.so.0.9.8
00582000-005b1000 r-xp 00000000 fb:00 3288149    /usr/local/lib/trisul/plugins/libflowhandlers.so
005b1000-005b3000 r--p 0002e000 fb:00 3288149    /usr/local/lib/trisul/plugins/libflowhandlers.so
005b3000-005b4000 rw-p 00030000 fb:00 3288149    /usr/local/lib/trisul/plugins/libflowhandlers.so
00600000-00610000 r-xp 00000000 fb:00 655619     /lib/libbz2.so.1.0.4
00610000-00611000 r--p 0000f000 fb:00 655619     /lib/libbz2.so.1.0.4
00611000-00612000 rw-p 00010000 fb:00 655619     /lib/libbz2.so.1.0.4
00612000-006a6000 r-xp 00000000 fb:00 3288322    /usr/local/lib/trisul_urlfilter/plugins/liburlfilter.so
006a6000-006a7000 r--p 00093000 fb:00 3288322    /usr/local/lib/trisul_urlfilter/plugins/liburlfilter.so
006a7000-006a8000 rw-p 00094000 fb:00 3288322    /usr/local/lib/trisul_urlfilter/plugins/liburlfilter.so
0070a000-00714000 r-xp 00000000 fb:00 3288143    /usr/local/lib/trisul/plugins/libmonmeters.so
00714000-00715000 r--p 00009000 fb:00 3288143    /usr/local/lib/trisul/plugins/libmonmeters.so
00715000-00716000 rw-p 0000a000 fb:00 3288143    /usr/local/lib/trisul/plugins/libmonmeters.so
00774000-00798000 r-xp 00000000 fb:00 3288151    /usr/local/lib/trisul/libtbb.so.2
00798000-00799000 r--p 00024000 fb:00 3288151    /usr/local/lib/trisul/libtbb.so.2
00799000-0079a000 rw-p 00025000 fb:00 3288151    /usr/local/lib/trisul/libtbb.so.2
0079a000-0079b000 rw-p 00000000 00:00 0 Aborted

I've tried disabling the dnswatcher addon but the segfault error persists.

Trisul is monitoring an Internet traffic via a SPAN port with a 30Mb, IPv6 enabled, connection.

Any suggestions on what may be the cause or if there's further info I can provide, please let me know.

Ubuntu and Trisul details below:

Thanks

The administrator has disabled public write access.

Re:Trisul segmentation fault error 12 years 1 month ago #660

vivek [unleash]
OFFLINE
Karma: 3

Hi Chris,

How did you disable the dnswatcher plugin ? Can you try removing the libdnswatcher.so from /usr/local/lib/trisul/plugins directory ? You need to restart trisul after that.

Let me double check the dns code from my end. It has passed quite a bit of fuzz testing but clearly there is an issue somewhere.

Can you try moving the libdnswatcher.so out of the plugins dir and see if it holds up ? Meanwhile I will double check the code (or) send you a lib with symbols if I cant track it down.

Cheers,

Vivek R
Unleash Networks
Support : www.unleashnetworks.com/forums

The administrator has disabled public write access.

Re:Trisul segmentation fault error 12 years 1 month ago #661

Chris	Hi Vivek, thanks for your response. I started Trisul and then deactivated the add-on from within the web interface under Customise > Plugins. After moving libdnswatcher.so out of the plugins folder, segfaults are no longer occuring and Trisul has been been running all day so far Kind Regards Chris
	The administrator has disabled public write access.

Re:Trisul segmentation fault error 12 years 1 month ago #662

vivek [unleash] OFFLINE Karma: 3	Hi Chris, Thats great, but you lose a bit of functionality without the DNS watcher. We hardened code extensively in 2.4 (releasing in 10 days), would you like to try this version as early access ? Just login and scroll down to the bottom to get those packages. Cheers,
	Vivek R Unleash Networks Support : www.unleashnetworks.com/forums The administrator has disabled public write access.

Re:Trisul segmentation fault error 12 years 1 month ago #663

Chris

Hi Vivek,

I've updated to the newer release of Trisul thanks:

root@trisul:~# /usr/local/bin/trisul --version
2.4.1061

I am now runnnig with the DNS water plugin, and no longer receiving the libdnswatcher.so error, however, I am still getting the following error, and it's started occuring more frequently. I've not been able to run Trisul for more than 10 minutes with it occuring:

root@trisul:~# /usr/local/bin/trisul -nodemon /usr/local/etc/trisul/trisulConfig.xml -mode onlinerxring
Segmentation fault

I've had the ns-00X.log tailed during the process but it seems pretty random as to when it segfaults so I couldn't identify a pattern.

I am running Trisul on an older box, it's a P4 with 1GB of RAM. That's always sufficed for running Snort and Sguil sensors but I am wondering if it may be asking a bit much, and if the system's overloaded/can't keep up, could that be causing timing problems? I'm looking to find a modern box to run it on in case of it being a hardware issue.

The other thought is that our network does have active IPv6 traffic on it, which I've not bpf filtered out.

Any other thoughts or comments most welcome, and thanks for your help to-date.

Kind Regards

Chris

The administrator has disabled public write access.

Re:Trisul segmentation fault error 12 years 1 month ago #664

vivek [unleash]
OFFLINE
Karma: 3

Hi Chris,

Thanks for trying out 2.4

I say the prime suspect now is IPv6. We have tested IPv6 extensively with MOME traces - but they are either headers only or too vanilla. So that is far from ideal.

It is quite easy to do the BPF in trisul. Go to Customize > Capture Adapter > Select the active profile. Enter "not ip6" in the Filter box and Save.

You can then start Trisul from the command line or Admin > Start / Stop tasks.

Let me know how this fares.

As far as the box is concerned - what you have sounds quite sufficient since you are only pushing 30mbps.

Vivek R
Unleash Networks
Support : www.unleashnetworks.com/forums

The administrator has disabled public write access.

Forum

Unleash Networks Forums

Trisul Network Metering and Forensics

Trisul segmentation fault error