BLOG     |     FORUM
Welcome, Guest
Username: Password: Remember me
Forgot your password? Forgot your username? Create an account
Issues related to installation, running, bugs, and features.

TOPIC: Segfaults and disk hogging

Segfaults and disk hogging 12 years 1 month ago #3670

  • lex
  • lex's Avatar
  • OFFLINE
  • Karma: 0
Hello.
Trisul crashes for me several times per day, leaving huge (~3.6GB) core file and several unknown files with weird filenames (like "????#?") in /tmp.
It fills whole disk (~100GB virtual HDD) with that files until that VM runs out of space
lex@dpi >>> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 12.04.3 LTS
Release:        12.04
Codename:       precise
lex@dpi >>> dpkg --list | grep trisul
ii  trisul                           3.6.1595 
ii  trisul-badfellas                 3.6.570 
ii  trisul-geo                       3.6.564 
ii  trisul-urlfilter                 3.6.558 
ii  webtrisul                        3.6.1130
The administrator has disabled public write access.

Segfaults and disk hogging 12 years 1 month ago #3673

Hi,

Are there any errors in the log files ? The logs are at

/usr/local/var/log/trisul/ns*.log ?

Can you post some sample filenames in /tmp ? Not looking for the actual content but the filenames only.

Thanks,
Vivek R
Unleash Networks
Support : www.unleashnetworks.com/forums
The administrator has disabled public write access.

Segfaults and disk hogging 12 years 1 month ago #3674

  • lex
  • lex's Avatar
  • OFFLINE
  • Karma: 0
I wiped it, so ATM i can't do that. In dmesg i see:
[35636.423832] trisul[2170]: segfault at 114 ip 00007ff4dd9577e7 sp 00007ff4c0637c50 error 4 in libcrypto.so.1.0.0[7ff4dd8c5000+1b1000]
Will wait until next crash.
About logs:
Thu Oct  3 04:08:00 2013.261102 ERROR Non Constraint error unable to open database file
Thu Oct  3 04:08:00 2013.261576 ERROR Flusher TBB Thread  Error  std::exception
Thu Oct  3 04:08:00 2013.270192 ERROR Error in GlobalFlush guid= {00990011-44BD-4C55-891A-77823D59161B} name=Country
Thu Oct  3 04:08:43 2013.609224 ERROR SQL Error creating slice index entry for [RPNIDX] unable to open database file
Thu Oct  3 04:08:43 2013.609257 ERROR Error creating slices index for [RPNIDX] f=�,�� db = /usr/local/var/lib/trisul/CONTEXT0/caps/METASLICE.SQDB
Thu Oct  3 04:08:43 2013.638770 ERROR RING Archiver: ERROR Locking SLICEs, will try next interval  unable to open database file
Thu Oct  3 04:09:00 2013.071419 ERROR SQL Error creating slice index entry unable to open database file
Thu Oct  3 04:09:09 2013.585154 ERROR Error unable to create tmp ring complex file  Too many open files File : /usr/local/var/lib/trisul/CONTEXT0/caps/oper/RCF_triscap.4ENTIF
Thu Oct  3 04:09:09 2013.595353 ERROR SQL Error creating slice index entry for [RPNIDX] Unable to create tmp rcmplx file, see log
Thu Oct  3 04:09:09 2013.595382 ERROR Error creating slices index for [RPNIDX] f=H� db = /usr/local/var/lib/trisul/CONTEXT0/caps/METASLICE.SQDB
Thu Oct  3 04:09:09 2013.606950 ERROR RING Archiver: ERROR Locking SLICEs, will try next interval  unable to open database file
Thu Oct  3 04:09:36 2013.346620 ERROR Error unable to create tmp ring complex file  Too many open files File : /usr/local/var/lib/trisul/CONTEXT0/caps/oper/RCF_triscap.jKi3Td
Thu Oct  3 04:09:36 2013.346675 ERROR SQL Error creating slice index entry for [RPNIDX] Unable to create tmp rcmplx file, see log
Thu Oct  3 04:09:36 2013.346686 ERROR Error creating slices index for [RPNIDX] f=8-�� db = /usr/local/var/lib/trisul/CONTEXT0/caps/METASLICE.SQDB
Thu Oct  3 04:09:36 2013.359219 ERROR RING Archiver: ERROR Locking SLICEs, will try next interval  unable to open database file
Thu Oct  3 04:10:00 2013.172846 ERROR SQL Error creating slice index entry unable to open database file
Thu Oct  3 04:10:01 2013.943575 ERROR Error unable to create tmp ring complex file  Too many open files File : /usr/local/var/lib/trisul/CONTEXT0/caps/oper/RCF_triscap.WxTO4f
Thu Oct  3 04:10:01 2013.943619 ERROR SQL Error creating slice index entry for [RPNIDX] Unable to create tmp rcmplx file, see log
Thu Oct  3 04:10:01 2013.943628 ERROR Error creating slices index for [RPNIDX] f=�,�� db = /usr/local/var/lib/trisul/CONTEXT0/caps/METASLICE.SQDB
Thu Oct  3 04:10:01 2013.968388 ERROR RING Archiver: ERROR Locking SLICEs, will try next interval  unable to open database file
And so on. You can see whole grepped log here
You can also see that broken files filenames in f= param of error, it's unreadable. Each file was ~3-4GB in /tmp
I also have full log file, i can send it to you (does not want to post it public, also it almost 5MB in size)
The administrator has disabled public write access.

Segfaults and disk hogging 12 years 1 month ago #3675

Hi Lex,

Thanks for getting back. Can you gzip the logs and email us at info at unleashnetworks dot com -

UPDATE: Can you momentarily disable the packet storage and retry ?
Set the <Ring> <Enabled> to FALSE in /usr/local/etc/trisul/trisulConfig.xml

You can also wipe and restart , keeping your configuration intact, in the following manner.
cd /usr/local/share/trisul
./cleanenv -f -saveinit

Thanks,
Vivek R
Unleash Networks
Support : www.unleashnetworks.com/forums
Last Edit: 12 years 1 month ago by vivek [unleash]. Reason: added more info & deleted a wrong idea
The administrator has disabled public write access.

Segfaults and disk hogging 12 years 1 month ago #3677

  • lex
  • lex's Avatar
  • OFFLINE
  • Karma: 0
Log file sent and i disabled packet storage ATM.
The administrator has disabled public write access.

Segfaults and disk hogging 12 years 1 month ago #3678

Hi Lex,

Thanks for the logs, looks like we have two problems here.


1. The segfaulting : Does this happen even when there is enough disk space ? I will post another message to help us debug that.


2. Strange file names : This is due to running out of disk. Currently trisul isnt very graceful. We added some extra diagnostics & checks in our upcoming build tomorrow.

3. When you started Trisul how much disk free space did you have ?

Thanks, appreciate your reports.
Vivek R
Unleash Networks
Support : www.unleashnetworks.com/forums
The administrator has disabled public write access.
Moderators: vivek [unleash]
Time to create page: 0.047 seconds

Sitemap | Privacy

Copyright © 2011-14 Unleash Networks