Debugger missing at IMA :(

[auser]

I'm trying to debug a failure that occurs rare after days or weeks on a PB-program running on debian wheezy 64bit with lots of logging. I thought my last logging gave me a clue but too less to find the bug. I enabled the debugger and purifier and recreated the binary (so my binary was almost 2x in size) however after the next crash that followed in about a week I got nothing from PB-debugger and my log (with increased loglevel) refused my previous idea (it crashed somewhere fully different). All I got was some "double free or corruption (fasttop)" from glibc (I did not get this first time without debugger) but nothing from PBs debugger itself.

This is a pretty annoying situation now. Usually (using C/++) I would run the binary through gdb and make some backtrace now. However this is not really helpful with that PB-bin. So it seems I'm delivered to the PB debugger which is quiet for unknown reason. I've no idea why because if I create some quick faulty test-binary that allocates once and frees twice I get some info from the debugger (and yet again some glibc-message that is a bit different). However the real code is not that small that it's easy to find (I've counted about 150 FreeMemory()) and it seems it occurs just after days, sometime weeks of running and lots of users connecting and disconnecting in meanwhile without issues - so a debugger that is reliable would be pretty important. Any ideas?

Greetings,
auser

[Fred]

If you compiler with "-ds" flag, it will keep the debug symbols and you can use gdb to backtrace the error. It helped here sometimes.

[auser]

I would give it a try with "-ds" and gdb... thanks.

[auser]

Looks like it does not work (or I made something wrong?):


Faulty PB test-code:

Code: Select all

Procedure ima()
  *mem = AllocateMemory(12000)
  FreeMemory(*mem)
  FreeMemory(*mem)
EndProcedure

ima()

Compiled with: "/usr/local/purebasic/compilers/pbcompiler -l -t -ds -d -pf test19.pb -e test19.pb.bin"
Running with: "gdb test19.pb.bin":

GNU gdb (GDB) 7.0.1-debian
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /tmp/test19.pb.bin...(no debugging symbols found)...done.
(gdb) run
Starting program: /tmp/test19.pb.bin
[Thread debugging using libthread_db enabled]
*** glibc detected *** /tmp/test19.pb.bin: double free or corruption (!prev): 0x00000000006672c0 ***
======= Backtrace: =========
/lib/libc.so.6(+0x71e16)[0x7ffff78d2e16]
/lib/libc.so.6(cfree+0x6c)[0x7ffff78d7b8c]
/tmp/test19.pb.bin[0x4320e2]
/tmp/test19.pb.bin[0x42d2b2]
/tmp/test19.pb.bin[0x406443]
======= Memory map: ========
00400000-00446000 r-xp 00000000 08:01 1570101 /tmp/test19.pb.bin
00645000-00649000 rwxp 00045000 08:01 1570101 /tmp/test19.pb.bin
00649000-00688000 rwxp 00000000 00:00 0 [heap]
7ffff0000000-7ffff0021000 rwxp 00000000 00:00 0
7ffff0021000-7ffff4000000 ---p 00000000 00:00 0
7ffff4f6f000-7ffff4f85000 r-xp 00000000 08:01 1046531 /lib/libgcc_s.so.1
7ffff4f85000-7ffff5184000 ---p 00016000 08:01 1046531 /lib/libgcc_s.so.1
7ffff5184000-7ffff5185000 rwxp 00015000 08:01 1046531 /lib/libgcc_s.so.1
7ffff5185000-7ffff73db000 rwxp 00000000 00:00 0
7ffff73db000-7ffff73dd000 r-xp 00000000 08:01 1065866 /lib/libdl-2.11.3.so
7ffff73dd000-7ffff75dd000 ---p 00002000 08:01 1065866 /lib/libdl-2.11.3.so
7ffff75dd000-7ffff75de000 r-xp 00002000 08:01 1065866 /lib/libdl-2.11.3.so
7ffff75de000-7ffff75df000 rwxp 00003000 08:01 1065866 /lib/libdl-2.11.3.so
7ffff75df000-7ffff765f000 r-xp 00000000 08:01 1065867 /lib/libm-2.11.3.so
7ffff765f000-7ffff785f000 ---p 00080000 08:01 1065867 /lib/libm-2.11.3.so
7ffff785f000-7ffff7860000 r-xp 00080000 08:01 1065867 /lib/libm-2.11.3.so
7ffff7860000-7ffff7861000 rwxp 00081000 08:01 1065867 /lib/libm-2.11.3.so
7ffff7861000-7ffff79ba000 r-xp 00000000 08:01 1065854 /lib/libc-2.11.3.so
7ffff79ba000-7ffff7bb9000 ---p 00159000 08:01 1065854 /lib/libc-2.11.3.so
7ffff7bb9000-7ffff7bbd000 r-xp 00158000 08:01 1065854 /lib/libc-2.11.3.so
7ffff7bbd000-7ffff7bbe000 rwxp 0015c000 08:01 1065854 /lib/libc-2.11.3.so
7ffff7bbe000-7ffff7bc3000 rwxp 00000000 00:00 0
7ffff7bc3000-7ffff7bda000 r-xp 00000000 08:01 1065849 /lib/libpthread-2.11.3.so
7ffff7bda000-7ffff7dd9000 ---p 00017000 08:01 1065849 /lib/libpthread-2.11.3.so
7ffff7dd9000-7ffff7dda000 r-xp 00016000 08:01 1065849 /lib/libpthread-2.11.3.so
7ffff7dda000-7ffff7ddb000 rwxp 00017000 08:01 1065849 /lib/libpthread-2.11.3.so
7ffff7ddb000-7ffff7ddf000 rwxp 00000000 00:00 0
7ffff7ddf000-7ffff7dfd000 r-xp 00000000 08:01 1065850 /lib/ld-2.11.3.so
7ffff7f6b000-7ffff7fed000 rwxp 00000000 00:00 0
7ffff7ff9000-7ffff7ffb000 rwxp 00000000 00:00 0
7ffff7ffb000-7ffff7ffc000 r-xp 00000000 00:00 0 [vdso]
7ffff7ffc000-7ffff7ffd000 r-xp 0001d000 08:01 1065850 /lib/ld-2.11.3.so
7ffff7ffd000-7ffff7ffe000 rwxp 0001e000 08:01 1065850 /lib/ld-2.11.3.so
7ffff7ffe000-7ffff7fff000 rwxp 00000000 00:00 0
7ffffffea000-7ffffffff000 rwxp 00000000 00:00 0 [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

Program received signal SIGABRT, Aborted.
0x00007ffff78931b5 in raise () from /lib/libc.so.6
(gdb) backtrace
#0 0x00007ffff78931b5 in raise () from /lib/libc.so.6
#1 0x00007ffff7895fc0 in abort () from /lib/libc.so.6
#2 0x00007ffff78c95bb in ?? () from /lib/libc.so.6
#3 0x00007ffff78d2e16 in ?? () from /lib/libc.so.6
#4 0x00007ffff78d7b8c in free () from /lib/libc.so.6
#5 0x00000000004320e2 in SYS_FreeMemoryWithSize ()
#6 0x000000000042d2b2 in PB_FreeMemory_PURIFIER ()
#7 0x0000000000406443 in PB_DEBUGGER_ChangeString ()
#8 0x00007ffff78df8f6 in memset () from /lib/libc.so.6
#9 0x0000000000405000 in ?? ()
#10 0x00007fffffffec00 in ?? ()
#11 0x0000000000000000 in ?? ()
(gdb)

I've no clue what's going on... all I know is some "free()" caused a crash somewhere unknown... which might or might not be executed by some other function somewhere previously.




Same with C:

Faulty C test-code:

Code: Select all

#include <stdlib.h>

void ima()
{
  void * mem = calloc(12000,1);

  free(mem);
  free(mem); // line 8 is bad
}


int main()
{
  ima(); // line 14
}

Compiled with: "gcc -g test19.c -o test19.c.bin"
Running with: "gdb test19.c.bin"

GNU gdb (GDB) 7.0.1-debian
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /tmp/test19.c.bin...done.
(gdb) run
Starting program: /tmp/test19.c.bin
*** glibc detected *** /tmp/test19.c.bin: double free or corruption (top): 0x0000000000601010 ***
======= Backtrace: =========
/lib/libc.so.6(+0x71e16)[0x7ffff7aeee16]
/lib/libc.so.6(cfree+0x6c)[0x7ffff7af3b8c]
/tmp/test19.c.bin[0x400567]
/tmp/test19.c.bin[0x400577]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7ffff7a9bc8d]
/tmp/test19.c.bin[0x400479]
======= Memory map: ========
00400000-00401000 r-xp 00000000 08:01 1569941 /tmp/test19.c.bin
00600000-00601000 rw-p 00000000 08:01 1569941 /tmp/test19.c.bin
00601000-00622000 rw-p 00000000 00:00 0 [heap]
7ffff0000000-7ffff0021000 rw-p 00000000 00:00 0
7ffff0021000-7ffff4000000 ---p 00000000 00:00 0
7ffff7867000-7ffff787d000 r-xp 00000000 08:01 1046531 /lib/libgcc_s.so.1
7ffff787d000-7ffff7a7c000 ---p 00016000 08:01 1046531 /lib/libgcc_s.so.1
7ffff7a7c000-7ffff7a7d000 rw-p 00015000 08:01 1046531 /lib/libgcc_s.so.1
7ffff7a7d000-7ffff7bd6000 r-xp 00000000 08:01 1065854 /lib/libc-2.11.3.so
7ffff7bd6000-7ffff7dd5000 ---p 00159000 08:01 1065854 /lib/libc-2.11.3.so
7ffff7dd5000-7ffff7dd9000 r--p 00158000 08:01 1065854 /lib/libc-2.11.3.so
7ffff7dd9000-7ffff7dda000 rw-p 0015c000 08:01 1065854 /lib/libc-2.11.3.so
7ffff7dda000-7ffff7ddf000 rw-p 00000000 00:00 0
7ffff7ddf000-7ffff7dfd000 r-xp 00000000 08:01 1065850 /lib/ld-2.11.3.so
7ffff7fea000-7ffff7fed000 rw-p 00000000 00:00 0
7ffff7ff9000-7ffff7ffb000 rw-p 00000000 00:00 0
7ffff7ffb000-7ffff7ffc000 r-xp 00000000 00:00 0 [vdso]
7ffff7ffc000-7ffff7ffd000 r--p 0001d000 08:01 1065850 /lib/ld-2.11.3.so
7ffff7ffd000-7ffff7ffe000 rw-p 0001e000 08:01 1065850 /lib/ld-2.11.3.so
7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0
7ffffffea000-7ffffffff000 rw-p 00000000 00:00 0 [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

Program received signal SIGABRT, Aborted.
0x00007ffff7aaf1b5 in raise () from /lib/libc.so.6
(gdb) backtrace
#0 0x00007ffff7aaf1b5 in raise () from /lib/libc.so.6
#1 0x00007ffff7ab1fc0 in abort () from /lib/libc.so.6
#2 0x00007ffff7ae55bb in ?? () from /lib/libc.so.6
#3 0x00007ffff7aeee16 in ?? () from /lib/libc.so.6
#4 0x00007ffff7af3b8c in free () from /lib/libc.so.6
#5 0x0000000000400567 in ima () at test19.c:8
#6 0x0000000000400577 in main () at test19.c:14
(gdb)

Success via gdb with C code: I know function "ima()" caused a crash via "free()" in line 8 which was executed previously by function main() in line 14.

[Fred]

The symbols are only PB related one, not the user procedures, so it won't really help in your case. Did you try onerror ?

[auser]

Did you try onerror ?

Thanks for the hint. Edit: I've tried with OnError and put 2 FreeMemory in a raw now. It's telling some wrong line without the debugger. Anyway compiling with "-l -t -pf -ds -d" reported the wrong faulty line on OnError... at least most time (it even yet again reported some fully different line nowhere near the real issue like without the debugger on). I'm not fully happy with that because I don't get a real backtrace so the last line is important and the real binary is running for about 14 days without any issue so far (so if it's crashing in a few weeks and it's not telling the truth afterwards it's a serious issue). Anyway I would give it a try with that.

Greetings,
auser

[auser]

I had a further "crash" of my application. Well at least it _should_ crash because I've got a segfault reported from the Error-Handler:

Error-message: Segmentation violation
Error-code: 11
Code-address: 139690827377649
Target-address: 0
Source-line: 1044
Source-file: srv_serv85.pb

However after printing me that lines the ErrorHandler did not End the process (there is an "End" 2 lines after the PrintN of the ErrorHandler function so it should finally End there). Any idea why? There was still lots of clients that tried to connect and got rejected (for minutes until I did some kill -9 to finally stop it). I've tried to crash it manually (with some easy to find double FreeMemory())... and it seems after the ErrorHandler gets called (OnErrorCall) the program is hanging in some more or less busy state. CTRL-C still runs my die() function... well at least the first few lines (that usually should write some file until it closes clean) ... but even that function gets busy and does not finish (nor the program dies). So I'm not sure why the ErrorHandler shows such a strange behaviour. Maybe it's because I've connected some signals to my own functions and the OnError is doing something with that as well?

Code: Select all

signal_(2,@die())  
signal_(15,@die())
signal_(13,@do_nothing()) ; Ignore broken pipes from GnuTLS

Unfortunately the reported line (ret = poll(@checkin,num.i,1)) does not seem to be the evil one to me as well (but just one of the most used functions):

Code: Select all

  Structure pollfd
    fd.l
    events.w
    revents.w
  EndStructure

  #POLLIN     = $001
  #POLLPRI    = $002
  #POLLOUT    = $004
  #POLLRDNORM = $040
  #POLLRDBAND = $080
  #POLLWRNORM = $100
  #POLLWRBAND = $200
  
  #POLLERR    = $008
  #POLLHUP    = $010
  #POLLNVAL   = $020
  
  PrototypeC proto_poll(*fds.pollfd,num.i,timeout.l)
  Global libc = OpenLibrary(#PB_Any,"libstdc++.so.6")
  If Not libc 
    libc = OpenLibrary(#PB_Any,"libc.so.6")
  EndIf 
  If Not libc
    PrintN("Error: Need at least libc.so.6 or libstdc++.so.6 - but could not open any of them.")
    End 1
  EndIf 
  Global poll.proto_poll = GetFunction(libc,"poll")
  If Not poll
    PrintN("Error: Could not get poll function from libc")
    End 1
  EndIf 
  Procedure sd_in(socket.l)
    Protected checkin.pollfd
    Protected ret.l
    Protected num.i = 1 
    
    checkin\fd = socket
    checkin\events = (#POLLIN|#POLLRDBAND|#POLLERR|#POLLHUP|#POLLNVAL)
    ret = poll(@checkin,num.i,1) ; <----------------------------------------------------------- Line 1044
    If checkin\revents &(#POLLERR|#POLLHUP|#POLLNVAL)
      ret = -1
    EndIf 

    ProcedureReturn(ret)
  EndProcedure

Somebody got any further ideas? It yet again took about 2-3 weeks until the application died.