Bug 48

Summary: unexpected crashes on Debian amd64 RT kernel 2.6.23-rc2
Product: LinuxSampler Reporter: Jose Luis <joseluisjazz>
Component: gig::EngineAssignee: Christian Schoenebeck <cuse>
Status: CLOSED FIXED    
Severity: critical CC: daaugusto
Priority: P3    
Version: SVN Trunk   
Hardware: PC   
OS: Linux   
Attachments: Debug Patch
2nd debug patch

Description Jose Luis 2007-08-27 22:32:59 CEST
To reproduce them, I only have to charge some instruments (a piano for 
example), play with it over several minutes until it crashes. I send 2 
backtraces, hours apart one from each other:

# ------------ This is the first one:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1074796880 (LWP 12505)]
LinuxSampler::EngineChannel::GetMute (this=0x0) at EngineChannel.cpp:58
58          int EngineChannel::GetMute() {

(gdb) backtrace full
#0  LinuxSampler::EngineChannel::GetMute (this=0x0) at EngineChannel.cpp:58
No locals.
#1  0x00002b742a00d7b1 in LinuxSampler::gig::Engine::ProcessNoteOn
(this=0x639100, pEngineChannel=0x0,
   itNoteOnEvent=@0x40101010) at Engine.cpp:730
       key = 69
       pKey = <value optimized out>
       itNoteOnEventOnKeyList = {current = 0x64fc10, fallback = 0x2aaabaa8b168}
#2  0x00002b742a00dc53 in LinuxSampler::gig::Engine::ProcessEvents
(this=0x639100,
   pEngineChannel=<value optimized out>, Samples=<value optimized
out>) at Engine.cpp:489
       itEvent = {current = 0x64fc10, fallback = 0x2aaabaa710c8}
#3  0x00002b742a00ea5d in LinuxSampler::gig::Engine::RenderAudio
(this=0x639100, Samples=256) at Engine.cpp:429
       i = 3
       command = {pEngineChannel = 0x5, pInstrument = 0x621c60}
#4  0x00002b742a0442b1 in LinuxSampler::AudioOutputDevice::RenderAudio
(this=0x61b150, Samples=256)
   at AudioOutputDevice.cpp:266
       res = <value optimized out>
       result = 0
#5  0x00002b742b45b39d in ?? () from /usr/lib64/libjack.so.0
No symbol table info available.
#6  0x00002b742bd63317 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00002b742ab2eb1d in clone () from /lib64/libc.so.6
No symbol table info available.
#8  0x0000000000000000 in ?? ()
No symbol table info available.

# ----------
# ---------- And this is the second one

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1074796880 (LWP 4928)]
LinuxSampler::EngineChannel::GetMute (this=0x0) at EngineChannel.cpp:58
58          int EngineChannel::GetMute() {
(gdb) bt full
#0  LinuxSampler::EngineChannel::GetMute (this=0x0) at EngineChannel.cpp:58
No locals.
#1  0x00002b64576df991 in LinuxSampler::gig::Engine::ProcessNoteOn (
    this=0x641b50, pEngineChannel=0x0, itNoteOnEvent=@0x40101010)
    at Engine.cpp:730
        key = 61
        pKey = <value optimized out>
        itNoteOnEventOnKeyList = {current = 0x66e4e0, fallback = 0x6aa3d8}
#2  0x00002b64576dfe33 in LinuxSampler::gig::Engine::ProcessEvents (
    this=0x641b50, pEngineChannel=<value optimized out>, 
    Samples=<value optimized out>) at Engine.cpp:489
        itEvent = {current = 0x66e4e0, fallback = 0x6a7d18}
#3  0x00002b64576e0ccd in LinuxSampler::gig::Engine::RenderAudio (
    this=0x641b50, Samples=64) at Engine.cpp:429
        i = 1
        command = {pEngineChannel = 0x8, pInstrument = 0x41103040}
#4  0x00002b6457717011 in LinuxSampler::AudioOutputDevice::RenderAudio (
    this=0x61b150, Samples=64) at AudioOutputDevice.cpp:266
        res = <value optimized out>
        result = 0
#5  0x00002b6458b2f39d in ?? () from /usr/lib64/libjack.so.0
No symbol table info available.
#6  0x00002b6459437317 in start_thread () from /lib64/libpthread.so.0

# ------------
Comment 1 Jose Luis 2007-08-28 18:48:34 CEST
Additional information on the system contitions:

This happens using jack SVN version. Up to 0.107.2-SVN20070826 that is the one 
I have installed now.

# uname -a
Linux oblivion 2.6.23-rc2-rt2-RT1000IOanticMCsched20070826 #4 SMP PREEMPT RT 
Sun Aug 26 02:40:25 CEST 2007 x86_64 GNU/Linux

Audio adapter: USB-Audio-UA-25
Video adapter: ATI radeon x800
Comment 2 Jose Luis 2007-09-01 16:25:14 CEST
Also crashes on 2.6.23-rc4-rt1

This is another backtrace. I had the idea of recording what I played, to see if 
I could reproduce the crash while playing it again. I played the recorded midi 
track, but I didn't manage to crash it again.

The crash happened when I was playing an easy part of a song. With 10 - 15 
simultaneous notes as much. So when I saw that what I played didn't repeated 
the effect, I started to play randomnly with 150 - 256 polyphony notes but 
still didn't manage to crash it...

I don't know yet the reason for this crashes.

# ---------------------8<----------

LSCPServer: GetBufferFill(ResponseType=1, SamplerChannel=0)
LSCPServer::AnswerClient(ReturnMessage=[2955]199,[2972]199%,[2961]199%,[2971]199%,[2949]199%,[2970]199%,[2968]199%,[2967]199%,[2959]199%,[2969]199%,[2964]199%,[2974]199%,[2973]199%,[2956]184%,[2942]85%
)LSCPServer: Done parsing on socket -1.
Disk Thread: new stream ordered
Disk voice launched (cached samples: 32768, total Samples: 706288, MaxRAMPos: 
30720, RAMLooping: no)
Depth=1500, DecayTime=1.000000
Engine::ContinuousController cc=64 v=88
Engine::ContinuousController cc=64 v=24
DAMPER (RIGHT) PEDAL UP
Engine::ContinuousController cc=64 v=0
new Stream launched by disk thread (OrderID:275,StreamHandle:2975)
Disk Thread: been asked if stream already created, OrderID=111 (yes created)
Disk Thread: stream deletion ordered
Disk Thread: stream deletion ordered
Disk Thread: stream deletion ordered
Disk Thread: stream deletion ordered
Disk Thread: new stream ordered
Disk voice launched (cached samples: 32768, total Samples: 706560, MaxRAMPos: 
30720, RAMLooping: no)
Depth=1500, DecayTime=1.000000
new Stream launched by disk thread (OrderID:276,StreamHandle:2976)
Disk Thread: stream deletion ordered
Key has no more voices now
Disk Thread: stream deletion ordered
Key has no more voices now
Disk Thread: stream deletion ordered
Disk Thread: stream deletion ordered
Key has no more voices now
Disk Thread: stream deletion ordered
Key has no more voices now
Engine::ContinuousController cc=64 v=24
Engine::ContinuousController cc=64 v=56
Younger Event, pos=383 ,Samples=256!

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1074796880 (LWP 8004)]
LinuxSampler::EngineChannel::GetMute (this=0x0) at EngineChannel.cpp:58
58          int EngineChannel::GetMute() {
(gdb) bt full
#0  LinuxSampler::EngineChannel::GetMute (this=0x0) at EngineChannel.cpp:58
No locals.
#1  0x00002af90d67f5b1 in LinuxSampler::gig::Engine::ProcessNoteOn 
(this=0x6261d0, pEngineChannel=0x0, itNoteOnEvent=@0x40101010) at Engine.cpp:730
        key = 0
        pKey = <value optimized out>
        itNoteOnEventOnKeyList = {current = 0x2af90e41f780, fallback = 
0x2af90e12ec4a}
#2  0x00002af90d67fa53 in LinuxSampler::gig::Engine::ProcessEvents 
(this=0x6261d0, pEngineChannel=<value optimized out>, 
    Samples=<value optimized out>) at Engine.cpp:489
        itEvent = {current = 0x720c88, fallback = 0x6af6d8}
#3  0x00002af90d68070d in LinuxSampler::gig::Engine::RenderAudio 
(this=0x6261d0, Samples=256) at Engine.cpp:429
        i = 1
        command = {pEngineChannel = 0x6321e0, pInstrument = 0x2aaaacffa8a0}
#4  0x00002af90d6b64b1 in LinuxSampler::AudioOutputDevice::RenderAudio 
(this=0x633c00, Samples=256) at AudioOutputDevice.cpp:266
        res = <value optimized out>
        result = 0
#5  0x00002af90eace39d in ?? () from /usr/lib64/libjack.so.0
No symbol table info available.
#6  0x00002af90f3d6317 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00002af90e1a1b1d in clone () from /lib64/libc.so.6
No symbol table info available.
#8  0x0000000000000000 in ?? ()
No symbol table info available.

# --------------->8--------------
Comment 3 Christian Schoenebeck 2007-09-01 19:49:38 CEST
Is this an instrument specific crash? If yes, what is the name & vendor of the 
instrument?
Comment 4 Jose Luis 2007-09-01 20:00:55 CEST
No, it seems it's not instrument specific. It has happened while playing 
completely different instruments, from different sources. If you want me to 
test with any speficic instrument, and/or to do more test to isolate the 
problem, propose them to me. For example, compiling linuxsampler with the 
specific options you say, in order to discard some variables out of the 
ecuation. I'll be willing to help to fix this.

At least the error seems it's always the same. hehe
Comment 5 Christian Schoenebeck 2007-09-04 15:07:25 CEST
Reduce the debug level and check if there was an error message on the console 
before the crash.

Did you send MIDI events during a program change / instrument change or during 
another sampler setup operation? If yes, if you don't do that, that is just 
sending MIDI events when the sampler session changes are completely done, do 
you still get this crash?
Comment 6 Christian Schoenebeck 2007-09-04 15:10:49 CEST
Sorry, i meant "reduce the debug level to exactly 1", which is the default 
setting ...
Comment 7 Christian Schoenebeck 2007-09-04 15:51:05 CEST
Created attachment 17 [details]
Debug Patch

Please try the uploaded patch. It won't fix the problem, but it might help to
sort out the cause. Simply download it somewhere and type the following on the
console:

cd YOUR_LS_SOURCE_DIR
patch -p0 < LOCATION_OF_PATCH/debug.patch
make
Comment 8 Christian Schoenebeck 2007-09-04 15:53:39 CEST
After compiling with that patch, continue to watch out for error messages on 
the console.
Comment 9 Christian Schoenebeck 2007-10-04 14:44:03 CEST
So did the problem vanish? I'm sorry but I need more information from your 
side, since I cannot reproduce this bug.

I'm marking this report as WORKSFORME now, feel free to reopen it in case this 
problem is still persistent.
Comment 10 Jose Luis 2007-10-04 15:40:05 CEST
Ok, sorry about the lack of news. So I decided to reinstall my debian system 
and since then I couldn't reproduce the crash again, with or without your 
patch. So I think you can leave it as resolved.

Anyway I've just moved back again to windows because in fact I was spending too 
much time dealing with linux things, compiling, writing scripts, etc. than 
actually working on my music.

I'll try again to work with linux some day in the future, but for now this is 
what it is.

thankyou Christian for your care,
Comment 11 Douglas A. Augusto 2007-11-10 01:43:18 CET
I'm experiencing this problem as well. Debian GNU/Linux AMD64, kernel
2.6.22-3-amd64.

Linuxsampler (I've tried version 0.4.0 and 0.5.0) crashes after some minutes. I
am using FreePiano.gig (http://www.alchemystudio.it/FreeSamples/FreePiano.zip)
gigasample, but LinuxSampler also crashes with other piano gigasamples, like
East-West "Bosendorfer 275" or PMI "The Old Lady Model 1923 Steinway D".

Steps to reproduce:

1) ./linuxsampler

2) cat script.lscp | netcat localhost 8888

----------------------------
script.lscp
----------------------------
CREATE AUDIO_OUTPUT_DEVICE ALSA
CREATE MIDI_INPUT_DEVICE ALSA
ADD CHANNEL
LOAD ENGINE gig 0
SET CHANNEL AUDIO_OUTPUT_DEVICE 0 0
SET CHANNEL MIDI_INPUT_DEVICE 0 0
LOAD INSTRUMENT 'FreePiano.gig' 0 0
QUIT
----------------------------

3) pmidi -p 128:0 any_midi_file.mid

4) wait a few minutes... crash!

Backtrace:

(gdb) bt full
#0  LinuxSampler::EngineChannel::GetMute (this=0x0) at EngineChannel.cpp:59
No locals.
#1  0x00002b8e48babeff in LinuxSampler::gig::Engine::ProcessNoteOn
(this=0x627dc0, pEngineChannel=0x0,
    itNoteOnEvent=@0x40101000) at Engine.cpp:959
        key = <value optimized out>
        pKey = <value optimized out>
        itNoteOnEventOnKeyList = {current = 0x2b8e4afaa780, fallback =
0x2b8e4acc4fdf}
#2  0x00002b8e48bac451 in LinuxSampler::gig::Engine::ProcessEvents (this=0x627dc0,
    pEngineChannel=<value optimized out>, Samples=<value optimized out>) at
Engine.cpp:718
        itEvent = {current = 0x63faa8, fallback = 0x67c5d8}
#3  0x00002b8e48bad10e in LinuxSampler::gig::Engine::RenderAudio (this=0x627dc0,
Samples=128) at Engine.cpp:653
        i = 1
        command = {pEngineChannel = 0x61d1a0, pInstrument = 0x699280}
#4  0x00002b8e48be529c in LinuxSampler::AudioOutputDevice::RenderAudio
(this=0x6180d0, Samples=128)
    at AudioOutputDevice.cpp:267
        res = <value optimized out>
        result = 0
#5  0x00002b8e48bf1fd7 in LinuxSampler::AudioOutputDeviceAlsa::Main
(this=0x6180d0) at AudioOutputDeviceAlsa.cpp:749
        res = 6527552
#6  0x00002b8e48c1b044 in __pthread_launcher (thread=0x6181d0) at Thread.cpp:253
        t = (LinuxSampler::Thread *) 0x0
#7  0x00002b8e4a2ac317 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#8  0x00002b8e4ad2cb1d in clone () from /lib/libc.so.6
No symbol table info available.
#9  0x0000000000000000 in ?? ()
No symbol table info available.

LinuxSampler was compiled with: 

$ CXXFLAGS="-O2 -g3" ./configure --enable-debug-level=10 && make


Some debug messages before it crashes:

(...)
RenderAudio(Samples=128)
RenderAudio(Samples=128)
RenderAudio(Samples=128)
RenderAudio(Samples=128)
RenderAudio(Samples=128)
Voice: switching to disk playback (Pos=31811.440342)
RenderAudio(Samples=128)
Disk Thread: stream deletion ordered
Disk Thread: been asked if stream already created, OrderID=1 (yes created)
RenderAudio(Samples=128)
Disk Thread: stream deletion ordered
RenderAudio(Samples=128)
RenderAudio(Samples=128)
Disk Thread: stream deletion ordered
Disk Thread: stream deletion ordered
Disk Thread: stream deletion ordered
Disk Thread: stream deletion ordered
Key has no more voices now
Key has no more voices now
RenderAudio(Samples=128)
 Refilled stream 40491 with 1724 (SamplePos: 380576)Refilled stream 40500 with
1724 (SamplePos: 304106)Refilled stream 40499 with
1628(SamplePos:306404)Refilled stream 40494 with 1628 (SamplePos:340578)Refilled
stream 40493 with 1628 (SamplePos: 349701)Refilled stream 40495 with 1627
(SamplePos: 333961)Refilled stream 40498 with 1627 (SamplePos: 312518) Refilled
stream 40497 with 1627 (SamplePos: 318083)Refilled stream 40496 with 1581
(SamplePos: 330788)Refilled stream 40483 with 1580 (SamplePos: 433071)Refilled
stream 40484 with 1580 (SamplePos: 426061)Refilled stream 40492 with 1580
(SamplePos: 373083)Refilled stream 40481 with 1536 (SamplePos: 426567)Refilled
stream 40490 with 1536 (SamplePos: 373468) Refilled stream 40482 with 1536
(SamplePos: 420237)Refilled stream 40489 with 1536 (SamplePos: 379973)Refilled
stream 4 0479 with 1536 (SamplePos: 438711)Refilled stream 40477 with 1536
(SamplePos: 451845)Refilled stream 40486 with 1536 (S amplePos: 397628)Refilled
stream 40467 with 1536 (SamplePos: 567215)Refilled stream 40474 with 1536
(SamplePos: 474411) Refilled stream 40487 with 1536 (SamplePos: 391298)Refilled
stream 40473 with 1536 (SamplePos: 482894)Refilled stream 4 0464 with 1536
(SamplePos: 567301)Refilled stream 40485 with 1536 (SamplePos: 402882)Refilled
stream 40468 with 1536 (S amplePos: 567129)Refilled stream 40480 with 1408
(SamplePos: 432811)Refilled stream 40501 with 0 (SamplePos: 293887)Ref illed
stream 40501 with 0 (SamplePos: 293887)Refilled stream 40485 with 0 (SamplePos:
402882)Refilled stream 40468 with 0 (SamplePos: 567129)Refilled stream 40480
with 0 (SamplePos: 432811)RenderAudio(Samples=128)
RenderAudio(Samples=128)
Disk Thread: stream deletion ordered
RenderAudio(Samples=128)
RenderAudio(Samples=128)
RenderAudio(Samples=128)
Disk Thread: stream deletion ordered
RenderAudio(Samples=128)
RenderAudio(Samples=128)
Younger Event, pos=1369 ,Samples=128!
RenderAudio(Samples=128)
Engine: Note on received

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x40101950 (LWP 4901)]
LinuxSampler::EngineChannel::GetMute (this=0x0) at EngineChannel.cpp:59
59          int EngineChannel::GetMute() {
Comment 12 Douglas A. Augusto 2007-11-10 03:34:38 CET
Hi,

LinuxSampler CVS HEAD also crashes. I tried the proposed patch but it doesn't
give any extra information. Here is the Backtrace:


# gdb ./linuxsampler
GNU gdb 6.6.90.20070912-debian
Copyright (C) 2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
Using host libthread_db library "/lib/libthread_db.so.1".
(gdb) run
Starting program: /tmp/sampler/bin/linuxsampler
[Thread debugging using libthread_db enabled]
[New Thread 0x2abbdcc782e0 (LWP 27412)]
LinuxSampler 0.5.0
Copyright (C) 2003,2004 by Benno Senoner and Christian Schoenebeck
Copyright (C) 2005-2007 Christian Schoenebeck
Detected features: disabled at compile time
Creating Sampler...OK
Registered sampler engines: 'GIG'
Registered MIDI input drivers: ALSA
Registered audio output drivers: ALSA,ARTS,JACK
Loading instrument editor plugins...OK
Registered instrument editors:
Starting LSCP network server (0.0.0.0:8888)...[New Thread 0x40080950 (LWP 27415)]
LSCPServer: Could not bind server socket, retrying for 180 seconds...OK
LinuxSampler initialization completed. :-)

LSCPServer: Client connection established on socket:7.
[New Thread 0x40101950 (LWP 27444)]
[New Thread 0x40182950 (LWP 27445)]
Starting disk thread...[New Thread 0x40203950 (LWP 27446)]
OK
[Thread 0x40203950 (LWP 27446) exited]
[New Thread 0x40284950 (LWP 27449)]
Loading gig file '/tmp/dvd/gigasamples/FreePiano.gig'...OK
Loading gig instrument ('/tmp/dvd/gigasamples/FreePiano.gig',0)...OK
Caching initial samples...OK
LSCPServer: Client connection terminated on socket:7.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x40101950 (LWP 27444)]
LinuxSampler::EngineChannel::GetMute (this=0x0) at EngineChannel.cpp:59
59          int EngineChannel::GetMute() {

(gdb) bt full
#0  LinuxSampler::EngineChannel::GetMute (this=0x0) at EngineChannel.cpp:59
No locals.
#1  0x00002abbda9a5dcf in LinuxSampler::gig::Engine::ProcessNoteOn
(this=0x627dc0, pEngineChannel=0x0, itNoteOnEvent=@0x40101000) at Engine.cpp:959
        key = <value optimized out>
        pKey = <value optimized out>
        itNoteOnEventOnKeyList = {current = 0x80006177a0, fallback = 0x61d1a0}
#2  0x00002abbda9a627b in LinuxSampler::gig::Engine::ProcessEvents
(this=0x627dc0, pEngineChannel=<value optimized out>,
    Samples=<value optimized out>) at Engine.cpp:718
        itEvent = {current = 0x63fa30, fallback = 0x67c5d8}
#3  0x00002abbda9a6e61 in LinuxSampler::gig::Engine::RenderAudio (this=0x627dc0,
Samples=128) at Engine.cpp:653
        i = 1
        command = {pEngineChannel = 0x61d1a0, pInstrument = 0x6992b0}
#4  0x00002abbda9e25cc in LinuxSampler::AudioOutputDevice::RenderAudio
(this=0x6180d0, Samples=128) at AudioOutputDevice.cpp:267
        res = <value optimized out>
        result = 0
#5  0x00002abbda9ef307 in LinuxSampler::AudioOutputDeviceAlsa::Main
(this=0x6180d0) at AudioOutputDeviceAlsa.cpp:749
        res = 6527392
#6  0x00002abbdaa17ed4 in __pthread_launcher (thread=0x6181d0) at Thread.cpp:253
        t = (LinuxSampler::Thread *) 0x0
#7  0x00002abbdb421317 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#8  0x00002abbdb709b1d in clone () from /lib/libc.so.6
No symbol table info available.
#9  0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) cont
Continuing.
>>> FATAL ERROR: Segmentation fault (SIGSEGV) occured! <<<
Showing stack trace...
Using host libthread_db library "/lib/libthread_db.so.1".
Attaching to program: /tmp/sampler/bin/linuxsampler, process 27412
(gdb) --- Stacktrace
--- Symbols
 > > > >---
Killing LinuxSampler...

Program terminated with signal SIGKILL, Killed.
The program no longer exists.
(gdb)
Comment 13 Douglas A. Augusto 2007-11-10 06:55:42 CET
The bug seems to be inside the function Engine::ProcessEvents(EngineChannel*
pEngineChannel, uint Samples), at linuxsampler/src/engines/gig/Engine.cpp.

When I replace the following code

   case Event::type_note_on:
      dmsg(5,("Engine: Note on received\n"));
      ProcessNoteOn((EngineChannel*)itEvent->pEngineChannel, itEvent);
      break;

by

   case Event::type_note_on:
      dmsg(5,("Engine: Note on received\n"));
      if (itEvent->pEngineChannel)
ProcessNoteOn((EngineChannel*)itEvent->pEngineChannel, itEvent);
      break;

then instead of crashing LinuxSampler produces just an annoying noise at
this moment. I couldn't figure out why at times "itEvent->pEngineChannel" is null.
Comment 14 Christian Schoenebeck 2007-11-10 13:40:49 CET
Have you placed equivalent NULL checks in EngineChannel::ImportEvents() as 
well?

Do our test cases all pass on your box?
( make tests && src/testcasese/linuxsamplertest )
Comment 15 Christian Schoenebeck 2007-11-10 14:05:24 CET
You could also check, whether it's a bug in the voice stealing algorithm. You 
can to that by:

    ./configure --enable-voice-steal-algo=FOO && make

where FOO is one of:

none:
    Disable voice stealing completely.
oldestvoiceonkey (default):
    Try to kill a voice on the same key first,
    if no success, proceed with the oldest key.
oldestkey:
    Try to kill a voice from the oldest active
    key. This voice stealing algorithm is simpler
    than the default one.

So "none" might be a good candidate to try out.
Comment 16 Douglas A. Augusto 2007-11-10 22:32:06 CET
> Have you placed equivalent NULL checks in EngineChannel::ImportEvents() as 
> well?

No, I didn't. What do you suggest I should do?

> Do our test cases all pass on your box?
> ( make tests && src/testcasese/linuxsamplertest )

See:

% src/testcases/linuxsamplertest

Running Pool Tests: ....................
Running Thread Tests: .....
Running Mutex Tests: .....
Running Condition Tests: ....
Running LSCP Tests: .Thread: WARNING, can't mlockall() memory!
.LSCPServer: Client connection established on socket:6.
LSCPServer: Client connection terminated on socket:6.
.LSCPServer: Client connection established on socket:6.
...Thread: WARNING, can't assign realtime scheduling to thread!
Thread: WARNING, can't mlockall() memory!
..LSCPServer: Client connection terminated on socket:6.



OK (47 tests)
Comment 17 Douglas A. Augusto 2007-11-10 22:37:25 CET
Testcases with linuxsampler running:

1) as normal user:

% src/testcases/linuxsamplertest

Running Pool Tests: ....................
Running Thread Tests: .....
Running Mutex Tests: .....
Running Condition Tests: ....
Running LSCP Tests: .Thread: WARNING, can't mlockall() memory!
LSCPServer: Could not bind server socket, retrying for 180 seconds..........


OK (47 tests)

2) as root:

# src/testcases/linuxsamplertest

Running Pool Tests: ....................
Running Thread Tests: .....
Running Mutex Tests: .....
Running Condition Tests: ....
Running LSCP Tests: .LSCPServer: Could not bind server socket, retrying for 180
seconds..........


OK (47 tests)
Comment 18 Douglas A. Augusto 2007-11-10 23:15:36 CET
> You could also check, whether it's a bug in the voice stealing algorithm. You 
> can to that by:

>    ./configure --enable-voice-steal-algo=FOO && make

I tried with both 'none' and 'oldestkey', but LinuxSampler still crashes.
Comment 19 Christian Schoenebeck 2007-11-12 17:19:30 CET
Created attachment 18 [details]
2nd debug patch

With NULL check I mean something like in the attached patch. The point is the
pEngineChannel argument of event objects may only be NULL on engine global
events, that is on SysEx messages, definitely not on note-on messages. So the
question is where and why this arguments gets NULL. Might be some dirty buffer
overflow bug, caused by a 64 bit issue, we'll see...

As you can see in src/engines/gig/EngineChannel.cpp, the pEngineChannel
argument of the Event object is set in the SendNoteOn() method, so that's why I
placed a sanity check / debug message in first place there. If you don't see
any of those debug messages, then we'll at least know that the argument is
modified after this point.

So in the SendNoteOn() method the Event object goes into a queue and is later
pulled out by ImportEvents() in the same source file. So that's why I placed
further debug code in ImportEvents() in that patch as well.

That way we can check if that pEngineChannel argument is modified somewhere in
that queue code.

At least we know that bug is not caused by voice stealing. For further
investigation better keep the voice stealing turned off though, just to be
sure.
Comment 20 Christian Schoenebeck 2007-11-12 17:21:48 CET
Oh, and please note, in ImportEvents() the patch intentionally forces a 
segfault in case the pEngineChannel argument is NULL on note-on events. Maybe 
the resulting stack trace can help us further to investigate the root cause of 
this whole issue.
Comment 21 Douglas A. Augusto 2007-11-12 18:14:31 CET
Hi Christian,

I applied your 2dn debug patch on LinuxSampler CVS HEAD. LinuxSampler was
configured with '--enable-voice-steal-algo=none' and '-g -O0' C++ flags.

Here is the crash report using the new patch:

(...)

No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
pEngineChanell is NULL in note-on event !!!!

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x40101950 (LWP 24001)]
0x00002b75ae0e81f8 in LinuxSampler::EngineChannel::GetMute (this=0x0)
    at EngineChannel.cpp:60
60              return iMute;
(gdb) bt full
#0  0x00002b75ae0e81f8 in LinuxSampler::EngineChannel::GetMute (this=0x0)
    at EngineChannel.cpp:60
No locals.
#1  0x00002b75ae0e2900 in LinuxSampler::gig::EngineChannel::ImportEvents (
    this=0x6211a0, Samples=128) at EngineChannel.cpp:697
        eventQueueReader = {pBuf = 0x61b720, read_ptr = 0}
        pEvent = (LinuxSampler::Event *) 0x622f70
#2  0x00002b75ae0b5140 in LinuxSampler::gig::Engine::ProcessEvents (
    this=0x62bdc0, pEngineChannel=0x6211a0, Samples=128) at Engine.cpp:708
No locals.
#3  0x00002b75ae0b5d3b in LinuxSampler::gig::Engine::RenderAudio (
    this=0x62bdc0, Samples=128) at Engine.cpp:653
        i = 0
        command = {pEngineChannel = 0x40101030, pInstrument = 0x4056ea}
#4  0x00002b75ae0fefcb in LinuxSampler::AudioOutputDevice::RenderAudio (
    this=0x61c0d0, Samples=128) at AudioOutputDevice.cpp:267
        res = 128
        iterEngine = {_M_node = 0x685680}
        end = {_M_node = 0x61c118}
        result = 0
        engines = (
    const
std::set<LinuxSampler::Engine*,std::less<LinuxSampler::Engine*>,std::allocator<LinuxSampler::Engine*>
> &) @0x61c110: {_M_t = {
    _M_impl = {<std::allocator<std::_Rb_tree_node<LinuxSampler::Engine*> >> =
{<__gnu_cxx::new_allocator<std::_Rb_tree_node<LinuxSampler::Engine*> >> = {<No
data fields>}, <No data fields>},
      _M_key_compare =
{<std::binary_function<LinuxSampler::Engine*,LinuxSampler::Engine*,bool>> = {<No
data fields>}, <No data fields>}, _M_header = {
        _M_color = std::_S_red, _M_parent = 0x685680, _M_left = 0x685680,
        _M_right = 0x685680}, _M_node_count = 1}}}
#5  0x00002b75ae10dc2d in LinuxSampler::AudioOutputDeviceAlsa::Main (
    this=0x61c0d0) at AudioOutputDeviceAlsa.cpp:749
        res = 0
#6  0x00002b75ae13ce99 in __pthread_launcher (thread=0x61c1d0)
    at Thread.cpp:253
        t = (LinuxSampler::Thread *) 0x61c1d0
#7  0x00002b75aeb79317 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#8  0x00002b75aee61b1d in clone () from /lib/libc.so.6
No symbol table info available.
#9  0x0000000000000000 in ?? ()
No symbol table info available.
Comment 22 Christian Schoenebeck 2007-11-12 18:43:44 CET
Was there an error message like "!!! ERROR !pEngineChannel !!! -> " as well? 
You might reduce the debug level for this to 1 to make sure of this, since 
this debug error message from the patch is printed for all debug levels. That 
would be important to know, because if there was no such error message, then 
we'll know the modification must be somewhere in between these two calls, thus 
probably a 64bit bug in the RingBuffer or RTList code.
Comment 23 Douglas A. Augusto 2007-11-12 18:58:25 CET
> Was there an error message like "!!! ERROR !pEngineChannel !!! -> " as well? 

No, there wasn't.

> You might reduce the debug level for this to 1 to make sure of this, since 
> this debug error message from the patch is printed for all debug levels. That 
> would be important to know, because if there was no such error message, then 
> we'll know the modification must be somewhere in between these two calls, thus 
> probably a 64bit bug in the RingBuffer or RTList code.

I'm using the default debug level (=1) already. See the complete output until
crashing:

(gdb) run
`/tmp/sampler/bin/linuxsampler' has changed; re-reading symbols.
Starting program: /tmp/sampler/bin/linuxsampler
[Thread debugging using libthread_db enabled]
[New Thread 0x2b75b03d02e0 (LWP 23990)]
LinuxSampler 0.5.0
Copyright (C) 2003,2004 by Benno Senoner and Christian Schoenebeck
Copyright (C) 2005-2007 Christian Schoenebeck
Detected features: disabled at compile time
Creating Sampler...OK
Registered sampler engines: 'GIG'
Registered MIDI input drivers: ALSA
Registered audio output drivers: ALSA,ARTS,JACK
Loading instrument editor plugins...OK
Registered instrument editors:
Starting LSCP network server (0.0.0.0:8888)...[New Thread 0x40080950 (LWP 23991)]
OK
LinuxSampler initialization completed. :-)

LSCPServer: Client connection established on socket:8.
[New Thread 0x40101950 (LWP 24001)]
[New Thread 0x40182950 (LWP 24002)]
Starting disk thread...[New Thread 0x40203950 (LWP 24003)]
OK
[New Thread 0x40284950 (LWP 24004)]
[Thread 0x40203950 (LWP 24003) exited]
Loading gig file '/tmp/dvd/gigasamples/FreePiano.gig'...OK
Loading gig instrument ('/tmp/dvd/gigasamples/FreePiano.gig',0)...OK
Caching initial samples...OK
LSCPServer: Client connection terminated on socket:8.
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
No free voice (voice stealing disabled)!
pEngineChanell is NULL in note-on event !!!!

Program received signal SIGSEGV, Segmentation fault.
Comment 24 Christian Schoenebeck 2007-11-14 16:28:55 CET
Ok, I think I found a bug in the RingBuffer code which could have caused that 
crash. It wasn't a 64 bit issue btw. Benno just commited a fix minutes ago. So 
please update from latest CVS and let us know if it fixes your problem.
Comment 25 Douglas A. Augusto 2007-11-16 16:36:55 CET
Hi Christian,

For the first time I was able to play continuously my entire classical MIDI
collection (several hours) with no crashes at all. So, the bug #48 seems fixed
to me.

Thank you for your interest.
Comment 26 Christian Schoenebeck 2007-11-16 16:47:30 CET
Thanks for your help and patience!