shorne in japan

blog archive about resume

Unwinding a Bug - How C++ Exceptions Work

13 Dec 2020

I have been working on porting GLIBC to the OpenRISC architecture. This has taken longer than I expected as with GLIBC upstreaming we must get every test to pass. This was different compared to GDB and GCC which were a bit more lenient.

My first upstreaming attempt was completely tested on the QEMU simulator. I have since added an FPGA LiteX SoC to my test platform options. LiteX runs Linux on the OpenRISC mor1kx softcore and tests are loaded over an SSH session. The SoC eliminates an issue I was seeing on the simulator where under heavy load it appears the MMU starves the kernel from getting any work done.

To get to where I am now this required:

Adding GDB Linux debugging support is great because it allows debugging of multithreaded processes and signal handling; which we are going to need.

A Bug

Our story starts when I was trying to fix a failing GLIBC NPTL test case. The test case involves C++ exceptions and POSIX threads. The issue is that the catch block of a try/catch block is not being called. Where do we even start?

My plan for approaching test case failures is:

  1. Understand what the test case is trying to test and where its failing
  2. Create a hypothesis about where the problem is
  3. Understand how the failing API’s works internally
  4. Debug until we find the issue
  5. If we get stuck go back to 2.

Let’s have a try.

Understanding the Test case

The GLIBC test case is nptl/tst-cancel24.cc. The test starts in the do_test function and it will create a child thread with pthread_create. The child thread executes function tf which waits on a semaphore until the parent thread cancels it. It is expected that the child thread, when cancelled , will call it’s catch block.

The failure is that the catch block is not getting run as evidenced by the except_caught variable not being set to true.

Below is an excerpt from the test showing the tf function.

static void *
tf (void *arg) {
  sem_t *s = static_cast<sem_t *> (arg);

  try {
      monitor m;

      pthread_barrier_wait (&b);

      while (1)
        sem_wait (s);
  } catch (...) {
      except_caught = true;
      throw;
  }
  return NULL;
}

So the catch block is not being run. Simple, but where do we start to debug that? Let’s move onto the next step.

Creating a Hypothesis

This one is a bit tricky as it seems C++ try/catch blocks are broken. Here, I am working on GLIBC testing, what does that have to do with C++?

To get a better idea of where the problem is I tried to modify the test to test some simple ideas. First, maybe there is a problem with catching exceptions throws from thread child functions.

static void do_throw() { throw 99; }

static void * tf () {
  try {
      monitor m;
      while (1) do_throw();
  } catch (...) {
      except_caught = true;
  }
  return NULL;
}

No, this works correctly. So try/catch is working.

Hypothesis: There is a problem handling exceptions while in a syscall. There may be something broken with OpenRISC related to how we setup stack frames for syscalls that makes the unwinder fail.

How does that work? Let’s move onto the next step.

Understanding the Internals

To find this bug we need to understand how C++ exceptions work. Also, we need to know what happens when a thread is cancelled in a multithreaded (pthread) glibc environment.

There are a few contributors pthread cancellation and C++ exceptions which are:

DWARF

ELF binaries provide debugging information in a data format called DWARF. The name was chosen to maintain a fantasy theme. Lately the Linux community has a new debug format called ORC.

Though DWARF is a debugging format and usually stored in .debug_frame, .debug_info, etc sections, a stripped down version it is used for exception handling.

Each ELF binary that supports unwinding contains the .eh_frame section to provide unwinding information. This can be seen with the readelf program.

$ readelf -S sysroot/lib/libc.so.6
There are 70 section headers, starting at offset 0xaa00b8:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .note.ABI-tag     NOTE            00000174 000174 000020 00   A  0   0  4
  [ 2] .gnu.hash         GNU_HASH        00000194 000194 00380c 04   A  3   0  4
  [ 3] .dynsym           DYNSYM          000039a0 0039a0 008280 10   A  4  15  4
  [ 4] .dynstr           STRTAB          0000bc20 00bc20 0054d4 00   A  0   0  1
  [ 5] .gnu.version      VERSYM          000110f4 0110f4 001050 02   A  3   0  2
  [ 6] .gnu.version_d    VERDEF          00012144 012144 000080 00   A  4   4  4
  [ 7] .gnu.version_r    VERNEED         000121c4 0121c4 000030 00   A  4   1  4
  [ 8] .rela.dyn         RELA            000121f4 0121f4 00378c 0c   A  3   0  4
  [ 9] .rela.plt         RELA            00015980 015980 000090 0c  AI  3  28  4
  [10] .plt              PROGBITS        00015a10 015a10 0000d0 04  AX  0   0  4
  [11] .text             PROGBITS        00015ae0 015ae0 155b78 00  AX  0   0  4
  [12] __libc_freeres_fn PROGBITS        0016b658 16b658 001980 00  AX  0   0  4
  [13] .rodata           PROGBITS        0016cfd8 16cfd8 0192b4 00   A  0   0  4
  [14] .interp           PROGBITS        0018628c 18628c 000018 00   A  0   0  1
  [15] .eh_frame_hdr     PROGBITS        001862a4 1862a4 001a44 00   A  0   0  4
  [16] .eh_frame         PROGBITS        00187ce8 187ce8 007cf4 00   A  0   0  4
  [17] .gcc_except_table PROGBITS        0018f9dc 18f9dc 000341 00   A  0   0  1
...

We can decode the metadata using readelf as well using the --debug-dump=frames-interp and --debug-dump=frames arguments.

The frames dump provides a raw output of the DWARF metadata for each frame. This is not usually as useful as frames-interp, but it shows how the DWARF format is actually a bytecode. The DWARF interpreter needs to execute these operations to understand how to derive the values of registers based current PC.

There is an interesting talk in Exploiting the hard-working DWARF.pdf.

An example of the frames dump:

$ readelf --debug-dump=frames sysroot/lib/libc.so.6

...
00016788 0000000c ffffffff CIE
  Version:               1
  Augmentation:          ""
  Code alignment factor: 4
  Data alignment factor: -4
  Return address column: 9

  DW_CFA_def_cfa_register: r1
  DW_CFA_nop

00016798 00000028 00016788 FDE cie=00016788 pc=0016b584..0016b658
  DW_CFA_advance_loc: 4 to 0016b588
  DW_CFA_def_cfa_offset: 4
  DW_CFA_advance_loc: 8 to 0016b590
  DW_CFA_offset: r9 at cfa-4
  DW_CFA_advance_loc: 68 to 0016b5d4
  DW_CFA_remember_state
  DW_CFA_def_cfa_offset: 0
  DW_CFA_restore: r9
  DW_CFA_restore_state
  DW_CFA_advance_loc: 56 to 0016b60c
  DW_CFA_remember_state
  DW_CFA_def_cfa_offset: 0
  DW_CFA_restore: r9
  DW_CFA_restore_state
  DW_CFA_advance_loc: 36 to 0016b630
  DW_CFA_remember_state
  DW_CFA_def_cfa_offset: 0
  DW_CFA_restore: r9
  DW_CFA_restore_state
  DW_CFA_advance_loc: 40 to 0016b658
  DW_CFA_def_cfa_offset: 0
  DW_CFA_restore: r9

The frames-interp argument is a bit more clear as it shows the interpreted output of the bytecode. Below we see two types of entries:

The CIE provides starting point information for each child FDE entry. Some things to point out: we see ra=9 indicates the return address is stored in register r9, we see CFA r1+0 indicates the canonical frame pointer is stored in register r1 and we see the stack frame size is 4 bytes.

An example of the frames-interp dump:

$ readelf --debug-dump=frames-interp sysroot/lib/libc.so.6

...
00016788 0000000c ffffffff CIE "" cf=4 df=-4 ra=9
   LOC   CFA
00000000 r1+0

00016798 00000028 00016788 FDE cie=00016788 pc=0016b584..0016b658
   LOC   CFA      ra
0016b584 r1+0     u
0016b588 r1+4     u
0016b590 r1+4     c-4
0016b5d4 r1+4     c-4
0016b60c r1+4     c-4
0016b630 r1+4     c-4
0016b658 r1+0     u

GLIBC

GLIBC provides pthreads which when used with C++ needs to support exception handling. The main place exceptions are used with pthreads is when cancelling threads. When using pthread_cancel a cancel signal is sent to the target thread using tgkill which causes an exception.

This is implemented with the below APIs.

Let’s look at pd->cancel_jmp_buf in more details. The cancel_jmp_buf is setup during pthread_create after clone in start_thread. It uses the setjmp and longjump non local goto mechanism.

Let’s look at some diagrams.

Pthread Normal

The above diagram shows a pthread that exits normally. During the Start phase of the thread setjmp will create the cancel_jmp_buf. After the thread routine exits it returns to the start_thread routine to do cleanup. The cancel_jmp_buf is not used.

Pthread Signalled

The above diagram shows a pthread that is cancelled. When the thread is created setjmp will create the cancel_jmp_buf. In this case while the thread routine is running it is cancelled, the unwinder runs and at the end it calls unwind_stop which calls longjmp. After the longjmp the thread is returned to start_thread to do cleanup.

A highly redacted version of our start_thread and unwind_stop functions is shown below.

start_thread()
{
  struct pthread *pd = START_THREAD_SELF;
  ...
  struct pthread_unwind_buf unwind_buf;

  int not_first_call;
  not_first_call = setjmp ((struct __jmp_buf_tag *) unwind_buf.cancel_jmp_buf);
  ...
  if (__glibc_likely (! not_first_call))
    {
      /* Store the new cleanup handler info.  */
      THREAD_SETMEM (pd, cleanup_jmp_buf, &unwind_buf);
      ...

      /* Run the user provided thread routine */
      ret = pd->start_routine (pd->arg);
      THREAD_SETMEM (pd, result, ret);
    }
  ... free resources ...
  __exit_thread ();
}

unwind_stop (_Unwind_Action actions,
	     struct _Unwind_Context *context, void *stop_parameter)
{
  struct pthread_unwind_buf *buf = stop_parameter;
  struct pthread *self = THREAD_SELF;
  int do_longjump = 0;
  ...

  if ((actions & _UA_END_OF_STACK)
      || ... )
    do_longjump = 1;

  ...
  /* If we are at the end, go back start_thread for cleanup */
  if (do_longjump)
    __libc_unwind_longjmp ((struct __jmp_buf_tag *) buf->cancel_jmp_buf, 1);

  return _URC_NO_REASON;
}

GCC

GCC provides the exception handling and unwinding capabilities to the C++ runtime. They are provided in the libgcc_s.so and libstdc++.so.6 libraries.

The libgcc_s.so library implements the IA-64 Itanium Exception Handling ABI. It’s interesting that the now defunct Itanium architecture introduced this ABI which is now the standard for all processor exception handling. There are two main entry points for the unwinder are:

There are also two data structures to be aware of:

The _Unwind_Context important parts:

struct _Unwind_Context {
  _Unwind_Context_Reg_Val reg[__LIBGCC_DWARF_FRAME_REGISTERS__+1];
  void *cfa;
  void *ra;
  struct dwarf_eh_bases bases;
  _Unwind_Word flags;
};

The _Unwind_FrameState important parts:

typedef struct {
  struct frame_state_reg_info { ... } regs;
  void *pc;

  /* The information we care about from the CIE/FDE.  */
  _Unwind_Personality_Fn personality;
  _Unwind_Sword data_align;
  _Unwind_Word code_align;
  _Unwind_Word retaddr_column;
  unsigned char fde_encoding;
  unsigned char signal_frame;
  void *eh_ptr;
} _Unwind_FrameState;

These two data structures are very similar. The _Unwind_FrameState is for internal use and closely ties to the DWARF definitions of the frame. The _Unwind_Context struct is more generic and is used as an opaque structure in the public unwind api.

Forced Unwinds

Exceptions that are raised for thread cancellation use a single phase forced unwind. Code execution will not resume, but catch blocks will be run. This is why cancel exceptions must be rethrown.

Forced unwinds use the unwind_stop handler which GLIBC provides as explained in the GLIBC section above.

Normal Exceptions

For exceptions raised programmatically unwinding is very similar to the forced unwind, but there is no stop function and exception unwinding is 2 phase.

The libstdc++.so.6 library provides the C++ standard library which includes the C++ personality routine __gxx_personality_v0. The personality routine is the interface between the unwind routines and the c++ (or other language) runtime, which handles the exception handling logic for that language.

As we saw above the personality routine is executed for each stack frame. The function checks if there is a catch block that matches the exception being thrown. If there is a match, it will update the context to prepare it to jump into the catch routine and return _URC_INSTALL_CONTEXT. If there is no catch block matching it returns _URC_CONTINUE_UNWIND.

In the case of _URC_INSTALL_CONTEXT then the _Unwind_ForcedUnwind_Phase2 loop breaks and calls uw_install_context.

Unwinding through a Signal Frame

When the GCC unwinder is looping through frames the uw_frame_state_for function will search DWARF information. The DWARF lookup will fail for signal frames and a fallback mechanism is provided for each architecture to handle this. For OpenRISC Linux this is handled by or1k_fallback_frame_state. To understand how this works let’s look into the Linux kernel a bit.

A process must be context switched to kernel by either a system call, timer or other interrupt in order to receive a signal.

The Stack Frame after an Interrupt

The diagram above shows what a process stack looks like after the kernel takes over. An interrupt frame is push to the top of the stack and the pt_regs structure is filled out containing the processor state before the interrupt.

The Stack Frame in a Sig Handler

This second diagram shows what happens when a signal handler is invoked. A new special signal frame is pushed onto the stack and when the process is resumed it resumes in the signal handler. In OpenRISC the signal frame is setup by the setup_rt_frame function which is called inside of do_signal which calls handle_signal which calls setup_rt_frame.

After the signal handler routine runs we return to a special bit of code called the Trampoline. The trampoline code lives on the stack and runs sigretrun.

Now back to or1k_fallback_frame_state.

The or1k_fallback_frame_state function checks if the current frame is a signal frame by confirming the return address points to a Trampoline. If it is a trampoline it looks into the kernel saved ucontext and pt_regs find the previous user frame. Unwinding, can then continue as normal.

Debugging the Issue

Now with a good background in how unwinding works we can start to debug our test case. We can recall our hypothesis:

Hypothesis: There is a problem handling exceptions while in a syscall. There may be something broken with OpenRISC related to how we setup stack frames for syscalls that makes the unwinder fail.

With GDB we can start to debug exception handling, we can trace right to the start of the exception handling logic by setting our breakpoint at _Unwind_ForcedUnwind.

This is the stack trace we see:

#0  _Unwind_ForcedUnwind_Phase2 (exc=0x30caf658, context=0x30caeb6c, frames_p=0x30caea90) at ../../../libgcc/unwind.inc:192
#1  0x30303858 in _Unwind_ForcedUnwind (exc=0x30caf658, stop=0x30321dcc <unwind_stop>, stop_argument=0x30caeea4) at ../../../libgcc/unwind.inc:217
#2  0x30321fc0 in __GI___pthread_unwind (buf=<optimized out>) at unwind.c:121
#3  0x30312388 in __do_cancel () at pthreadP.h:313
#4  sigcancel_handler (sig=32, si=0x30caec98, ctx=<optimized out>) at nptl-init.c:162
#5  sigcancel_handler (sig=<optimized out>, si=0x30caec98, ctx=<optimized out>) at nptl-init.c:127
#6  <signal handler called>
#7  0x303266d0 in __futex_abstimed_wait_cancelable64 (futex_word=0x7ffffd78, expected=1, clockid=<optimized out>, abstime=0x0, private=<optimized out>)
    at ../sysdeps/nptl/futex-internal.c:66
#8  0x303210f8 in __new_sem_wait_slow64 (sem=0x7ffffd78, abstime=0x0, clockid=0) at sem_waitcommon.c:285
#9  0x00002884 in tf (arg=0x7ffffd78) at throw-pthread-sem.cc:35
#10 0x30314548 in start_thread (arg=<optimized out>) at pthread_create.c:463
#11 0x3043638c in __or1k_clone () from /lib/libc.so.6
Backtrace stopped: frame did not save the PC
(gdb)

In the GDB backtrack we can see it unwinds through, the signal frame, sem_wait all the way to our thread routine tf. It appears everything, is working fine. But we need to remember the backtrace we see above is from GDB’s unwinder not GCC, also it uses the .debug_info DWARF data, not .eh_frame.

To really ensure the GCC unwinder is working as expected we need to debug it walking the stack. Debugging when we unwind a signal frame can be done by placing a breakpoint on or1k_fallback_frame_state.

Debugging this code as well shows it works correctly.

#0  or1k_fallback_frame_state (context=<optimized out>, context=<optimized out>, fs=<optimized out>) at ../../../libgcc/unwind-dw2.c:1271
#1  uw_frame_state_for (context=0x30caeb6c, fs=0x30cae914) at ../../../libgcc/unwind-dw2.c:1271
#2  0x30303200 in _Unwind_ForcedUnwind_Phase2 (exc=0x30caf658, context=0x30caeb6c, frames_p=0x30caea90) at ../../../libgcc/unwind.inc:162
#3  0x30303858 in _Unwind_ForcedUnwind (exc=0x30caf658, stop=0x30321dcc <unwind_stop>, stop_argument=0x30caeea4) at ../../../libgcc/unwind.inc:217
#4  0x30321fc0 in __GI___pthread_unwind (buf=<optimized out>) at unwind.c:121
#5  0x30312388 in __do_cancel () at pthreadP.h:313
#6  sigcancel_handler (sig=32, si=0x30caec98, ctx=<optimized out>) at nptl-init.c:162
#7  sigcancel_handler (sig=<optimized out>, si=0x30caec98, ctx=<optimized out>) at nptl-init.c:127
#8  <signal handler called>
#9  0x303266d0 in __futex_abstimed_wait_cancelable64 (futex_word=0x7ffffd78,  expected=1, clockid=<optimized out>, abstime=0x0, private=<optimized out>) at ../sysdeps/nptl/futex-internal.c:66
#10 0x303210f8 in __new_sem_wait_slow64 (sem=0x7ffffd78, abstime=0x0, clockid=0) at sem_waitcommon.c:285
#11 0x00002884 in tf (arg=0x7ffffd78) at throw-pthread-sem.cc:35

Debugging when the unwinding stops can be done by setting a breakpoint on the unwind_stop function.

When debugging I was able to see that the unwinder failed when looking for the __futex_abstimed_wait_cancelable64 frame. So, this is not an issue with unwinding signal frames.

A second Hypothosis

Debugging showed that the uwinder is working correctly, and it can properly unwind through our signal frames. However, the unwinder is bailing out early before it gets to the tf frame which has the catch block we need to execute.

Hypothesis 2: There is something wrong finding DWARF info for __futex_abstimed_wait_cancelable64.

Looking at libpthread.so with readelf this function was missing completely from the .eh_frame metadata. Now we found something.

What creates the .eh_frame anyway? GCC or Binutils (Assembler). If we run GCC with the -S argument we can see GCC will output inline .cfi directives. These .cfi annotations are what gets compiled to the to .eh_frame. GCC creates the .cfi directives and the Assembler puts them into the .eh_frame section.

An example of gcc -S:

         .file   "unwind.c"
        .section        .text
        .align 4
        .type   unwind_stop, @function
unwind_stop:
.LFB83:
        .cfi_startproc
        l.addi  r1, r1, -28
        .cfi_def_cfa_offset 28
        l.sw    0(r1), r16
        l.sw    4(r1), r18
        l.sw    8(r1), r20
        l.sw    12(r1), r22
        l.sw    16(r1), r24
        l.sw    20(r1), r26
        l.sw    24(r1), r9
        .cfi_offset 16, -28
        .cfi_offset 18, -24
        .cfi_offset 20, -20
        .cfi_offset 22, -16
        .cfi_offset 24, -12
        .cfi_offset 26, -8
        .cfi_offset 9, -4
        l.or    r24, r8, r8
        l.or    r22, r10, r10
        l.lwz   r18, -1172(r10)
        l.lwz   r20, -692(r10)
        l.lwz   r17, -688(r10)
        l.add   r20, r20, r17
        l.andi  r16, r4, 16
        l.sfnei r16, 0

When looking at the glibc build I noticed the .eh_frame data for __futex_abstimed_wait_cancelable64 is missing from futex-internal.o. The one where unwinding is failing we find it was completely mising .cfi directives. Why is GCC not generating .cfi directives for this file?

        .file   "futex-internal.c"
        .section        .text
        .section        .rodata.str1.1,"aMS",@progbits,1
.LC0:
        .string "The futex facility returned an unexpected error code.\n"
        .section        .text
        .align 4
        .global __futex_abstimed_wait_cancelable64
        .type   __futex_abstimed_wait_cancelable64, @function
__futex_abstimed_wait_cancelable64:
        l.addi  r1, r1, -20
        l.sw    0(r1), r16
        l.sw    4(r1), r18
        l.sw    8(r1), r20
        l.sw    12(r1), r22
        l.sw    16(r1), r9
        l.or    r22, r3, r3
        l.or    r20, r4, r4
        l.or    r16, r6, r6
        l.sfnei r6, 0
        l.ori   r17, r0, 1
        l.cmov  r17, r17, r0
        l.sfeqi r17, 0
        l.bnf   .L14
         l.nop

Looking closer at the build line of these 2 files I see the build of futex-internal.c is missing -fexceptions.

This flag is needed to enable the eh_frame section, which is what powers C++ exceptions, the flag is needed when we are building C code which needs to support C++ exceptions.

So why is it not enabled? Is this a problem with the GLIBC build?

Looking at GLIBC the nptl/Makefile set’s -fexceptions explicitly for each c file that needs it. For example:

# The following are cancellation points.  Some of the functions can
# block and therefore temporarily enable asynchronous cancellation.
# Those must be compiled asynchronous unwind tables.
CFLAGS-pthread_testcancel.c += -fexceptions
CFLAGS-pthread_join.c += -fexceptions -fasynchronous-unwind-tables
CFLAGS-pthread_timedjoin.c += -fexceptions -fasynchronous-unwind-tables
CFLAGS-pthread_clockjoin.c += -fexceptions -fasynchronous-unwind-tables
CFLAGS-pthread_once.c += $(uses-callbacks) -fexceptions \
                        -fasynchronous-unwind-tables
CFLAGS-pthread_cond_wait.c += -fexceptions -fasynchronous-unwind-tables
CFLAGS-sem_wait.c += -fexceptions -fasynchronous-unwind-tables
CFLAGS-sem_timedwait.c += -fexceptions -fasynchronous-unwind-tables
CFLAGS-sem_clockwait.c = -fexceptions -fasynchronous-unwind-tables

It is missing such a line for futex-internal.c. The following patch and a libpthread rebuild fixes the issue!

--- a/nptl/Makefile
+++ b/nptl/Makefile
@@ -220,6 +220,7 @@ CFLAGS-pthread_cond_wait.c += -fexceptions -fasynchronous-unwind-tables
 CFLAGS-sem_wait.c += -fexceptions -fasynchronous-unwind-tables
 CFLAGS-sem_timedwait.c += -fexceptions -fasynchronous-unwind-tables
 CFLAGS-sem_clockwait.c = -fexceptions -fasynchronous-unwind-tables
+CFLAGS-futex-internal.c += -fexceptions -fasynchronous-unwind-tables

 # These are the function wrappers we have to duplicate here.
 CFLAGS-fcntl.c += -fexceptions -fasynchronous-unwind-tables

I submitted this patch to GLIBC but it turns out it was already fixed upstream a few weeks before. Doh.

Summary

I hope the investigation into debugging this C++ exception test case proved interesting. We can learn a lot about the deep internals of our tools when we have to fix bugs in them. Like most illusive bugs, in the end this was a trivial fix but required some key background knowledge.

Additional Reading

Stafford Horne
Computer Engineer

github.com/stffrdhrn
twitter.com/stffrdhrn