What Is .SUNW_ldynsym?

Ali Bahrami — Wednesday February 07, 2007

Surfing with the Linker-Aliens

Solaris ELF files have a new ELF symbol table. The section type is SHT_SUNW_LDYNSYM, and the section is named .SUNW_ldynsym. In the 20+ years in which the ELF standard has been in use, we have only needed two symbol tables (.symtab, and .dynsym) to support linking, so the addition of a third symbol table is a notable event for ELF cognoscenti. Even if you aren't one of those, you may encounter these sections, and wonder what they are for. I hope to explain that here.

Solaris has many tools that examine running processes or core files and generate stack traces. For example, consider the following call to pstack(1), made on an Xterm process currently running on my system:

% pstack 3094
3094:   xterm -ls -geometry 80x51+0+175
 fef4bea7 pollsys  (8046600, 2, 0, 0)
 fef0767e pselect  (5, 8400168, 84001e8, fef95260, 0, 0) + 19e
 fef0798e select   (5, 8400168, 84001e8, 0, 0) + 7e
 0805b250 in_put   (10, 8416720, 0, fedd561e, 8416720, 0) + 1b0
 08059b20 VTparse  (84166a8, 8057acc, fed387c5, 8416720, 84166a8, 804688c) + 90
 0805d1f1 VTRun    (8046a28, 8046870, feffa7c0, 8046808, 8046858, 804685c) + 205
 08057add main     (0, 80468b4, 80468c8) + 945
 08056eee _start   (4, 8046a90, 0, 8046a9a, 8046aa4, 0) + 7a

In order to show you those function names, pstack (really the libproc library used by pstack) needs to map the addresses of functions on the stack to the ELF symbols that correspond to them. Usually, these symbols come from the symbol table (.symtab). If this symbol table has been removed with the strip(1) program, then the dynamic symbol table (.dynsym) will be used instead. As described in a previous blog entry, the .dynsym contains the subset of global symbols from .symtab that are needed by the runtime linker ld.so.1(1). This fallback allows us to map global functions to their names, but local function symbols are not available. Observability tools like pstack(1) will display the hexidecimal address of such local functions when a name is not available. This is better than nothing, but is not particularly helpful.

It used to be common practice for system binaries to be stripped in order to save space. However, observability is a central tenet of the Solaris philosophy. Solaris objects and executables are therefore shipped in unstripped form, and have been for many years, in order to support such symbol lookups. For the most part, this has been a winning strategy, but there are still issues that come up from time to time:

strip(1) removes much more than the symbol table. Usually the size of this extra data is not a significant concern, but there are certain very large programs where the space savings might be worthwhile. It would be great to strip those particular things, but losing the local function symbols and the ability to make accurate stack traces is a bitter pill to swallow. This has led to a number of proposed features to "strip everything except local function symbols". These ideas are reasonable, but complicated. We like the fact that "strip" is a simple straightforward operation, and want to avoid complicating the concept.
We don't strip our files, but many Solaris users do. This becomes a problem when those applications misbehave, and they (or we, if you have a high end support contract) are trying to figure out why. Often, it is not possible to rebuild such applications in order to debug them. The ability to observe unmodified applications running in a production environment is another key Solaris virtue, as exemplified by DTrace.

Over the years, we have observed that these problems would be largely solved if we could add local function symbols to the .dynsym, and that in most programs, the additional space used would be minimal. Last fall, I embarked on a project to do this.

I tried hard to avoid adding a new symbol table type, and instead tried several experiments in which the additional local function symbols were placed in the dynsym. The reason for wanting this was to avoid having to modify ELF utilities and debuggers to know about a new symbol table. If the added symbols are in the existing .dynsym, those tools will automatically see them, without needing modification. As detailed in the ARC case that I filed for this work (PSARC/2006/526), I tried many different permutations. In every case, I discovered undesirable backward compatibility issues that kept me from using that solution. It turns out that the layout of .dynsym, and the other ELF sections that interact with it, are completely constrained, and there is no 100% backward compatible way to add local symbols to it.

ELF was designed from the very beginning to make it possible to introduce new section types with full backward/forward compatibility. You can always safely add a new section, with a moderate amount of care, and it will work. More than anything, this ability to extend ELF accounts for its long life. Given that the .dynsym cannot be extended with local symbols, I made the obvious (in hindsight) decision to to introduce a new section type (SHT_SUNW_LDYNSYM), and add a new symbol table section named .SUNW_ldynsym to every Solaris file that has a .dynsym section. Once that decision was made, the implementation was straightforward, giving me confidence that it was the right way to go.

The .SUNW_ldynsym section can be thought of as the local initial part of the .dynsym that we wish to build, but can't. The Solaris linker ( ld(1)) takes care to actually place them side by side, so that the end of the .SUNW_ldynsym section leads directly into the start of the .dynsym section. The runtime linker ( ld.so.1(1)) takes advantage of this to treat them as a single table within the implementation of dladdr(3C). Note that this trick works for applications that mmap(2) the file and access it directly. If you are accessing an ELF file via libelf, as many utilities do, you can't make any assumptions about the relative positions of different sections.

As with .dynsym, .SUNW_ldynsym sections are allocable, meaning that they are part of the process text segment. This means that they are available at runtime for dladdr(3C). It also means that they cannot be stripped. Although you cannot strip .SUNW_ldynsym sections, you can prevent them from being generated by ld(1), by using the -znoldynsym linker option.

.SUNW_ldynsym sections consume a small amount of additional space. We found that for all of core Solaris (OS and Networking), the increase in size was on the order of 1.4%. This small increase pays off by letting our observability tools do a better job. Furthermore, the presence of .SUNW_ldynsym means that in many cases, you can strip programs that you might not have been willing to strip before.

Example

Let's use the following program to see how .SUNW_ldynsym sections improve Solaris observability of local functions:

/*
 * Program to demonstrate SHT_SUNW_LDYNSYM sections. The
 * global main program calls a local function named
 * static_func(). static_func() uses printstack() to exercise
 * the dladdr(3C) function provided by the runtime linker,
 * and then deliberately causes a segfault. The resulting core
 * file can be examined by pstack(1) or mdb(1).
 *
 * In all these cases, if a stripped binary of this program
 * contains a .SUNW_ldynsym section, the static_func() function
 * will be observable by name, and otherwise simply as an
 * address.
 */


#include <ucontext.h>

static void
static_func(void)
{
	/* Use dladdr(3C) to print a call stack */
	printstack(1);

	/*
	 * Write to address 0, killing the process and
	 * producing a core file.
	 */
	*((char *) 0) = 1;
}


int main(int argc, char *argv[])
{
	static_func();
	return (0);
}

Let's build two versions of this program, one containing the .SUNW_ldynsym section, and one without:

% cc -Wl,-znoldynsym test.c -o test_noldynsym
% cc test.c -o test_ldynsym

The elfdump(1) command can be used to let us examine the three symbol tables contained in test_ldynsym. There is no need to examine this (large) output too carefully, but there are some interesting facts worth noticing:

Every symbol in .SUNW_ldynsym or .dynsym is also found in .symtab, because .symtab is a superset of the other two tables. This is why it is always preferred to the other two, when available.
.symtab is much larger than the other two tables combined, which leads to the temptation to strip it, along with the other things strip(1) removes.
The symbols in .dynsym are strictly limited to those needed by the runtime linker.
If you consider the .SUNW_ldynsym and .dynsym symbol tables as a single logical entity, you can see that the result follows the rules for ELF symbol table layout.

% elfdump -s test_ldynsym

Symbol Table Section:  .SUNW_ldynsym
     index    value      size      type bind oth ver shndx          name
       [0]  0x00000000 0x00000000  NOTY LOCL  D    0 UNDEF          
       [1]  0x00000000 0x00000000  FILE LOCL  D    0 ABS            test_ldynsym
       [2]  0x00000000 0x00000000  FILE LOCL  D    0 ABS            crti.s
       [3]  0x00000000 0x00000000  FILE LOCL  D    0 ABS            crt1.o
       [4]  0x00000000 0x00000000  FILE LOCL  D    0 ABS            crt1.s
       [5]  0x00000000 0x00000000  FILE LOCL  D    0 ABS            fsr.s
       [6]  0x00000000 0x00000000  FILE LOCL  D    0 ABS            values-Xa.c
       [7]  0x00000000 0x00000000  FILE LOCL  D    0 ABS            test.c
       [8]  0x080507f0 0x00000019  FUNC LOCL  D    0 .text          static_func
       [9]  0x00000000 0x00000000  FILE LOCL  D    0 ABS            crtn.s

Symbol Table Section:  .dynsym
     index    value      size      type bind oth ver shndx          name
       [0]  0x00000000 0x00000000  NOTY LOCL  D    0 UNDEF          
       [1]  0x08050668 0x00000000  OBJT GLOB  D    0 .plt           _PROCEDURE_LINKAGE_TABLE_
       [2]  0x08060974 0x00000004  OBJT WEAK  D    0 .data          environ
       [3]  0x0806088c 0x00000000  OBJT GLOB  D    0 .dynamic       _DYNAMIC
       [4]  0x080609c0 0x00000000  OBJT GLOB  D    0 .bssf          _edata
       [5]  0x08060990 0x00000004  OBJT GLOB  D    0 .data          ___Argv
       [6]  0x08050868 0x00000000  OBJT GLOB  D    0 .rodata        _etext
       [7]  0x0805082c 0x0000001b  FUNC GLOB  D    0 .init          _init
       [8]  0x00000000 0x00000000  NOTY GLOB  D    0 ABS            __fsr_init_value
       [9]  0x08050810 0x00000019  FUNC GLOB  D    0 .text          main
      [10]  0x08060974 0x00000004  OBJT GLOB  D    0 .data          _environ
      [11]  0x08060868 0x00000000  OBJT GLOB  P    0 .got           _GLOBAL_OFFSET_TABLE_
      [12]  0x080506b8 0x00000000  FUNC GLOB  D    0 UNDEF          printstack
      [13]  0x080506a8 0x00000000  FUNC GLOB  D    0 UNDEF          _exit
      [14]  0x08050864 0x00000004  OBJT GLOB  D    0 .rodata        _lib_version
      [15]  0x08050698 0x00000000  FUNC GLOB  D    0 UNDEF          atexit
      [16]  0x08050678 0x00000000  FUNC GLOB  D    0 UNDEF          __fpstart
      [17]  0x0805076c 0x0000007b  FUNC GLOB  D    0 .text          __fsr
      [18]  0x08050688 0x00000000  FUNC GLOB  D    0 UNDEF          exit
      [19]  0x080506c8 0x00000000  FUNC WEAK  D    0 UNDEF          _get_exit_frame_monitor
      [20]  0x080609c0 0x00000000  OBJT GLOB  D    0 .bss           _end
      [21]  0x080506e0 0x0000008b  FUNC GLOB  D    0 .text          _start
      [22]  0x08050848 0x0000001b  FUNC GLOB  D    0 .fini          _fini
      [23]  0x08060978 0x00000018  OBJT GLOB  D    0 .data          __environ_lock
      [24]  0x0806099c 0x00000004  OBJT GLOB  D    0 .data          __longdouble_used
      [25]  0x00000000 0x00000000  NOTY WEAK  D    0 UNDEF          __1cG__CrunMdo_exit_code6F_v_

Symbol Table Section:  .symtab
     index    value      size      type bind oth ver shndx          name
       [0]  0x00000000 0x00000000  NOTY LOCL  D    0 UNDEF          
       [1]  0x00000000 0x00000000  FILE LOCL  D    0 ABS            test_ldynsym
       [2]  0x080500f4 0x00000000  SECT LOCL  D    0 .interp        
       [3]  0x08050108 0x00000000  SECT LOCL  D    0 .SUNW_cap      
       [4]  0x08050118 0x00000000  SECT LOCL  D    0 .hash          
       [5]  0x080501fc 0x00000000  SECT LOCL  D    0 .SUNW_ldynsym  
       [6]  0x0805029c 0x00000000  SECT LOCL  D    0 .dynsym        
       [7]  0x0805043c 0x00000000  SECT LOCL  D    0 .dynstr        
       [8]  0x080505c4 0x00000000  SECT LOCL  D    0 .SUNW_version  
       [9]  0x080505f4 0x00000000  SECT LOCL  D    0 .SUNW_dynsymso 
      [10]  0x08050630 0x00000000  SECT LOCL  D    0 .rel.data      
      [11]  0x08050638 0x00000000  SECT LOCL  D    0 .rel.plt       
      [12]  0x08050668 0x00000000  SECT LOCL  D    0 .plt           
      [13]  0x080506e0 0x00000000  SECT LOCL  D    0 .text          
      [14]  0x0805082c 0x00000000  SECT LOCL  D    0 .init          
      [15]  0x08050848 0x00000000  SECT LOCL  D    0 .fini          
      [16]  0x08050864 0x00000000  SECT LOCL  D    0 .rodata        
      [17]  0x08060868 0x00000000  SECT LOCL  D    0 .got           
      [18]  0x0806088c 0x00000000  SECT LOCL  D    0 .dynamic       
      [19]  0x08060974 0x00000000  SECT LOCL  D    0 .data          
      [20]  0x080609c0 0x00000000  SECT LOCL  D    0 .bssf          
      [21]  0x080609c0 0x00000000  SECT LOCL  D    0 .bss           
      [22]  0x00000000 0x00000000  SECT LOCL  D    0 .symtab        
      [23]  0x00000000 0x00000000  SECT LOCL  D    0 .strtab        
      [24]  0x00000000 0x00000000  SECT LOCL  D    0 .comment       
      [25]  0x00000000 0x00000000  SECT LOCL  D    0 .debug_info    
      [26]  0x00000000 0x00000000  SECT LOCL  D    0 .debug_line    
      [27]  0x00000000 0x00000000  SECT LOCL  D    0 .debug_abbrev  
      [28]  0x00000000 0x00000000  SECT LOCL  D    0 .shstrtab      
      [29]  0x080609c0 0x00000000  OBJT LOCL  D    0 .bss           _END_
      [30]  0x08050000 0x00000000  OBJT LOCL  D    0 .interp        _START_
      [31]  0x00000000 0x00000000  FILE LOCL  D    0 ABS            crti.s
      [32]  0x00000000 0x00000000  FILE LOCL  D    0 ABS            crt1.o
      [33]  0x00000000 0x00000000  FILE LOCL  D    0 ABS            crt1.s
      [34]  0x08060994 0x00000004  OBJT LOCL  D    0 .data          __get_exit_frame_monitor_ptr
      [35]  0x08060998 0x00000004  OBJT LOCL  D    0 .data          __do_exit_code_ptr
      [36]  0x00000000 0x00000000  FILE LOCL  D    0 ABS            fsr.s
      [37]  0x080609a0 0x00000020  OBJT LOCL  D    0 .data          trap_table
      [38]  0x00000000 0x00000000  FILE LOCL  D    0 ABS            values-Xa.c
      [39]  0x08060974 0x00000000  NOTY LOCL  D    0 .data          Ddata.data
      [40]  0x080609c0 0x00000000  NOTY LOCL  D    0 .bss           Bbss.bss
      [41]  0x08050868 0x00000000  NOTY LOCL  D    0 .rodata        Drodata.rodata
      [42]  0x00000000 0x00000000  FILE LOCL  D    0 ABS            test.c
      [43]  0x080507f0 0x00000019  FUNC LOCL  D    0 .text          static_func
      [44]  0x080609c0 0x00000000  OBJT LOCL  D    0 .bss           Bbss.bss
      [45]  0x08060974 0x00000000  OBJT LOCL  D    0 .data          Ddata.data
      [46]  0x08050864 0x00000000  OBJT LOCL  D    0 .rodata        Drodata.rodata
      [47]  0x00000000 0x00000000  FILE LOCL  D    0 ABS            crtn.s
      [48]  0x08050668 0x00000000  OBJT GLOB  D    0 .plt           _PROCEDURE_LINKAGE_TABLE_
      [49]  0x08060974 0x00000004  OBJT WEAK  D    0 .data          environ
      [50]  0x0806088c 0x00000000  OBJT GLOB  D    0 .dynamic       _DYNAMIC
      [51]  0x080609c0 0x00000000  OBJT GLOB  D    0 .bssf          _edata
      [52]  0x08060990 0x00000004  OBJT GLOB  D    0 .data          ___Argv
      [53]  0x08050868 0x00000000  OBJT GLOB  D    0 .rodata        _etext
      [54]  0x0805082c 0x0000001b  FUNC GLOB  D    0 .init          _init
      [55]  0x00000000 0x00000000  NOTY GLOB  D    0 ABS            __fsr_init_value
      [56]  0x08050810 0x00000019  FUNC GLOB  D    0 .text          main
      [57]  0x08060974 0x00000004  OBJT GLOB  D    0 .data          _environ
      [58]  0x08060868 0x00000000  OBJT GLOB  P    0 .got           _GLOBAL_OFFSET_TABLE_
      [59]  0x080506b8 0x00000000  FUNC GLOB  D    0 UNDEF          printstack
      [60]  0x080506a8 0x00000000  FUNC GLOB  D    0 UNDEF          _exit
      [61]  0x08050864 0x00000004  OBJT GLOB  D    0 .rodata        _lib_version
      [62]  0x08050698 0x00000000  FUNC GLOB  D    0 UNDEF          atexit
      [63]  0x08050678 0x00000000  FUNC GLOB  D    0 UNDEF          __fpstart
      [64]  0x0805076c 0x0000007b  FUNC GLOB  D    0 .text          __fsr
      [65]  0x08050688 0x00000000  FUNC GLOB  D    0 UNDEF          exit
      [66]  0x080506c8 0x00000000  FUNC WEAK  D    0 UNDEF          _get_exit_frame_monitor
      [67]  0x080609c0 0x00000000  OBJT GLOB  D    0 .bss           _end
      [68]  0x080506e0 0x0000008b  FUNC GLOB  D    0 .text          _start
      [69]  0x08050848 0x0000001b  FUNC GLOB  D    0 .fini          _fini
      [70]  0x08060978 0x00000018  OBJT GLOB  D    0 .data          __environ_lock
      [71]  0x0806099c 0x00000004  OBJT GLOB  D    0 .data          __longdouble_used
      [72]  0x00000000 0x00000000  NOTY WEAK  D    0 UNDEF          __1cG__CrunMdo_exit_code6F_v_

Now, we strip the two versions of our program to remove the .symtab symbol table, and force the system to use the dynamic tables instead:

% strip test_ldynsym test_noldynsym 
% file test_ldynsym test_noldynsym 
test_ldynsym:   ELF 32-bit LSB executable 80386 Version 1, dynamically linked, stripped
test_noldynsym: ELF 32-bit LSB executable 80386 Version 1, dynamically linked, stripped

Running the version without a .SUNW_ldynsym section:

% ./test_noldynsym 
/home/ali/test/test_noldynsym:0x6ca
/home/ali/test/test_noldynsym:main+0xb
/home/ali/test/test_noldynsym:_start+0x7a
Segmentation Fault (core dumped)
% pstack core
core 'core' of 5041:    ./test_noldynsym
 080506d2 ???????? (804692c, 80467a4, 805062a, 1, 80467b0, 80467b8)
 080506eb main     (1, 80467b0, 80467b8) + b
 0805062a _start   (1, 8046994, 0, 80469a5, 80469bf, 8046a03) + 7a

Our program used the printstack(3C) function to display its own stack. Afterwards, we use the pstack command to view the same data from the core file. In both cases, the top line represents the call to the local function static_func(), a fact that we know from examining the source code, since the number and/or '????????' used to represent it are less than obvious to an external observer.

Running the version with a .SUNW_ldynsym section, the system is able to put a name to the local function:

% ./test_ldynsym 
/home/ali/test/test_ldynsym:static_func+0xa
/home/ali/test/test_ldynsym:main+0xb
/home/ali/test/test_ldynsym:_start+0x7a
Segmentation Fault (core dumped)
% pstack core
core 'core' of 5044:    ./test_ldynsym
 08050802 static_func (8046930, 80467a8, 805075a, 1, 80467b4, 80467bc) + 12
 0805081b main     (1, 80467b4, 80467bc) + b
 0805075a _start   (1, 8046998, 0, 80469a7, 80469c1, 8046a05) + 7a

Conclusions

Sometimes it is the little things that make a difference. I expect that the local dynamic symbol table will provide valuable information in difficult debugging situations where one is examining large stripped programs running in a production environment. The rest of the time, the additional data is small, and will have little or no impact on performance.

.SUNW_ldynsym sections have been part of the Solaris development (Nevada) builds since last fall, and are also available in OpenSolaris.

Surfing with the Linker-Aliens

Comments

Luke — Wednesday February 07, 2007

It's interesting that you say "the additional data is small, and will have little or no impact on performance". The GCC developers seem to have gone to some length to remove symbols from dynamic shared objects with their -fvisibility=hidden support. They claim that this option "can very substantially improve linking and load times of shared object libraries".

So do you disagree with them, or is there a key difference between your work and theirs? Are you testing plain C code while they're considering the symbol bloat in C++ code which uses templates? I suppose that you are able to ignore the .SUNW_ldynsym section when resolving undefined references during dynamic loading, so maybe its presence doesn't hurt performance so much?

Ali Bahrami — Wednesday February 07, 2007

I don't see a contradiction between the claims for GCC -fvisibility=hidden, and the Solaris .SUNW_ldynsym. The reason is as you suggest --- .SUNW_ldynsym is not used by the runtime linker for symbol resolution. Those local symbols are only there to allow dladdr() and debuggers to do a better job. This means that the cost of .SUNW_ldynsym is only the amount of address space it consumes, which is very small. It isn't free of course, but you have to evaluate these sort of issues within the bigger context --- we feel that the win in observability for real world binaries in the field is worth it. In any event, there's no additional load on the runtime linker.

The GCC -fvisibility=hidden reduces the number of symbols the runtime linker needs to examine at runtime from .dynsym, which would certainly speed up loading. We do similar things --- a big favorite is to use mapfiles to reduce the number of global symbols visible from a sharable object.

It is certainly true that C++ pushes all linker features harder than plain C. Even in that case though, I believe that the added overhead implied by .SUNW_ldynsym will be tiny in comparison to the rest of those same C++ objects, probably in a similar proportion as with C.

I think .SUNW_ldynsym is in general, a win, and without a significant performance issue. However, the ld -znoldynsym option is there, just in case.

Thanks for your interest... - Ali

DP — Thursday February 08, 2007

Hi Ali, This feature seems to be very good. Good work.

Brian Utterback — Thursday February 08, 2007

Just a nit. On a current Solaris 10 system, I find that there are 719 ELF binaries in /usr/bin, and 490 are stripped with only 229 with their symbols intact.

Ali Bahrami — Thursday February 08, 2007

Interesting! I'll stand by the underlying idea though... :-)

Your comment made me curious, so I just did the following on my desktop system (fairily recent Nevada non-debug build):

% file /usr/bin/\* | grep ', stripped' | wc -l
      51

Rather different results, and I'm not sure what accounts for it. Looking at the stripped files on this system, they are mainly non-core things, like bash, bzip, tcsh, etc.

Brian Utterback — Thursday February 08, 2007

And I stand by my nit! You said that Solaris has shipped for years with unstripped binaries, but you are looking at Nevada which hasn't shipped at all! Seriously, that appears to be the difference; I got the same results you did on a Nevada system. The question is, will it ship that way or not? Either way, the new section should give us the best of both worlds.

Ali Bahrami — Friday February 09, 2007

> And I stand by my nit!

Understandable, since you are absolutely correct!

See Which Solaris Files Are Stripped? for a better answer.

Thanks!

Surfing with the Linker-Aliens

Published Elsewhere

https://blogs.sun.com/ali/entry/what_is_sunw_ldynsym/
https://blogs.oracle.com/ali/entry/what_is_sunw_ldynsym/
https://blogs.oracle.com/ali/what-is-sunwldynsym/

Surfing with the Linker-Aliens

[4] Symbol Tables Blog Index (ali) [6] Which...Files Are Stripped?