Dependencies - define what you need, and nothing else

Rod Evans — Thursday July 15, 2004

I recently attended Usenix, where Bryan explained how DTrace had been used to uncover some excessive system load brought on by the behavior of one application. A member of the audience asked whether the application was uncovering a poorly implemented part of the system. Bryan responded that in such cases the system will always be analyzed to determine whether it could do better. But there comes a point, that if an application requests an expensive service, that's what it will get. Perhaps the application should be reexamined to see if it needs the service in the first place?

This observation is very applicable to the runtime linking environment. Over the years we've spent countless hours pruning the cost of ld.so.1(1), only to see little improvement materialize with real applications. Alternatively, there's no better way of reducing the overhead of servicing a particular operation, than not requesting the operation in the first place :-)

Think about the work the runtime linker has to do to load an object. It has to find the object (sometimes through a plethora of LD_LIBRARY_PATH components), load the object, and relocate it. The runtime linker then repeats this process for any dependencies of the loaded object. That's a lot of work. So why do so many applications load dependencies they don't need?

Perhaps it's sloppyness, too much Makefile cut-and-pasting, or the inheritance of global build flags. Or, perhaps the developer doesn't realize a dependency isn't required. One way to discover dependency requirements is with ldd(1) and the -u option. For example, this application, nor any of its dependencies, make reference to libmd5.so.1:

    % ldd -u -r app
    ...
    unused object=/lib/libmd5.so.1

Note the use of the -r option. We want to force ldd(1) to bind all relocations, data and functions. However, here we're wastfully loading libmd5.so.1. This should be removed as a dependency.

The -u option uncovers totally unused objects, but there can still be wasteful references. For example, the same application reveals that a number of objects have wasteful dependencies:

    % ldd -U -r app
    ...
    unreferenced object=/usr/openwin/lib/libX11.so.4; unused dependency of app
    unreferenced object=/usr/openwin/lib/libXt.so.4; unused dependency of app
    unreferenced object=/lib/libw.so.1; unused dependency of app

Although the X libraries are used by some dependency within the process, they're not referenced by the application. There are data structures maintained by the runtime linker that track dependencies. If a dependency isn't required, we've wasted time creating these data structures. Also, should the object that requires the X libraries be redelivered in a form that no longer needs the X libraries, the application is still going to cause them to be wastefully loaded.

To reduce system overhead, only record those dependencies you need, and nothing else. As part of building the core OS, we run scripts that perform actions such as ldd(1) -U in an attempt to prevent unnecessary dependency loading from creeping in.

Note, you can also observe unused object processing using the runtime linkers debugging capabilities (LD_DEBUG=unused). Or, you can uncover unused objects during a link-edit using the same debugging technique (LD_OPTIONS=-Dunused). Another way of pruning unwanted dependencies is to use the -z ignore option of ld(1) when building your application or shared object.

Published Elsewhere

[2] LD_LIBRARY_PATH - just say no

Blog Index (rie)

[4] Linker Alien Spotting