Where Did The 32-Bit Linkers Go?

Ali Bahrami — Monday December 21, 2015

Surfing with the Linker-Aliens

In Solaris 11 Update 4 (and Solaris 11 Update 3), the 32-bit version of the link-editor, and related linking utilities, are gone. This is part of a larger effort to move Solaris to LP64, a project that Alan Coopersmith wrote about in 2014. Have no concern, your ability to user 32-bit objects on Solaris is unaffected. In fact, it's unlikely that you've actually executed the 32-bit versions of any of these commands in years (except ld, see below).

Here, I'll cover the history that underlies this change, and describe how our world has become a bit simpler.

In the beginning, it was simple. Solaris was originally a 32-bit system, and so, all the utilities were 32-bit. Then, in 1995, Solaris 7 added the ability to build and run 64-bit code, in support of the new sparcv9 architecture that debuted with the UltraSparc 1 CPU. In order to support older systems as well as the new ones, we delivered both 32 and 64-bit kernels. A 32-bit kernel can only run 32-bit code. A 64-bit kernel can run both, but for some programs, there are advantages to 64-bit code, which can access more memory, and create larger files without requiring special large file APIs.

Unlike many 64-bit systems, where 32-bit programs are run via a layer of shim translation code ("thunking"), Solaris supports both 32 and 64-bit programs on an equal footing. The Solaris kernel has independent 32 and 64-bit system call traps, and all system libraries are built in 32 and 64-bit form. This is how the current organization of our system library directories was established, with 32-bit libraries in /lib and /usr/lib and 64-bit versions found in /lib/64 and /usr/lib/64 (where '64' is either 'amd64' or 'sparcv9'). At that time, we started delivering all of the linking utilities in both 32 and 64-bit form, with the 32-bit version found in /usr/bin (or /usr/ccs/bin, in those days), and the 64-bit versions in an amd64 or sparcv9 subdirectory below.

At that time, the linker utilities were modified so that any given utility could handle either 32 or 64-bit ELF objects, and then those utilities were themselves built in both 32 and 64-bit form. Hence, the 32-bit ld can link either 32 or 64-bit objects, and the same is true of the 64-bit ld. When you ran the 32-bit version, which would normally be in your PATH, the first thing it would do was to check to see if a 64-bit kernel was running, and if a 64-bit version of itself was available, and if so, it would immediately exec() the 64-bit version on top of itself. You can see this with truss on a Solaris 10 system:

% truss -ft exec elfdump a.out > /dev/null
19739:  execve("/usr/ccs/bin/elfdump", 0xFFBFFBAC, 0xFFBFFBB8)  argc = 2
19739:  execve("/usr/ccs/bin/sparcv9/elfdump", 0xFFBFFBAC, 0xFFBFFBB8)  argc = 2

As 64-bit hardware became common, most people ended up automatically running the 64-bit linkers, albiet via their 32-bit front ends. The 32-bit versions essentially became little more than a way to call isaexec(). There was one exception to this: ld did not play the isaexec game, and so, most people continued to use the 32-bit version of ld. This exception came to pass shortly after the original 64-bit version was delivered, and I believe that it had to do with existing makefiles initially being broken by passing 32-bit support libraries to the 64-bit version of ld. Some customers, who needed the 64-bit ld to link large programs, did use it the 64-bit ld, by directly specifying a path to /usr/bin/{amd64,sparcv9}. There was also an environment variable, LD_NOEXEC_64, that would prevent the exec to the 64-bit version. I think we here at Sun were probably the main users of that. We used it to test the 32-bit linkers, to ensure that they continued to work.

That rather complex state of affairs persisted for many years. Eventually, we ended support for the 32-bit sparc kernel, with Solaris 10, and later, it was dropped for x86 with Solaris 11. Since then, it's been 64-bit, all the time. With Solaris 11 Update 4, we've taken the opportunity to lose most of the complexity that came with delivering the linkers as both 32 and 64-bit programs:

The 32-bit versions of the following utilities are gone. Note again that the 64-bit versions are all capable of handing 32-bit objects, so there is no loss of functionality: ar, crle, dump, elfcompress, elfdump, elfedit, elffile, elfwrap, ld, ldd, mcs, nm, pvs, size, strip.
The 64-bit versions of all those programs moved up to /usr/bin from their old home in /usr/bin/{amd64,sparcv9}.
A symlink for ld is left behind in /usr/bin/{amd64,sparcv9}, as it was documented common knowledge that ld could be run directly from that path:
```
% ls -alF /usr/bin/amd64/ld 
lrwxrwxrwx 1 root root 5 Jan 30 2018 /usr/bin/amd64/ld -> ../ld*
```
The runtime linker, ld.so.1, must match the ELFCLASS of the executable it is associated with, and so, both 32 and 64-bit runtime linkers are delivered with the system.
The moe command calls dlopen() to analyze objects, and must have an ELFCLASS that matches the objects being examined. moe therefore also continues to be delivered in both 32 and 64-bit forms.
It is harmless to set LD_NOEXEC_64, but it has no effect.

Surfing with the Linker-Aliens

Published Elsewhere

https://blogs.oracle.com/ali/lp64/

Surfing with the Linker-Aliens

[26] Solaris 11.4 Beta Is Available Blog Index (ali) [28] Weak Filters