If you enjoy this article, subscribe (via RSS or e-mail) and follow me on twitter.
The intention of this post is to highlight some of the similarities and differences between
Mach-O dynamic linking that I encountered while building memprof.
I hope to write more posts about similarities and differences in other aspects of Mach-O and ELF that I stumbled across to shed some light on what goes on down there and provide (in some cases) the only documentation.
Procedure Linkage Table
The procedure linkage table (PLT) is used to determine the absolute address of a function at runtime. Both Mach-O and ELF objects have PLTs that are generated at compile time. The initial table simply invokes the dynamic linker which finds the symbol you want. The way this works is very similar at a high level in ELF and Mach-O, but there are some implementation differences that I thought were worth mentioning.
Mach-O PLT arrangement
Mach-O objects have several different sections across different segments that are all involved to create a PLT entry for a specific symbol.
Consider the following assembly stub which calls out to the PLT entry for
# MACH-O calling a PLT entry (ELF is nearly identical) 0x000000010008c504 [str_new+52]: callq 0x10009ebbc [dyld_stub_malloc]
dyld_stub prefix is added by GDB to let the user know that the
callq instruction is calling a PLT entry and not
malloc itself. The address
0x10009ebbc is the first instruction of
malloc‘s PLT entry in this Mach-O object. In Mach-O terminology, the instruction at
0x10009ebbc is called a symbol stub. Symbol stubs in Mach-O objects are found in the
__TEXT segment in the
Let’s examine some instructions at the symbol stub address above:
# MACH-O "symbol stubs" for malloc and other functions 0x10009ebbc [dyld_stub_malloc]: jmpq *0x3ae46(%rip) # 0x1000d9a08 0x10009ebc2 [dyld_stub_realloc]: jmpq *0x3ae48(%rip) # 0x1000d9a10 0x10009ebc8 [dyld_stub_seekdir$INODE64]: jmpq *0x3ae4c(%rip) # 0x1000d9a20 . . . .
Each Mach-O symbol stub is just a single
jmpq instruction. That
jmpq instruction either:
- Invokes the dynamic linker to find the symbol and transfer execution there
- Transfers execution directly to the function.
via an entry in a table.
In the example above, GDB is telling us that the address of the table entry for
0x1000d9a08. This table entry is stored in a section called the
__la_symbol_ptr within the
Before malloc has been resolved, the address in that table entry points to a helper function which (eventually) invokes the dynamic linker to find
malloc and fill in its address in the table entry.
Let’s take a look at what a few entries of the helper functions look like:
# MACH-O stub helpers 0x1000a08d4 [stub helpers+6986]: pushq $0x3b73 0x1000a08d9 [stub helpers+6991]: jmpq 0x10009ed8a [stub helpers] 0x1000a08de [stub helpers+6996]: pushq $0x3b88 0x1000a08e3 [stub helpers+7001]: jmpq 0x10009ed8a [stub helpers] 0x1000a08e8 [stub helpers+7006]: pushq $0x3b9e 0x1000a08ed [stub helpers+7011]: jmpq 0x10009ed8a [stub helpers] . . . .
Each symbol that has a PLT entry has 2 instructions above; a pair of
jmpq. This instruction sequence sets an ID for the desired function and then invokes the dynamic linker. The dynamic linker looks up this ID so it knows which function it should be looking for.
ELF PLT arrangement
ELF objects have the same mechanism, but organize each PLT entry into chunks instead of splicing them out across different sections. Let’s take a look at a PLT entry for malloc in an ELF object:
# ELF complete PLT entry for malloc 0x40f3d0 [malloc@plt]: jmpq *0x2c91fa(%rip) # 0x6d85d0 0x40f3d6 [malloc@plt+6]: pushq $0x2f 0x40f3db [malloc@plt+11]: jmpq 0x40f0d0 . . . .
Much like a Mach-O object, an ELF object uses a table entry to direct the flow of execution to either invoke the dynamic linker or transfer directly to the desired function if it has already been resolved.
Two differences to point out here:
- ELF puts the entire PLT entry together in nicely named section called
pltinstead of splicing it out across multiple sections.
- The table entries indirected through with the initial
jmpqinstruction are stored in a section named:
Both invoke an assembly trampoline…
Both Mach-O and ELF objects are set up to invoke the runtime dynamic linker. Both need an assembly trampoline to bridge the gap between the application and the linker. On 64bit Intel based systems, linkers in both systems must comply to the same Application Binary Interace (ABI).
Both trampolines ensure that the program stack is 16-byte aligned to comply with the amd64 ABI’s calling convention. Both trampolines also take care to save the “general purpose” caller-saved registers prior to invoking the dynamic link, but it turns out that the trampoline in Linux does not save or restore the SSE registers. It turns out that this “shouldn’t” matter, so long as glibc takes care not to use any of those registers in the dynamic linker. OSX takes a more conservative approach and saves and restores the SSE registers before and after calling out the dynamic linker.
I’ve included a snippet from the two trampolines below and some comments so you can see the differences up close.
Different trampolines for the same ABI
The OSX trampoline:
dyld_stub_binder: pushq %rbp movq %rsp,%rbp subq $STACK_SIZE,%rsp # at this point stack is 16-byte aligned because two meta-parameters where pushed movq %rdi,RDI_SAVE(%rsp) # save registers that might be used as parameters movq %rsi,RSI_SAVE(%rsp) movq %rdx,RDX_SAVE(%rsp) movq %rcx,RCX_SAVE(%rsp) movq %r8,R8_SAVE(%rsp) movq %r9,R9_SAVE(%rsp) movq %rax,RAX_SAVE(%rsp) movdqa %xmm0,XMMM0_SAVE(%rsp) movdqa %xmm1,XMMM1_SAVE(%rsp) movdqa %xmm2,XMMM2_SAVE(%rsp) movdqa %xmm3,XMMM3_SAVE(%rsp) movdqa %xmm4,XMMM4_SAVE(%rsp) movdqa %xmm5,XMMM5_SAVE(%rsp) movdqa %xmm6,XMMM6_SAVE(%rsp) movdqa %xmm7,XMMM7_SAVE(%rsp) movq MH_PARAM_BP(%rbp),%rdi # call fastBindLazySymbol(loadercache, lazyinfo) movq LP_PARAM_BP(%rbp),%rsi call __Z21_dyld_fast_stub_entryPvl
The OSX trampoline saves all the caller saved registers as well as the the
%xmm0 - %xmm7 registers prior to invoking the dynamic linker with that last call instruction. These registers are all restored after the call instruction, but I left that out for the sake of brevity.
The Linux trampoline:
subq $56,%rsp cfi_adjust_cfa_offset(72) # Incorporate PLT movq %rax,(%rsp) # Preserve registers otherwise clobbered. movq %rcx, 8(%rsp) movq %rdx, 16(%rsp) movq %rsi, 24(%rsp) movq %rdi, 32(%rsp) movq %r8, 40(%rsp) movq %r9, 48(%rsp) movq 64(%rsp), %rsi # Copy args pushed by PLT in register. movq %rsi, %r11 # Multiply by 24 addq %r11, %rsi addq %r11, %rsi shlq $3, %rsi movq 56(%rsp), %rdi # %rdi: link_map, %rsi: reloc_offset call _dl_fixup # Call resolver.
The Linux trampoline doesn’t touch the SSE registers because it assumes that the dynamic linker will not modify them thus avoiding a save and restore.
- Tracing program execution from call site to the dynamic linker is pretty interesting and there is a lot to learn along the way.
- glibc not saving and restoring
%xmm0-%xmm7kind of scares me, but there is a unit test included that disassembles the built ld.so searching it to make sure that those registers are never touched. It is still a bit frightening.
- Stay tuned for more posts explaining other interesting similarities and differences between Mach-O and ELF coming soon.