Debugging with Library Interposition

At some point when debugging one will encounter a scenario where it is desirable to modify or observe an application’s interaction with an external library (or the system). Various techniques are available, but few are quite as straightforward as `library interposition’: the practice of overriding references to symbols from one library with another library at executable load-time.

To dramatically simplify the relevant process: when an application is being prepared to execute, the executable loader will extract from the target executable an ordered list of required symbols and an ordered list of requested libraries. The libraries are loaded in the requested order; the list of libraries is searched for the first instance of each required symbol; references to these symbols are inserted into the target application; and then the application is started.

Because the requested libraries are searched in order, it is apparent that we could inject our own library earlier in this list, replacing references to a symbol of interest. This functionality happens to be exposed by the LD_PRELOAD environment variable, which lists the libraries to prepend to the requested library list. eg, using the shell command LD_PRELOAD="mymalloc.so" ./malloc_test.

Additionally, given we started with an ordered list of libraries, it is possible to scan for the address of the next symbol of the same name. This lets us forward calls to the original target, allowing the original functionality to be maintained.

Verbose mmap failure

One of my recent debugging tasks was to discover why calls to mmap were failing under FreeBSD but not Linux. I was receiving EINVAL, but there are a number of conditions which trigger this error code.

A quick look through the man-page and an inspection of parameter values should be sufficient for on-the-spot debugging. But I know I’m likely to do something similarly stupid in the future. A more verbose alert system would allow more rigorous testing in the future.

Given mmap is typically linked from a dynamic libc, and its error conditions tend to be trivially verifiable given the provided parameters, it is a prime candidate for validating through library interposition.

I created a tiny library that prints error messages to stderr as they are detected and then forwards the call to the system for the inevitable failure. A portion of the code is replicated below.

#include <sys/mman.h>

#include <mutex>
#include <cassert>

static std::once_flag s_init_flag;

static void* (*_mmap)(void*,size_t,int,int,int,off_t);


static
void
init (void)
{
    std::call_once (s_init, [] {
        assert (!_mmap);

        fprintf (stderr, "loading mmap tracer\n");

        _mmap = reinterpret_cast<decltype(_mmap)> (dlsym (RTLD_NEXT, "mmap"));
        if (!_mmap) {
            fprintf (stderr, "mmap init: %s", dlerror ());
            abort ();
        }
    });
}


extern "C"
void*
mmap [[gnu::visibility ("default")]] (
    void *addr,
    size_t len,
    int prot,
    int flags,
    int fd,
    off_t offset
) {
    init ();

    /* non-exhaustive list of EINVAL triggers */

    // MAP_ANONYMOUS requires an invalid file descriptor, however Linux
    // will happily accept various other values (such as 0).
    if (flags & MAP_ANONYMOUS && fd != -1)
        fprintf (stderr, "%s: invalid fd for MAP_ANONYMOUS, fd = %d\n", __func__, fd);

    // MAP_PRIVATE and MAP_SHARED are exclusive flags.
    if (flags & MAP_PRIVATE && flags & MAP_SHARED)
        fprintf (stderr, "%s: both MAP_PRIVATE and MAP_SHARED were specified\n", __func__);

    // At least one known visibility flag must be present.
    if (!(flags & MAP_ANON)     &&
        !(flags & MAP_PRIVATE)  &&
        !(flags & MAP_SHARED)   &&
        !(flags & MAP_STACK))
    {
        fprintf (stderr, "%s: no visibility/usage flag was specified", __func__);
    }

    return _mmap (addr, len, prot, flags, fd, offset);
}

We provide a single externally visible function: mmap.

The first time our copy of mmap is called we ask dlopen for the address of the next highest priority instance of mmap and record it for forwarding in the future. Thankfully dlopen is specified in POSIX so it should be available under most platforms (although predictably, not under Windows).

Our implementation performs three trivial checks on the flags and file descriptor, possibly printing an error message, then hands off control to (what is probably) the system. There is no real limit as to what we could do here over-and-above printing an error message; I have used this mechanism in the past to simulate non-trivial chunks of procfs, track memory and resource allocations, and emulate unsupported ‘syscalls’.

Minuatae

We must specific linkage using extern "C" to suppress name mangling if using C++. Library interposition is most straightforward when targeting C APIs, as one doesn’t have to contend with name mangling and inlining that is common with CXX APIs; which is not to say that targeting C++ is impossible. Thankfully a good number of low level libraries are expressed in terms of C (eg. OpenGL, OpenCL, Vulkan, POSIX).

The symbol we intend to override must be exported from our library, so we use the gnu::visibility function attribute to mark it so. GCC and clang default to exposing all extern symbols publicly so this may not be necessary in some projects, but it is good practice to indicate the visibility regardless.

If I was targeting a higher level API I might consider using GCC’s gnu::constructor function attribute to automatically trigger the call to init at library load time, but in this particular instance it is possible that a call to mmap may be required while loading this library; so we must instead be ready to self-initialise. This requirement was specifically observed under FreeBSD, but is possible on all systems.

As we need to self-initialise it is then important that we protect calls to init with thread safe mechanisms due to the potential for race conditions. While it matters less in this scenario because the call is idempotent, it is better to add this feature now rather than forget it later.

If your target executable is compiled to use ASAN then libasan.so must be loaded before other libraries. Simply list libasan.so earlier in the LD_PRELOAD list than your interposition library.

Resolution

After an hour or so of rough coding I discovered, embarrassingly, that I was passing a valid file descriptor to anonymous memory mappings. Surprisingly this behaviour did not trigger an error under Linux.

But now I have the start of a nice POSIX testing and tracing library, which is a decent win.