Many powerusers are used to exploring and operating their linux system through basic utilities like ls, find or cat. But what happens when these programs seem to function properly, but their output cannot be trusted anymore?
Dynamically linked executables
The vast majority of software on linux is linked dynamically, which means they do not include most dependencies / libraries, instead expecting the system executing them to have it locally available. Sharing libraries between programs like this has two advantages: Performance and security. By sharing a single library between multiple programs, each of these programs does not need to bring its own copy of it, meaning less copies on disk and in memory at runtime. Secondly, if one of these dependencies needs to patch a bug or vulnerability, it is enough to replace the shared library - no need to update every single program using it.
Preloading dynamic libraries
A much lesser known advantage of dynamic linking on linux systems is the ability to alter the behavior of the linked library without changing the library file itself, by preloading another library that overrides specific functions. This has many uses, for example to debug specific behavior, benchmarking functions or patching a function for only a set of programs, without enforcing the change in all software using the library.
Let's start with a simple program:
main.c
#include <stdio.h>
#include <math.h>
int main() {
double value = 7.5;
double result = round(value);
printf("Rounding %.2f results in %.2f\n", result);
return 0;
}
And compile it:
gcc -o main main.c -lm
Running it shows the expected output:
Rounding 7.50 results in 8.00
Now let's write a library to preload:
preload_math.c
#include <stdio.h>
double round(double n){
printf("Math lib is on strike today\n");
return 0.0;
}
Note how the round() function fingerprint (arguments and return value) matches the one from math.h, this is important to override it - any mismatch may break the main program.
Compile the shared library:
gcc -shared -fPIC -o preload_math.so preload_math.c -lm
Running the compiled main executable still returns the same output, until we add the LD_PRELOAD environment variable:
LD_PRELOAD=./preload_math ./main
Now the output instead uses our patched round() function:
Math lib is on strike today
Rounding 7.50 results in 0.00
As you can see, using dynamic library preloading can completely change the internal behavior of any function call. Internally, the ld program invoked when executing dynamically compiled programs, will load the library passed in through the LD_PRELOAD environment variable before including any other imported library, even before the C standard library itself.
Dealing with dynamic linking side effects
When the C runtime resolves symbols like function names, it stops searching as soon as it finds the first definition - the one from the preloaded library - and uses that. You can still access the original (or any other) version by invoking dlsym() (dynamic linking symbol) from the dlfcn.h (dynamic linking functions) library, passing in RTLD_NEXT to instruct the runtime dynamic linker to find and use the next definition of a round() function:
preload_math.c
#include <stdio.h>
#include <math.h>
#include <dlfcn.h>
double round(double x) {
double (*real_round)(double) = dlsym(RTLD_NEXT, "round");
double result = real_round(x);
return result + 1.0;
}
This time, the replacement round() function first calls dlsym() to retrieve the real round() function from math.h. It then calls the real rounding function on the passed argument, but returns result + 1.0 at the end, slightly skewing every rounding operation upwards by 1.
Don't forget to add libdl to the compile flags:
gcc -shared -fPIC -o preload_math.so preload_math.c -lm -ldl
Now when running the program with the preloaded library:
LD_PRELOAD=./preload_math ./main
The program contains no suspicious output anymore, but the result is slightly off:
Rounding 7.50 results in 9.00
By invoking the original function, an attacker does not even need to know what exactly it does, or adjust their payload when the original function gets updated - they can simply wrap the original function, but attach other code to run before or after it. This strategy effectively mitigates all noticeable differences between the original and the patched function. Well, almost all.
A last difference could still be observed on functions that are invoked a lot: speed. Resolving a function at runtime is slow (at least compared to invoking a resolved one), so resolving the real round() function on every invocation could significantly slow the runtime performance of the original program, especially if it was used in a loop.
But using lazy loading, most of this side effect can be negated as well:
preload_math.c
#include <stdio.h>
#include <math.h>
#include <dlfcn.h>
double (*real_round)(double) = NULL
double round(double x) {
if (!real_round) {
real_round = dlsym(RTLD_NEXT, "round");
}
double result = real_round(x);
return result + 1.0;
}
The real_round variable is initialized to NULL, because the actual round() function isn't available when the preloaded library is compiled. The first time our patched round() function is invoked, it will set the real_round variable to the pointer of the round() function from math.h, skipping this step in later runs.
Lazy loading limits the performance penalty to a single invocation per function, a tradeoff unnoticeable for most programs.
Silently enabling dynamic library preloading for all programs
When thinking about it for a moment, having the ability to rewrite standard library functions for practically any program on the linux machine sounds extremely powerful, it only comes with a slight problem: programs need to be start with the LD_PRELOAD environment variable set.
Unfortunately, there are plenty of files where an attacker can set this variable, for example:
~/.bashrc~/.bash_profile~/.profile
And that's just the list that doesn't require root privileges! Setting this up for a single user is no problem, but if the malware installer has root privileges, it gets even worse. Adding the path of the preload library to /etc/ld.so.preload will quietly preload the shared library in every dynamically linked program for every user on the machine, as would adding LD_PRELOAD= into /etc/environment.
Hiding information from the user
The primary purpose of using dynamic preload libraries within malware is to intercept the standard library functions and change their behavior. This isn't necessarily interesting for infection or privilege escalation, but for evading detection in the post-exploitation phase.
Say a malicious software installed a trojan to record keyboard inputs and make screenshots every few seconds: That process would need to keep some file (executable) on disk, make network requests and would be visible in process / task managers. A savvy user or admin would quickly spot it and figure out something isn't right.
But how do tools like ls, ps or netstat retrieve information about the system? By reading files about processes in /proc, devices in /dev or the system in /sys. As it happens, these programs are all written in C, and use the standard library to open and read these file contents. Because of this, a preloaded library could easily wrap the getenv() function and remove the LD_REPLOAD environment variable from it, so even if the user ran env in their terminal, it wouldn't show up (but all the others remain intact). Similarly, wrapping readdir() could easily hide the existence of the file /etc/ls.so.preload.
They could further wrap the readdir() function to hide the malware's directory and contents, by quietly stripping any path beginning with /malware out of the results, so ls and find couldn't be used to find them anymore.
On a deeper level, they could change functions like open(), fopen(), fgets() or read() to pretend that the process-specific files for the malware in /proc/<process-id> don't exist, effectively hiding the process form the output of ps and other task managers. They could even strip lines out of /proc/net/tcp to hide the network traffic it causes when sending keystrokes or pings to a command server.
While this sounds simple, there are a lot of loose ends to tie up to make this cover perfect, like fixing device and networking statistics (bytes sent/received), hardware usage (cpu/memory), disk utilization (used space, bandwidth utilization), ... - the list goes on. It is possible, but most attacker don't go the extra mile, and any new linux patch could add a new endpoint to doctor, or change the output format of an existing one.
It would also need to change the output of some dynamic files like /proc/self/maps, because that would include the preloaded library for every program ran by the user (or on the system, depending on persistence config).
Detection
Using preloaded libraries as an evasion technique mostly targets the trust of the administrator, specifically the trust they are used to putting into their toolbelt and classic utilities like ls, ps, netstat etc. The technique only affects dynamically compiled executables written in C/C++, but the blast radius of that is quite large: Anything from most linux system utilities, language interpreters (bash, php, nodejs, perl, ...) and even security-centric tools like clamav.
That said, the evasion itself doesn't actually change anything on disk or the OS itself, and kernel calls still remain fully functioning. The easiest way to check if you are running a preload config is to add the LD_TRACE_LOADED_OBJECTS=1 environment variable when running a program:
LD_TRACE_LOADED_OBJECTS=1 /usr/bin/cat
This returns a list of libraries that the /usr/bin/cat utility would load, in order:
linux-vdso.so.1 (0x00007ffcdd724000) /usr/local/lib/rm_protect/rm_protect.so (0x00007fd015ec1000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd015cdb000) /lib64/ld-linux-x86-64.so.2 (0x00007fd015ec9000)
A dynamically linked library will almost always include the vdso library (think of this as a virtual library to interact with the kernel) and libc (the C runtime itself), followed by any other shared libraries. Look through the list and try to spot any unusual shared libraries (notice the rm_protect.so in the sample output?), or compare the list to the ones used by a known good version of the same command from a different device.
(Note: do not use ldd to check for these libraries, because it could be manipulated by a preloaded library itself! Always pass the LD_TRACE_LOADED_OBJECTS=1 environment variable instead! The variable is interpreted by the linker and short-circuits the loading process before any other library is loaded, so not even a preloaded library can interfere with this approach)
Recovery and cleanup
You have detected the preloaded library, and know which file is loaded, but now what?
The best course of action might sound extreme, but hear me out: Pull the plug. Seriously, unplug the system's power cord at runtime. You may lose some unsaved data in the process, but even commands like poweroff or shutdown are dynamically linked and could do any number of things instead of shutting down the system. There have been occurences where servers got their harddrives encrypted when shutting down or restarting, you don't want to experience that. Unplug the system, take out the harddrive and plug it into a different PC, then investigate the contents from there.
If you absolutely have to work on a system infected with a preloaded library, you should stay away from normal system utilities. You simply do not know if rm removes the shared library - it could instead decide to encrypt your harddrive, hide deeper within the system or do any number of nasty things instead. You should only use statically compiled programs to interact with the system to avoid the preloaded library injection (no dynamic linking means no preloaded dynamic library). The easiest way is to install busybox, as it provides you with a shell and many builtin tools like ls or rm, but is entirely statically compiled, so they all remain unaffected. From within a busybox session, you can then explore the disk and remove the library.
But remember: dynamic library preloading isn't an exploit on its own, it is only used to evade detection of the real malicious software. Even if you remove the preloaded library, there is a very high chance something far more dangerous is still running on your device, just a little less hidden. As a precaution, you should seriously consider fully reinstalling the entire system and only moving necessary data over to the new system, to be sure any infection is actually fully removed.