Some pods need to change the host system or kernel parameters in order to
perform administrative tasks, obvious examples are pods loading a kernel
module to extend the operating system functionality, or pods managing the
network.
However, there are also other cases where a compromised container may want to
load a kernel module to hide its behaviour.
In this aspect, monitoring such host system changes helps to identify pods
and containers that affect the host system.
Monitor Linux kernel modules
A kernel module is a code that can be loaded into the kernel image at runtime,
without rebooting. These modules, which can be loaded by pods and containers,
can modify the host system. The
Monitor Linux kernel modules
guide will assist you in observing such events.
1 - Monitor Linux Kernel Modules
Monitor Linux Kernel Modules operations
Monitoring kernel modules helps to identify processes that load kernel modules to add features,
to the operating system, to alter host system functionality or even hide their behaviour. This
can be used to answer the following questions:
Which process or container is changing the kernel?
Which process or container is loading or unloading kernel modules in the cuslter?
Which process or container requested a feature that triggered the kernel to automatically load a module?
Are the loaded kernel modules signed?
Monitor Loading kernel modules
Kubernetes Environments
After deploying Tetragon, use the monitor-kernel-modules tracing policy which generates ProcessKprobe events
to trace kernel module operations.
do_init_module: the function call where the module is finaly loaded.
module_arg: the kernel module information, it contains:
name: the name of the kernel module as a string.
tainted: the module tainted flags that will be applied on the kernel. In the example above, it indicates we are loading an out-of-tree module, that is unsigned module which may compromise the integrity of our system.
Monitor Kernel Modules Signature
Kernels compiled with CONFIG_MODULE_SIG option will check if the modules being loaded were cryptographically signed.
This allows to assert that:
If the module being loaded is signed, the kernel has its key and the signature verification succeeded.
The integrity of the system or the kernel was not compromised.
Note Module signing increases security by identifying malicious modules loaded into the kernel. It is also possible to
deny loading such modules if the signature verification fails.
Before going forward, deploy the test-pod into the demo-app namespace, which has its security context set to privileged.
This allows to run the demo by mountig an xfs file system inside the test-pod which requires privileges,
but will also trigger an automatic xfs module loading operation.
In addition to the process metadata from exec events, ProcessKprobe event contains the arguments of the observed call. In the above case they are:
security_kernel_module_request: the kernel security hook where modules are loaded on-demand.
string_arg: the name of the kernel module. When modules are automatically loaded, for security reasons,
the kernel prefixes the module with the name of the subsystem that requested it. In our case, it’s requested
by the file system subsystem, hence the name is fs-xfs.
2. Kernel calls modprobe to load the kernel module
The kernel will then call user space modprobe to load the kernel module.
The ProcessExec event where modprobe tries to load the xfs module.
Note Here modprobe is started in the initial Linux host namespaces, outside of the container namespaces. When kernel
modules are loaded on-demand, the kernel will spawn a user space process modprobe that finds and load the appropriate
module from the host file system. This is done on behalf of the container and since its originate from the kernel then
the inherited Linux namespaces including the file system are eventually from the host.
3. Reading the kernel module from the file system
modprobe will read the passed xfs kernel module from the host file system.
This ProcessKprobe event contains the module argument.
find_module_sections: the function call where the kernel parses the module sections.
module_arg: the kernel module information, it contains:
name: the name of the kernel module as a string.
signature_ok: a boolean value, if set to true then module signature was successfully verified by the kernel. If it is false
or missing then the signature verification was not performed or probably failed. In all cases this means the integrity of the system has been compromised. Depends on kernels compiled with CONFIG_MODULE_SIG option.
Monitor Unloading of kernel modules
Using the same monitor-kernel-modules tracing policy allows to monitor unloading of kernel modules.
The following ProcessKprobe event will be generated:
Note Please note that some kernel module rootkits hide themselves by deleting their
entries from the kernel internal module lists while continuing to run in the background.
Monitoring module load operations allows to detect such cases