Containers have become a vital feature of most modern ci/cd and production workloads, with tools like docker and incus offering increasingly more capable ecosystems. When dealing with such abstract programs and quality of life features it can be easy to forget that containers are just a combination of linux kernel primitives, so let's take this opportunity to containerize an application manually, without using any container runtime.
What containers are made of
Modern containers are mostly made of three features. The first are linux namespaces, used to separate a process from host resources and give it an isolated view of its environment, including processes, mounts and network, among others. The second integral part are linux capabilities, which allow setting fine-grained permissions to kernel features for a process individually.
Combining these kernel primitives can create an isolated environment where even executing programs as root inside is not enough to easily escape the namespace anymore.
The last feature are cgroups, a kernel feature allowing to set hardware quotas and restrictions for process groups. Applying it to the container process ensures that neither it nor any of its child processes can consume resources beyond the set limit.
Explaining the primitives is beyond the scope of this article, see man namespaces and man capabilities for more information.
Preparing the host
In order to follow the next steps, we will make a few assumptions about the host system running the commands. This article will assume you are running linux debian 13 or later, preferably in a virtual machine. You can also run it on bare metal, but some of the mounting commands can cause real damage to your system if not run in the correct sequence and context, so a vm is preferred. Do not use a container (including incus system containers!), as they will likely cause problems due to namespace nesting.
Finally, we need a few dependencies on the host:
sudo apt install util-linux debootstrap nftablesWith the host prepared, we are ready to build a container.
Root filesystem
The container will need a filesystem containing the base operating system components, core programs and configuration. We can use debootstrap to quickly generate one into a local directory:
mkdir rootfs
sudo debootstrap \
--variant=minbase \
bookworm \
rootfs \
http://deb.debian.org/debianThe resulting rootfs/ directory will contain a minimal debian installation ready to use for the container.
Real container runtimes would use a container image here and provide the base root filesystem as readonly, with an overlayfs layered on top that the container can write to. Since we are constructing a simple container for educational purposes, we stick to this simplified container storage approach.
Creating a namespace
Building a namespace for the containerized app is done by calling unshare with options enabling different layers of isolation:
sudo unshare \
--pid \
--mount \
--uts \
--ipc \
--fork \
--net \
env ROOTFS=$(realpath rootfs) bashThe command will launch a new bash session in the current terminal, but with isolated views of processes (--pid), mounts (--mount), hostname (--uts), process messaging/shared memory (--ipc) and network (--net). Using --fork is necessary to launch the program (env) inside the newly created pid namespace rather than the current one.
We call env to set the ROOTFS environment variable in the new bash session, since we need the absolute path to the previously created rootfs/ directory for mounting inside the container in the next step.
Nothing will change visually after executing the command, but it is important that further commands are executed in this bash session and not a different one to preserve the namespace isolation.
Note that we intentionally did not use --user and --map-root-user because we need to create devices and mount special filesystems inside the container later, which is difficult in an unprivileged environment. This makes our container roughly equivalent to normal OCI containers, not rootless variants.
Switching the filesystem
The bash program inside the new namespace can still see the old host root filesystem and all system directories, leaving the host exposed.
This is typically fixed by mounting a new filesystem, our case the already prepared rootfs, into the container and switching it in as the root filesystem using pivot_root.
We start by giving the container (and thus the bash prompt) a new hostname so it is visually distinct from other shells:
hostname toycontainerThen we continue by preventing new mounts inside the container from propagating back to the host, for mounting and bind mounting the rootfs/ directory into the container under /rootfs:
mount --make-rprivate /
mkdir /rootfs
mount --bind $ROOTFS /rootfsIn order to make /rootfs the root filesystem inside the container, we need to call pivot_root and give it a directory under /rootfs where the old root filesystem can be mounted, effectively switching it out:
mkdir -p /rootfs/oldroot
pivot_root /rootfs /rootfs/oldroot
cd /The previous /rootfs has become /, and the old root filesystem is now mounted at /oldroot (it was at /rootfs/oldroot, but /rootfs is now mounted at / so it is only /oldroot now).
The last part is to unmount /oldroot to make it fully inaccessible within the container. While the kernel call umount() could do that just fine, the userspace umount program needs to read /proc to work. Since the container has no /proc mounted yet, we need to create it before unmounting /oldroot:
mount -t proc proc /proc
umount -l /oldrootAnd with that, the host root filesystem is finally fully gone from the container's reach.
Special filesystems
While most programs won't touch the contents of /sys, /proc or /dev, a small subset may be necessary to have. That said, these are pseudo-directories calling kernel functions when read or written, or allow direct access to hardware.
Mounting them presents considerable attack surface and must be done carefully. We already had to mount /proc to make umount work earlier, but that is not sufficient for security. Some paths still provide way too much host information, such as /proc/kcore, which exposes system memory for debugging purposes.
Furthermore, most programs expect at least the default devices /dev/null, /dev/zero, /dev/random and /dev/urandom.
Creating a tmpfs at /dev and the four base device files inside is fairly simple:
mount -t tmpfs tmpfs /dev
mknod /dev/null c 1 3
mknod /dev/zero c 1 5
mknod /dev/random c 1 8
mknod /dev/urandom c 1 9
chmod 666 /dev/*And with access to a /dev/null device, we can hide the problematic /proc/kcore file with a bind mount:
mount --bind -o ro /dev/null /proc/kcoreFor /sys, we can mount a restricted readonly version:
mount -t sysfs sysfs /sys -o ro,nosuid,nodev,noexecNote that this setup is extremely simplified and not secure. Real container runtimes populate system directories with more entries like /dev/pts for terminal handling and mount a much more restricted version of /sys and /proc to prevent kernel features from affecting the host. Even read access to /sys may be problematic in some cases, because every access invokes kernel functions.
Running the final program without capabilities
The last step for container setup is to run the program we actually want to be containerized, but without access to any kernel capabilities that may let it escape the container environment.
We can use setpriv to run a program with a specific set of capabilities and choose the trusty bash shell as the program to containerize:
/usr/bin/setpriv \
--bounding-set=-all,+chown,+dac_override,+fowner,+fsetid,+kill,+setgid,+setuid,+setpcap,+net_bind_service,+net_raw,+sys_chroot,+mknod,+audit_write,+setfcap \
--inh-caps=-all \
--ambient-caps=-all \
--reuid=0 \
--regid=0 \
--clear-groups \
/bin/bashThis starts the /bin/bash shell as root (reuid=0/rgid=0) but with preexisting groups cleared (--clear-groups), and clears all capabilities except the ones named in the allowlist. The important part is that all capabilities-related flags start with -all to strip all previous entries, and only --bounding-set then adds common ones back to the set. Bounding capabilities define the maximum capabilities the newly launched process may ever obtain, acting as a safety fence.
The capabilities listed here were simply copied from running capsh --print inside a default docker container. You don't need to understand each of them, just be aware that advanced permissions e.g. to switch namespaces or mount filesystems are excluded so the root user inside the container cannot break out of it.
Limiting hardware resources with cgroups
In order to limit CPU or memory access for the container, you will need to configure and assign a cgroup to it.
This has to be done from outside the container, not the shell you have been using for all previous commands.
Creating a directory in /sys/fs/cgroup instructs the kernel to create a new cgroup:
sudo mkdir /sys/fs/cgroup/toycontainerThe directory will automatically be populated with pseudo-files for cgroup administration. For now, we are only interested in memory.max for a memory limit and cpu.max to limit cpu usage.
Set a 500MB memory limit:
echo "500M" | sudo tee /sys/fs/cgroup/toycontainer/memory.maxAnd allow using 50% of a single CPU core:
echo "50000 100000" | sudo tee /sys/fs/cgroup/toycontainer/cpu.maxThe syntax for CPU quotas consists of a quota and a scheduling period. The default period is 100000 representing 100ms, and setting a quota of 50000 allows using 50ms of cpu time every 100ms, effectively 50%.
Lastly, we need the pid (process ID) of the container process started by unshare. If you are following along, you likely only have one, so you can easily find it with pidof unshare. Should the command return multiple pids, you have to figure out which one you need manually by checking ps aux.
Once you found the pid, export it as a variable. In this example, we assume the container process is running as id 12345:
export CONTAINER_PID=12345Now we can add the container process to the cgroup by using the pid:
echo "$CONTAINER_PID" | sudo tee /sys/fs/cgroup/toycontainer/cgroup.procsAnd just like that, resource usage inside the container is now limited.
Enabling network access
Again, this has to be done from outside the container. We assume you already found the container pid and exported the CONTAINER_PID variable from the previous paragraph.
The most basic requirement is a network connection allowing communication with the host. We can create a veth device and put one end into the container's network namespace.
sudo ip link add toy-host type veth peer name toy-container
sudo ip link set toy-container netns "$CONTAINER_PID"Think of a veth device like a virtual ethernet cable, giving you access to both ends of the cable, in this case toy-host for the host side and toy-container moved into the container network namespace.
Now we need to configure and enable the virtual device on the host:
sudo ip addr add "10.10.10.1/24" dev toy-host
sudo ip link set toy-host upAnd also inside the container:
sudo nsenter -t "$CONTAINER_PID" -n \
ip addr add "10.10.10.99/24" dev toy-container
sudo nsenter -t "$CONTAINER_PID" -n \
ip link set toy-container upThe host is assigned the IP address 10.10.10.1 and the container 10.10.10.99. Each can reach the other using these private static addresses.
Enabling internet access is done by forwarding packets through the host. Make sure the host kernel has packet forwarding enabled:
sudo sysctl -w net.ipv4.ip_forward=1 >/dev/nullThen create a simple nftables forwarding config for the container interface:
sudo nft add table inet toynet
sudo nft add chain inet toynet forward \
'{ type filter hook forward priority 0; policy accept; }'
sudo nft add chain inet toynet postrouting \
'{ type nat hook postrouting priority 100; policy accept; }'
sudo nft add rule inet toynet forward \
iifname toy-host accept
sudo nft add rule inet toynet forward \
oifname toy-host accept
sudo nft add rule inet toynet postrouting \
ip saddr "10.10.10.0/24" masqueradeAnd finally register the host ip 10.10.10.1 as the default gateway inside the container:
sudo nsenter -t "$CONTAINER_PID" -n \
ip route add default via 10.10.10.1You probably also want to enable the loopback interface so programs in the container can resolve localhost:
sudo nsenter -t "$CONTAINER_PID" -n \
ip link set lo upLastly, enabling DNS resolution is solved by setting a public nameserver in the containers /etc/resolv.conf file:
sudo nsenter -t "$CONTAINER_PID" -m \
bash -c 'echo "nameserver 1.1.1.1" > /etc/resolv.conf'That should cover most basic networking needs, at least for communication with the internet and between container and host.
Real container runtimes heavily extend networking features with virtual bridges and separated virtual networks, allowing select containers to reach other containers or not. Most importantly, their networking does not rely on static device naming and ephemeral process ids to function.
The value of container runtimes
While creating a naive container is possible manually as proven in this article, container runtimes offer quality of life abstractions beyond it: standardized container images and build instructions (OCI images) utilizing cacheable overlayfs layers, easy to use command-line tooling, better implementations of /proc /sys and /dev inside containers and advanced seccomp or apparmor profiles for further hardening.
The runtime doesn't implement the containers or security features themselves, but ties them all together with convenient tooling and abstractions, which make it viable for productive use in the first place. Looking back at the commands and care needed just to construct a basic container, you really start to value the ease of use that docker run or incus launch provide.