First step to document my work: have a place to document my work!

This website is running in a Podman pod on a Debian 10 virtual machine.
Why Podman? Well because I’ve drank Red Hat’s kool-aid, (and I’ve been gifted the Podman for DevOps book by a Red Hat instructor).

I wanted Podman 4, the latest release, but Debian unstable is still stuck on 3.

federico@centrostella:~$ podman -v
podman version 3.0.1

So, let’s install it from source.

Installing Podman

Everything is well documented on their website, in the Podman Installation page, but we want Podman 4 with Netavark support: the newest, fastest, rust-yest network driver available

Fire up your terminal and type:

sudo apt-get install \
  btrfs-progs \
  crun \
  git \
  golang-go \
  go-md2man \
  iptables \
  libassuan-dev \
  libbtrfs-dev \
  libc6-dev \
  libdevmapper-dev \
  libglib2.0-dev \
  libgpgme-dev \
  libgpg-error-dev \
  libprotobuf-dev \
  libprotobuf-c-dev \
  libseccomp-dev \
  libselinux1-dev \
  libsystemd-dev \
  pkg-config \
  uidmap \
  curl \
  fuse-overlayfs

Then, we’re going to need to install golang, conmon and crun/runc.
I won’t bore you with the deatails of what they do – just run this

# Install conmon
cd
git clone https://github.com/containers/conmon
cd conmon
export GOCACHE="$(mktemp -d)"
make
sudo make podman

# Install runc
cd
git clone https://github.com/opencontainers/runc.git
cd runc
make BUILDTAGS="selinux seccomp"
sudo cp runc /usr/bin/runc

Finally, build and install podman

git clone https://github.com/containers/podman/
cd podman
make
sudo make install

and just like that:

$ podman -v
podman version 4.3.1

---

Install Netavark and Aardvark

Next, it’s time divert from their guide and install Netavark and Aardvark.
To do so, you’re gonna need Rust.

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Type 1 at the prompt to use all defaults and then type

source "$HOME/.cargo/env"

---

Luckily a couple days ago aardvark and netavark have been added to the Cargo repository so it’s as easy as typing

cargo install netavark aardvark-dns

We need to copy the build binary in one of the paths that podman will look for it, otherwise it will throw an error. While I haven’t found any documentation for it, strace comes to our help

So we just need to

sudo cp ~/.cargo/bin/aardvark-dns /usr/lib/podman/.

---

We’re not done yet

If we try to run our first container everything seems fine

$ podman run docker.io/hello-world
Trying to pull docker.io/library/hello-world:latest...
Getting image source signatures
Copying blob 2db29710123e done  
Copying config feb5d9fea6 done  
Writing manifest to image destination
Storing signatures

Hello from Docker!
This message shows that your installation appears to be working correctly.

Unfortunately, I had a terrible, hidden problem that quickly showed up on my tiny VPS with 25G of disk.

Let’s try to run something bigger, like a postgres container

$ podman container run --rm docker.io/postgres:15
Trying to pull docker.io/library/postgres:15...
Getting image source signatures
Copying blob 025c56f98b67 done  
Copying blob 26dc25c16f4e done  
Copying blob 813fd6cf203b done  
Copying blob a032d8a894de done  
Copying blob 8ebb44a56070 done  
Copying blob 40dba7d35750 done  
Copying blob 7024f61bf8f5 done  
Copying blob 23f986b322e8 done  
Copying blob 1fb05ff7a8d6 done  
Copying blob 74afc7d9bc5c done  
Copying blob 7c2c7eebef2f done  
Copying blob bdd9df7f1d37 done  
Copying blob 33d269a3a052 done  
Copying config 4c6b3cc10e done  
Writing manifest to image destination
Storing signatures

And now let’s check our disk usage:

$ du -h ~/.local/share/containers/storage
2,8G    .

What? 2.8G for a simple container?
Oh yes! Because podman is using the wrong driver to store container images

$ podman system info | grep graphDriver
  graphDriverName: vfs

Docker’s documentation defines VFS as:

The VFS storage driver is not a union filesystem; instead, each layer is a directory on disk, and there is no copy-on-write support. To create a new layer, a “deep copy” is done of the previous layer. This leads to lower performance and more space used on disk than other storage drivers.

https://docs.docker.com/storage/storagedriver/vfs-driver/

This is absolutely not what we want. Luckily in the first step I had you install fuse-overlayfs, which allows for the stratification of images the container system is known for.

sudo curl https://raw.githubusercontent.com/containers/podman/main/vendor/github.com/containers/storage/storage.conf -o /etc/containers/storage.conf

The magic lines, in this case, are the following:

[storage]
driver = "overlay"

Give the podman system a nice reset:

podman system reset

If it complains about User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve, try running it again. If it still complains, delete the .config/container/ folder

Let’s try running our postgres container once again

podman run docker.io/postgres:15
Trying to pull docker.io/library/postgres:15...
Getting image source signatures
Copying blob 26dc25c16f4e done
[...]

This time, disk usage has SIGNIFICANTLY reduced:

du -h ~/.local/share/containers/storage
388M    .

Much better!

---

Switching to Netavark networking

After all this work, we are still not one. One last step is enabling our hard-earned, self-compiled Netavark network stack

Copy the default containers.conf file and edit it

sudo cp /usr/share/containers/containers.conf /etc/containers/.
sudo nano /etc/containers/containers.conf

Change these lines:

[network]
network_backend = ""

To this:

[network]
network_backend = "netavark"

Give another reset just to be sure

$ podman system reset

So we can finally see our well-earned goal:

$ podman system info | grep networkBackend
  networkBackend: netavark

Next post I’ll explain how to set up a podman pod with wordpress and mysql.