In this post we will set up a Pod running the WordPress and Mysql containers to serve this website

We’ll start with creating a Pod.

Every Pod in Podman has its own IP, and unlike docker-compose “pods” no new network will be configured to allow containers to talk, and all containers will talk to eachother using localhost.

This unfortunately means that it won’t be single containers to expose ports, but the entire Pod.
Ports cannot be modified once a Pod is created, so we need to have our ideas clear from the beginning.

Luckily the WordPress container only exposes port 8080, which we want to map to port 80, so all we need to do is:

federico@centrostella:~$ podman pod create --name website -p 8081:80
dd2203eaf845fa1af37e7828184584c0ad93da5d3e9fa1be71f3573173da79ce

We can verify our Pod is alive with the command

Every Pod, to exist, needs a container to be active inside it. So Podman has kindly created an empty container running catatonit, which just… waits

Side note: check that your catatonit install supports the -P flag. If it does not, upgrade it following these instructions

Next we will create two containers, one for wordpress and one for postgres

podman run \
  -d \
  --pod website \
  --name mysql \
  -e MYSQL_DATABASE=blog \
  -e MYSQL_USER=tabblog \
  -e MYSQL_PASSWORD=antani \
  -e MYSQL_RANDOM_ROOT_PASSWORD=true \
  -v mysql-data:/var/lib/mysql \
  docker.io/mysql:8

podman run \
  -d \
  --pod website \
  --name wordpress \
  -e WORDPRESS_DB_HOST=127.0.0.1 \
  -e WORDPRESS_DB_NAME=blog \
  -e WORDPRESS_DB_USER=tabblog \
  -e WORDPRESS_DB_PASSWORD=antani \
  -v wordpress-data:/var/www/html \
  docker.io/wordpress

After a few moments we should be able to reach our new website on port 8081

Notice how we have bound two volumes to the data folders of the containers:

-v mysql-data:/var/lib/mysql
-v wordpress-data:/var/www/html

This means mapping the Podman volumes “mysql-data” and “wordpress-data” to the respective paths inside the container, so the application data can persist across container creation and deletions.

If we hadn’t mounted these volumes, everything would have been lost upon container destruction.

Notice also how Podman, unlike Docker, has automatically created the volumes for us

Let’s make it repeatable

Destroy the pod.

No really, tear it down. That’s what Podman is for. But first, we want to make a copy to make it repeatable.
While Docker’s docker-compose files are probably more readable, this is a journey to Kubernetes. So let’s create a Kube file.

podman generate kube website > website.yml

Warning: unfortunately I’ve been unable to have the next steps work on Debian.
While the previous steps all worked, I kept getting an error that told me that the previous steps are just unsupported. Further googling seemed to indicate a problem with the kernel itself.

This is not a rabbit hole I want to go down to, at least for now.

Afterall, if I want to play Red Hat’s game, I have to play by Red Hat’s rules.

All steps from now on will be run on a Fedora Server installation, where the entire previous post is just “yum install podman“

Then, reimport the pod with podman play kube website.yml.
That’s it! The entire pod will be recreated and spun up.

And then it’s just a matter of adding a bit of reverse proxying on my nginx frontend:

server {
    listen 443 ssl http2;
    server_name blog.tabbo.it;

    # standard ssl config and cert paths

    location / {
        proxy_pass http://127.0.0.1:8081/;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host blog.tabbo.it;
    }
}

Done!

This website is running in a Podman pod on a Debian 10 virtual machine.
Why Podman? Well because I’ve drank Red Hat’s kool-aid, (and I’ve been gifted the Podman for DevOps book by a Red Hat instructor).

I wanted Podman 4, the latest release, but Debian unstable is still stuck on 3.

federico@centrostella:~$ podman -v
podman version 3.0.1

So, let’s install it from source.

Installing Podman

Everything is well documented on their website, in the Podman Installation page, but we want Podman 4 with Netavark support: the newest, fastest, rust-yest network driver available

Fire up your terminal and type:

sudo apt-get install \
  btrfs-progs \
  crun \
  git \
  golang-go \
  go-md2man \
  iptables \
  libassuan-dev \
  libbtrfs-dev \
  libc6-dev \
  libdevmapper-dev \
  libglib2.0-dev \
  libgpgme-dev \
  libgpg-error-dev \
  libprotobuf-dev \
  libprotobuf-c-dev \
  libseccomp-dev \
  libselinux1-dev \
  libsystemd-dev \
  pkg-config \
  uidmap \
  curl \
  fuse-overlayfs

Then, we’re going to need to install golang, conmon and crun/runc.
I won’t bore you with the deatails of what they do – just run this

# Install conmon
cd
git clone https://github.com/containers/conmon
cd conmon
export GOCACHE="$(mktemp -d)"
make
sudo make podman

# Install runc
cd
git clone https://github.com/opencontainers/runc.git
cd runc
make BUILDTAGS="selinux seccomp"
sudo cp runc /usr/bin/runc

Finally, build and install podman

git clone https://github.com/containers/podman/
cd podman
make
sudo make install

and just like that:

$ podman -v
podman version 4.3.1

Install Netavark and Aardvark

Next, it’s time divert from their guide and install Netavark and Aardvark.
To do so, you’re gonna need Rust.

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Type 1 at the prompt to use all defaults and then type

source "$HOME/.cargo/env"

Luckily a couple days ago aardvark and netavark have been added to the Cargo repository so it’s as easy as typing

cargo install netavark aardvark-dns

We need to copy the build binary in one of the paths that podman will look for it, otherwise it will throw an error. While I haven’t found any documentation for it, strace comes to our help

So we just need to

sudo cp ~/.cargo/bin/aardvark-dns /usr/lib/podman/.

We’re not done yet

If we try to run our first container everything seems fine

$ podman run docker.io/hello-world
Trying to pull docker.io/library/hello-world:latest...
Getting image source signatures
Copying blob 2db29710123e done  
Copying config feb5d9fea6 done  
Writing manifest to image destination
Storing signatures

Hello from Docker!
This message shows that your installation appears to be working correctly.

Unfortunately, I had a terrible, hidden problem that quickly showed up on my tiny VPS with 25G of disk.

Let’s try to run something bigger, like a postgres container

$ podman container run --rm docker.io/postgres:15
Trying to pull docker.io/library/postgres:15...
Getting image source signatures
Copying blob 025c56f98b67 done  
Copying blob 26dc25c16f4e done  
Copying blob 813fd6cf203b done  
Copying blob a032d8a894de done  
Copying blob 8ebb44a56070 done  
Copying blob 40dba7d35750 done  
Copying blob 7024f61bf8f5 done  
Copying blob 23f986b322e8 done  
Copying blob 1fb05ff7a8d6 done  
Copying blob 74afc7d9bc5c done  
Copying blob 7c2c7eebef2f done  
Copying blob bdd9df7f1d37 done  
Copying blob 33d269a3a052 done  
Copying config 4c6b3cc10e done  
Writing manifest to image destination
Storing signatures

And now let’s check our disk usage:

$ du -h ~/.local/share/containers/storage
2,8G    .

What? 2.8G for a simple container?
Oh yes! Because podman is using the wrong driver to store container images

$ podman system info | grep graphDriver
  graphDriverName: vfs

Docker’s documentation defines VFS as:

The VFS storage driver is not a union filesystem; instead, each layer is a directory on disk, and there is no copy-on-write support. To create a new layer, a “deep copy” is done of the previous layer. This leads to lower performance and more space used on disk than other storage drivers.

https://docs.docker.com/storage/storagedriver/vfs-driver/

This is absolutely not what we want. Luckily in the first step I had you install fuse-overlayfs, which allows for the stratification of images the container system is known for.

sudo curl https://raw.githubusercontent.com/containers/podman/main/vendor/github.com/containers/storage/storage.conf -o /etc/containers/storage.conf

The magic lines, in this case, are the following:

[storage]
driver = "overlay"

Give the podman system a nice reset:

podman system reset

If it complains about User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve, try running it again. If it still complains, delete the .config/container/ folder

Let’s try running our postgres container once again

podman run docker.io/postgres:15
Trying to pull docker.io/library/postgres:15...
Getting image source signatures
Copying blob 26dc25c16f4e done
[...]

This time, disk usage has SIGNIFICANTLY reduced:

du -h ~/.local/share/containers/storage
388M    .

Much better!

Switching to Netavark networking

After all this work, we are still not one. One last step is enabling our hard-earned, self-compiled Netavark network stack

Copy the default containers.conf file and edit it

sudo cp /usr/share/containers/containers.conf /etc/containers/.
sudo nano /etc/containers/containers.conf

Change these lines:

[network]
network_backend = ""

To this:

[network]
network_backend = "netavark"

Give another reset just to be sure

$ podman system reset

So we can finally see our well-earned goal:

$ podman system info | grep networkBackend
  networkBackend: netavark

Next post I’ll explain how to set up a podman pod with wordpress and mysql.