Using Containers on Gentoo

Containers are a great tool that caters to some specific, and important, needs. But be aware that containers are not the solution to selfhosting-made-easy and, specifically, containers have been created to solve different issues than self-hosting!

Bear in mind, always, that containers while being an astounding piece of technology, are NOT meant and have NOT been created with self-hosting in mind. Since it's easy and simple to use them for self-hosting, at least let's see how to do that properly.

Containerization technology can roughly be divided into two big categories:

Containers for development (Docker and the like)
Clusters for deployment (Kubernetes and the like)

Over the course of time the use of Docker in the self-hosted world has increased so much that it's getting really troublesome in my opinion. In fact containers have some great positive points for self-hosters like:

Ease of deployment (docker compose up, and there you go)
Ease of upgrade (docker pull, and there you go)
Ease of management (just move your containers around to a new server when needed)
For developers, it's easier to provide a docker-compose.yml rather than proper installation instruction

I am realizing more and more the added value of using containers, so i have changed my mind over time. I still think there are negative points in relying on Docker for self-hosting:

All services running as root (against Linux policies)
Lots of duplicated services (how many postgress databases have you running around?)
You don't learn anything new deploying your services
When it breaks, it breaks hard because you don't know what broke
To understand and fix it, you still need all those knowledge that yo where trying to run from
It's back to Windows approach: black boxes everywhere that you can only roll-back or reinstall

In short, the spreading usage of Docker in the self-hosting world bring to Linux some bad practices and philosophy of the Windows world. Is this really necessary? Do we really want to become forced to be idiot-users who don't know any better? Is this really what means to be Linux users?

Question for your thoughts, now let's see how, at least, to use containers in a slightly better way.

"TLDR"

We are already at the point where some services are provided only as containers, and this is what i think the the worse possible outcome of this using containers more and more. Where is the choice? Why would i be forced to use some semi-proprietary technology to deploy Open Source services?

I think that this is a trend that should be stopped. Please do provide containers and docker files, it's good and why not, but always also support bare-metal installations which means, please always provide binary distributions and installation instructions because without those, we will be nothing better than idiot-users and your service cannot be really called open.

Docker

Everybody uses Docker. You know why? Because it was the first kid on the block.

Docker, in today's tech lingo, means container, but it's not true. Docker was only the first one and it has a few drawbacks:

Docker was designed for development not for production
Security has been kind of an afterthought added later on
Base Docker is not secure (socket exploits, all processes running as root unless…)
Docker does NOT fit into the Linux/Unix way of thinking and breaks the rules
Docker is not fully Open Source and it's controlled by a single company, which might paywall some stuff in the future (i don't think so, but as Qt is proving, you might never know)

Luckily there are alternatives, which are better in my opinion.

Podman

Podman is a toolset to manage containers which follow the Open Containers Initiative standards and is fully compatible with Docker, but provides a few improvements:

Doesn't run as root
Doesn't require a daemon
Integrates with OpenRC/SystemD without reinventing the wheel
It's fully Open Source
It's been designed with security in mind
It's not monolithic but it's actually a set of tools
And it's also fully Docker-compatible (set alias docker=podman and you are dood to go, almost)

Overall Podman is much more adherent to the Linux/Unix way of doing things.

Installing Podman is pretty easy since it's in Portage repository, but let's enable docker wrapper as well, so you can “forget” you run Podman and use the docker command instead:

echo "app-containers/podman wrapper" > /etc/portage/package.use/podman
emerge app-containers/podman

and it will automatically act as “podman” You will need to unmerge docker-cli and docker-compose, if you have them installed.

To ensure that Podman will use Docker repository to pull images, add the following line to your /etc/containers/registries.conf:

unqualified-search-registries = ["docker.io" ]

You can, as with Docker, test your Podman installation with the command:

 > podman run --rm hello-world

which i suggest to run as un-priviledged user to verify everything is working as non-root too.

Now, install podman-compose which is a bit more complex because at this time there is no official ebuild yet.

Follow my repo guide to create a custom repo (or use your already existing custom repo) to add the following ebuild:

podman-compose-1.0.6.ebuild

# Copyright 2024 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2

EAPI=8

DISTUTILS_USE_SETUPTOOLS=rdepend
PYTHON_COMPAT=( python3_{10..11} )

inherit distutils-r1

DESCRIPTION="Run docker-compose files without root with podman"
HOMEPAGE="https://pypi.org/project/podman-compose https://github.com/containers/podman-compose"
SRC_URI="https://files.pythonhosted.org/packages/65/a8/d77d2eaa85414d013047584d3aa10fac47edb328f5180ca54a13543af03a/podman-compose-1.0.6.tar.gz"
#SRC_URI="mirror://pypi/${PN:0:1}/${PN}/${P}.tar.gz"

LICENSE="GPL-2"
SLOT="0"
KEYWORDS="~amd64"

DEPEND=""
RDEPEND="
        ${DEPEND}
        dev-python/pyyaml[${PYTHON_USEDEP}]
        dev-python/python-dotenv[${PYTHON_USEDEP}]
"
BDEPEND=""

as app-containers/podman-compose-1.0.6.ebuild and then emerge it (see https://bugs.gentoo.org/717748).

I hope that this ebuild will be merged to Gentoo official repo soon.

Podman rootless users

When running a container rootless, which is the main point behind Podman, you might end up having some issues with user IDs.

This article is a very good read on the topic.

In short: when running rootless, the user 0, or root, of the container, will be mapped to your user ID, and any additional user will be remapped according to the content of /etc/subuid and /etc/subgid files. This means that if the container defines additioanl users, they will not map to your user but to a different UID that is also sub-mapped to your user, but Linux itself will not recognize that and any files and folders created by the container will belog to this strange user.

The easiest way to fix it, is to force your container to run… as user 0! In fact, since we are using rootless podman, that only means as your own user nad not actually root.

Podman networks

Since you want to run containers as non-root, you need to create as root the networks. My approach is to create one subnetwork for each group of containers, and each group of containers will run a non-root user.

To create a Podman subnet you need to run the following command after each reboot, as root:

 >   podman network create my-container-net

Inside your docker-compose.yml you will add a new network called my-container-net, for eaxmple:

    services:
   .....
    environment:
    ....
    ports:
      - xxx:yyy
    volumes:
      - my-volume:/root/.config/:ro
    networks:
      - my-container-net

....

networks:
  my-container-net: {}

I strongly suggest that you edit your docker compose files and ensure each service has it's own independent network. I will give more details for each service on it's respective page.

Podman containers autostart

Autostarting containers is pretty easy if you use SystemD, but even if you don't, and i don't, it's easy enough too.

/etc/init.d/user-containers

#!/sbin/openrc-run
# Copyright 2024 Willy Garidol
# Distributed under the terms of the GNU General Public License v3

depend() {
        need localmount net
}

UC_LOG_PATH=/var/log/user-containers
UC_SLOT="${SVCNAME#user-containers.}"
UC_USER=${USER:-${UC_SLOT}}
UC_COMPOSER_FILE=${COMPOSER_FILE:-docker-compose.yml}
UC_CHOWN_DIR=${CHOWN_DIR}
if [ "${UC_SLOT}" != "user-containers" ]
then
        UC_HOME=${HOME:-$(su - ${UC_USER} -c "pwd")}
fi

extra_commands="update"
update() {
        if [ "${UC_SLOT}" != "user-containers" ]
        then
                COMMAND="$(which podman)"
                stop
                ebegin "Running podman compose pull..."
                su - ${UC_USER} -c "${COMMAND} compose -f ${UC_COMPOSER_FILE} pull"
                start
        else
                ebegin "Error: do not run this script, run a link to it!"
                eend 255
        fi
}

description=${DESCRIPTION:-You forgot to describe your container}
pidfile="/run/${RC_SVCNAME}.pid"

start_pre() {
        if [ "${UC_SLOT}" != "user-containers" ]
        then
                test -e "${UC_LOG_PATH}" || mkdir "${UC_LOG_PATH}"
                test -e "${UC_LOG_PATH}/${UC_SLOT}" || {
                        mkdir "${UC_LOG_PATH}/${UC_SLOT}"
                } && chown -R ${UC_USER} "${UC_LOG_PATH}/${UC_SLOT}"
                if [ -n "${UC_CHOWN_DIR}" -a -e ${UC_CHOWN_DIR} ]
                then
                        chown -R ${UC_USER} ${UC_CHOWN_DIR}
                fi
        else
                ebegin "Error: do not run this script, run a link to it!"
                eend 255
        fi
}


start() {
        ebegin "Starting container '${UC_SLOT}' for user '${UC_USER}' (${UC_HOME})"
        COMMAND="$(which podman)"
        ARGUMENTS=(compose -f ${UC_COMPOSER_FILE} up)
        ebegin " ... ensuring nat table is loaded ..."
        iptables -L -t nat &> /dev/null
        ebegin " ... creating '${UC_SLOT}-net' ..."
        podman network create ${UC_SLOT}-net &> /dev/null
        su - "${UC_USER}" -c "$(which podman) compose down" &> /dev/null
        start-stop-daemon -p ${pidfile} \
                          -1 "${UC_LOG_PATH}/${UC_SLOT}/${UC_SLOT}.out.log" \
                          -2 "${UC_LOG_PATH}/${UC_SLOT}/${UC_SLOT}.err.log" \
                          -u ${UC_USER} \
                          -d ${UC_HOME} \
                          -b -m \
                          --start  "${COMMAND}" \
                          -- ${ARGUMENTS[@]}
        eend $?
}

stop() {
        ebegin " ... running 'podman compose down' ..."
        su - "${UC_USER}" -c "$(which podman) compose -f ${UC_COMPOSER_FILE} down" &> /dev/null
        start-stop-daemon -p ${pidfile} \
                          -u ${UC_USER} \
                          -d ${UC_HOME} \
                          --stop ${UC_SLOT}
        eend $?
}

I assume you have aservice_name that runs with a podman compose file as user myuser.

Just link the above script to your new service file:

ln -s /etc/init.d/user-containers /etc/init.d/user-containers.service_name

and create simple configuration file with user name and description in it:

/etc/conf.d/user-containers.service_name

USER=myuser
DESCRIPTION="my containerized service"

and add the service to your desired runlevel:

rc-update add user-containers.service_name default

That's it.

The above script also provide an “update” commnand that will update your containers automatically.

Using Podman for new containers

Well, just replace any docker command with podman and you are good to go. Including the usual:

 > podman compose up

That's it.

Things to remember (differences from Docker):

always run Podman as unpriviledged user
Podman containers are not restarted at boot: i will give you instructions for those services as needed.
Networks needs to be explicitly declared and created as root.

Migration from Docker to Podman

If you already have some services (let's call it service) running with Docker, here is how to convert them.

Start by creating a non-privileged user (let's call it service to match the service name), let it point to where you stored the docker-compose.yml for the service (which should be /data/daemons/service already) and fix the permissions:

 > useradd -d /data/daemons/service -m service
 > chown service:service -R /data/daemons/service

At this point you might want to edit the network part of the docker compose file according to what i wrote above.

Now, most probably all you need to do is the classic (but rewritten):

 > su - service
 > podman compose -f docker-compose.yml up

and enjoy.

(remember to create the network, as root)

Migrating Images Podman

If you need to export your images from Docker to Podman (you don't if they are public images), as root, export all docker images relevant to your service (you can see them in the composer file), use docker image ls to list images and docker save -o … to save each one of them as a tar file:

 > mkdir /data/daemons/service/docker-export
 > docker image ls
 > docker image save -o /data/daemons/service/docker-export/image_name.tar image-id
 > chown service:service /data/daemons/service/docker-export/image_name.tar

(repeat for each image for the service!)

If your container uses also volumes, copy them to your service user, as root:

 > mkdir /data/daemons/service/volumes
 > cp -a /var/lib/docker/volumes/service-image /data/daemons/service/volumes
 > chown service:service -R /data/daemons/service/volumes

(repeat for each volume for the service!)

Now, as user server, import the images and create the volumes:

 > su - service
 > podman load -i docker-export/image_name.tar

Migrating Volumes Podman

Volumes might be more interesting when migrating existing services, because volumes will contain your data, including databases and configurations.

The quickest way is to:

Start the containers once (podman compose up), so that new volumes gets created
Stop the containers (podman compose down)
Manually copy the data from Docker volumes to Podman volumes

The last step is easy because you can find your Docker volumes under /var/lib/docker/volumes and your Podman volumes (with the same names!) under /data/daemons/service/.local/share/containers/storage/volumes, so all you need to do (as root) is perform a full copy, but don't forget to check owership of the files, since that might change from Docker to Podman:

 > cp -a /var/lib/docker/volumes/my-volume /data/daemons///service///.local/share/containers/storage/volumes

Some additional notes

- ping is restricted and cannot b performed from containers. If you need to enable it, type as root:

 > sysctl -w "net.ipv4.ping_group_range=0 2000000"

This can be made permanent in /etc/sysctl.d.

- Gentoo on OpenRC will NOT kill your apps in background when logging-out from a user. You don't have anything to setup for Podman containers to properly run in background (point to OpenRC over SystemD).

- Running as simple user a container will not be allowed to bind to ports under 1024. Some ill-designed containers will insist on this. The only recurse (except don't use such broken images) is to allow ports for normal users with:

 > sysctl -w "net.ipv4.ip_unprivileged_port_start=80"