Using Containers on Gentoo

Containers are a great tool that caters to some specific, and important, needs. But be aware that containers are not the solution to selfhosting-made-easy and, specifically, containers have been created to solve different issues than self-hosting!

Bear in mind, always, that containers while being an astounding piece of technology, are NOT meant and have NOT been created with self-hosting in mind. Since it's easy and simple to use them for self-hosting, at least let's see how to do that properly.

Containerization technology can roughly be divided into two big categories:

• Containers for development (Docker and the like)
• Clusters for deployment (Kubernetes and the like)

Over the course of time the use of Docker in the self-hosted world has increased so much that it's getting really troublesome in my opinion. In fact containers have some great positive points for self-hosters like:

• Ease of deployment (docker compose up, and there you go)
• Ease of upgrade (docker pull, and there you go)
• Ease of management (just move your containers around to a new server when needed)
• For developers, it's easier to provide a docker-compose.yml rather than proper installation instruction

I am realizing more and more the added value of using containers, so i have changed my mind over time. I still think there are negative points in relying on Docker for self-hosting:

• All services running as root (against Linux policies)
• Lots of duplicated services (how many postgress databases have you running around?)
• You don't learn anything new deploying your services
• When it breaks, it breaks hard because you don't know what broke
• To understand and fix it, you still need all those knowledge that yo where trying to run from
• It's back to Windows approach: black boxes everywhere that you can only roll-back or reinstall

In short, the spreading usage of Docker in the self-hosting world bring to Linux some bad practices and philosophy of the Windows world. Is this really necessary? Do we really want to become forced to be idiot-users who don't know any better? Is this really what means to be Linux users?

Question for your thoughts, now let's see how, at least, to use containers in a slightly better way.

We are already at the point where some services are provided only as containers, and this is what i think the the worse possible outcome of this using containers more and more. Where is the choice? Why would i be forced to use some semi-proprietary technology to deploy Open Source services?

I think that this is a trend that should be stopped. Please do provide containers and docker files, it's good and why not, but always also support bare-metal installations which means, please always provide binary distributions and installation instructions because without those, we will be nothing better than idiot-users and your service cannot be really called open.

Everybody uses Docker. You know why? Because it was the first kid on the block.

Docker, in today's tech lingo, means container, but it's not true. Docker was only the first one and it has a few drawbacks:

• Docker was designed for development not for production
• Security has been kind of an afterthought added later on
• Base Docker is not secure (socket exploits, all processes running as root unless…)
• Docker does NOT fit into the Linux/Unix way of thinking and breaks the rules
• Docker is not fully Open Source and it's controlled by a single company, which might paywall some stuff in the future (i don't think so, but as Qt is proving, you might never know)

Luckily there are alternatives, which are better in my opinion.

Podman is a toolset to manage containers which follow the Open Containers Initiative standards and is fully compatible with Docker, but provides a few improvements:

• Doesn't run as root
• Doesn't require a daemon
• Integrates with OpenRC/SystemD without reinventing the wheel
• It's fully Open Source
• It's been designed with security in mind
• It's not monolithic but it's actually a set of tools
• And it's also fully Docker-compatible (set alias docker=podman and you are dood to go, almost)

Overall Podman is much more adherent to the Linux/Unix way of doing things.

Installing Podman is pretty easy since it's in Portage repository, but let's enable docker wrapper as well, so you can “forget” you run Podman and use the docker command instead:

echo "app-containers/podman wrapper" > /etc/portage/package.use/podman
emerge podman

and it will automatically act as “podman” You will need to unmerge docker-cli and docker-compose, if you have them installed.

To ensure that Podman will use Docker repository to pull images, add the following line to your /etc/containers/registries.conf:

unqualified-search-registries = ["docker.io" ]

You can, as with Docker, test your Podman installation with the command:

 > podman run --rm hello-world

which i suggest to run as un-priviledged user to verify everything is working as non-root too.

Now, install podman-compose which is a bit more complex because at this time there is no official ebuild yet.

Follow my repo guide to create a custom repo (or use your already existing custom repo) to add the following ebuild:

podman-compose-1.0.6.ebuild

# Copyright 2024 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2

EAPI=8

DISTUTILS_USE_SETUPTOOLS=rdepend
PYTHON_COMPAT=( python3_{10..11} )

inherit distutils-r1

DESCRIPTION="Run docker-compose files without root with podman"
HOMEPAGE="https://pypi.org/project/podman-compose https://github.com/containers/podman-compose"
SRC_URI="https://files.pythonhosted.org/packages/65/a8/d77d2eaa85414d013047584d3aa10fac47edb328f5180ca54a13543af03a/podman-compose-1.0.6.tar.gz"
#SRC_URI="mirror://pypi/${PN:0:1}/${PN}/${P}.tar.gz"

LICENSE="GPL-2"
SLOT="0"
KEYWORDS="~amd64"

DEPEND=""
RDEPEND="
        ${DEPEND}
        dev-python/pyyaml[${PYTHON_USEDEP}]
        dev-python/python-dotenv[${PYTHON_USEDEP}]
"
BDEPEND=""

as app-containers/podman-compose-1.0.6.ebuild and then emerge it (see https://bugs.gentoo.org/717748).

I hope that this ebuild will be merged to Gentoo official repo soon.

Since you want to run containers as non-root, you need to create as root the networks. My approach is to create one subnetwork for each group of containers, and each group of containers will run a non-root user.

To create a Podman subnet you need to run the following command after each reboot, as root:

 >   podman network create my-container-net

Inside your docker-compose.yml you will add a new network called my-container-net, for eaxmple:

    services:
   .....
    environment:
    ....
    ports:
      - xxx:yyy
    volumes:
      - my-volume:/root/.config/:ro
    networks:
      - my-container-net

....

networks:
  my-container-net: {}
  

I strongly suggest that you edit your docker compose files and ensure each service has it's own independent network. I will give more details for each service on it's respective page.

Well, just replace any docker command with podman and you are good to go. Including the usual:

 > podman compose up

That's it.

Things to remember (differences from Docker):

• always run Podman as unpriviledged user
• Podman containers are not restarted at boot: i will give you instructions for those services as needed.
• Networks needs to be explicitly declared and created as root.

If you already have some services (let's call it service) running with Docker, here is how to convert them.

Start by creating a non-privileged user (let's call it service to match the service name), let it point to where you stored the docker-compose.yml for the service (which should be /data/daemons/service already) and fix the permissions:

 > useradd -d /data/daemons/service -m service
 > chown service:service -R /data/daemons/service

At this point you might want to edit the network part of the docker compose file according to what i wrote above.

Now, most probably all you need to do is the classic (but rewritten):

 > su - service
 > podman compose -f docker-compose.yml up

and enjoy.

(remember to create the network, as root)

If you need to export your images from Docker to Podman (you don't if they are public images), as root, export all docker images relevant to your service (you can see them in the composer file), use docker image ls to list images and docker save -o … to save each one of them as a tar file:

 > mkdir /data/daemons/service/docker-export
 > docker image ls
 > docker image save -o /data/daemons/service/docker-export/image_name.tar image-id
 > chown service:service /data/daemons/service/docker-export/image_name.tar

(repeat for each image for the service!)

If your container uses also volumes, copy them to your service user, as root:

 > mkdir /data/daemons/service/volumes
 > cp -a /var/lib/docker/volumes/service-image /data/daemons/service/volumes
 > chown service:service -R /data/daemons/service/volumes

(repeat for each volume for the service!)

Now, as user server, import the images and create the volumes:

 > su - service
 > podman load -i docker-export/image_name.tar

Volumes might be more interesting when migrating existing services, because volumes will contain your data, including databases and configurations.

The quickest way is to:

1. Start the containers once (podman compose up), so that new volumes gets created
2. Stop the containers (podman compose down)
3. Manually copy the data from Docker volumes to Podman volumes

The last step is easy because you can find your Docker volumes under /var/lib/docker/volumes and your Podman volumes (with the same names!) under /data/daemons/service/.local/share/containers/storage/volumes, so all you need to do (as root) is perform a full copy, but don't forget to check owership of the files, since that might change from Docker to Podman:

 > cp -a /var/lib/docker/volumes/my-volume /data/daemons///service///.local/share/containers/storage/volumes 

- ping is restricted and cannot b performed from containers. If you need to enable it, type as root:

 > sysctl -w "net.ipv4.ping_group_range=0 2000000" 

This can be made permanent in /etc/sysctl.d.

- Gentoo on OpenRC will NOT kill your apps in background when logging-out from a user. You don't have anything to setup for Podman containers to properly run in background (point to OpenRC over SystemD).

- Running as simple user a container will not be allowed to bind to ports under 1024. Some ill-designed containers will insist on this. The only recurse (except don't use such broken images) is to allow ports for normal users with:

 > sysctl -w "net.ipv4.ip_unprivileged_port_start=80"

This can be made permanent in /etc/sysctl.d.

- Podman containers are not restarted at boot. To achieve the same behaviour as Docker, my suggestion is to follow my lead on leveraging local.d with Custom User Services where i show you how i manage very easily to auto-start containers at boot and properly shut them down too, without writing a single line of script.

If you want to do things manually, just keen in mind that you need to ensure nat ip-table is loaded. While it is not used by podman, it seems to be necessary somehow to exist:

modprobe iptable_nat
iptable -L -t nat