Introduction
Docker Hub recently introduced changes to their policy which impacts communities working with and using Docker Hub for sharing container images they produce.
Summary of policy changes are
- Image Retention Period: Inactive images that are not pushed/pulled within last 6 months is candidate for removal. (Reference)
- Image Pull Rate Limitation: Anonymous pulls are rate limited which means that images pulls will be blocked once the limit is reached. (Reference)
In order to limit impacts on the environment, Nordix Infra team made Docker Hub proxy available through Nordix Container Image Registry (Nordix Harbor).
This document describes how this works and what steps to follow in order to use proxy provided by Nordix.
Considerations
Docker client has docker.io hardcoded in its code base and there is no straightforward way to configure it to use proxy to access the images available on Docker Hub.
Whenever you attempt to pull images using unqualified names (such as docker pull hello-world instead of docker.io/library/hello-world), docker client ends up looking at Docker Hub by default which may result in hitting image pull rate limitation over time if the user is frequently pulling images with free Docker Hub plan.
Another limitation is with the Dockerfiles. Majority of existing Dockerfiles created and used for projects tend to use unqualified names for container images like the one shown below.
FROM openjdk:12 AS build COPY . /temp WORKDIR temp RUN ./gradlew clean jar FROM openjdk:12 # Use the shell form of ENTRYPOINT to allow environment variable expansion. # exec is used to make sure that the JVM gets pid 1 so that it receives # SIGTERM et al and gets an opportunity to shut down cleanly. ENTRYPOINT exec java ${JAVA_OPTS:-} -jar /app/eiffel-gerrit-herald.jar ${HERALD_OPTS:-} RUN mkdir /app WORKDIR /app COPY --from=build /temp/build/libs/eiffel-gerrit-herald-*.jar /app/eiffel-gerrit-herald.jar
What happens when a user attempts to build the image using docker build command is same as pulling image using docker client, thus increased possibility of hitting rate limits.
With docker client, it is challenging to provide none-intrusive proxy support for docker pull or docker build since the client will fall back to docker.io even with fully qualified image names.
Using the image registry which is where the proxy is setup as part of fully qualified name for the images is not preferred either since it will then not possible to access such registry in all cases due to different reasons.
Various alternatives are investigated to address the issues summarized above and provide proxy support for the Nordix environment without touching Dockerfiles.
The most promising alternative is the open source project Podman.
Podman provides straightforward way to use proxy registries in none-intrusive and transparent way without making any changes to Dockerfiles or how people use docker client on a daily basis.
Even though there are limitations with Podman comparing to docker client itself, Podman can easily be used as drop in replacement for most frequently used container image operations/commands such as push, pull, build, and so on.
Podman has various advantages over docker itself. The first and perhaps the most important one is the rootless interaction with containers.
The other advantage is that Podman is daemon-less. (No Big Fat Daemons)
Please consult Podman documentation for more details.
Rest of the guide will demonstrate how Podman can be installed, configured, and used to allow users to utilise Nordix Harbor as proxy to Docker Hub.
As usual, please send any questions and/or issues you have with this to Nordix maillist nordix@lists.openinfra.dev since this is a Nordix Environment by OpenInfra Foundation maintained service. (You need to subscribe to maillist in order to send & receive mails to/from this maillist.)
Installation and Configuration Guide
Installation and configuration instructions for Ubuntu 18.04 and Ubuntu 20.04.
Please consult Podman Installation Guide for instructions for other operating systems.
Instructions for Ubuntu 18.04
These instructions are valid for podman version 2.1.1.
It is assumed that you have right to execute sudo commands.
It is not straightforward to configure Podman on Ubuntu 1804 to interact with containers in a rootless mode.
There are multiple reasons for this such as old Linux Kernel version (4.15) or lack of fuse-liboverlayfs debian package.
Because of this, there are differences while configuring and using Podman on Ubuntu 18.04. The biggest difference is that all Podman commands used while interacting with images and containers are executed as root on Ubuntu 18.04.
Install podman
. /etc/os-release echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/Release.key | sudo apt-key add - sudo apt-get update sudo apt-get -y upgrade sudo apt-get -y install podman
Verify podman installation
podman version podman info --debug
Configure podman to use Nordix Harbor as proxy to Docker Hub
mkdir -p ~/.config/containers cat << EOF > ~/.config/containers/registries.conf unqualified-search-registries = ["docker.io"] [[registry]] prefix = "docker.io" location = "registry.nordix.org/docker-hub-proxy" insecure = false EOF
Verify podman pulls images through Docker Hub proxy set up on Nordix Harbor. You should be seeing references to registry.nordix.org in the output and image getting pulled from the Nordix Harbor.
sudo podman pull hello-world --log-level debug ...snip... DEBU[0000] GET https://registry.nordix.org/v2/ DEBU[0000] Ping https://registry.nordix.org/v2/ status 401 DEBU[0000] GET https://registry.nordix.org/service/token?scope=repository%3Adocker-hub-proxy%2Flibrary%2Fhello-world%3Apull&service=harbor-registry DEBU[0000] GET https://registry.nordix.org/v2/docker-hub-proxy/library/hello-world/manifests/latest DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json" DEBU[0001] Using blob info cache at /home/ubuntu/.local/share/containers/cache/blob-info-cache-v1.boltdb DEBU[0001] Source is a manifest list; copying (only) instance sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042 for current system DEBU[0001] GET https://registry.nordix.org/v2/docker-hub-proxy/library/hello-world/manifests/sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042 DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" DEBU[0001] IsRunningImageAllowed for image docker:docker.io/library/hello-world:latest DEBU[0001] Using default policy section DEBU[0001] Requirement 0: allowed DEBU[0001] Overall: allowed DEBU[0001] Downloading /v2/docker-hub-proxy/library/hello-world/blobs/sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b DEBU[0001] GET https://registry.nordix.org/v2/docker-hub-proxy/library/hello-world/blobs/sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b ...snip... sudo podman images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/library/hello-world latest bf756fb1ae65 9 months ago 20 kB sudo podman run hello-world ...snip... Hello from Docker! This message shows that your installation appears to be working correctly. ...snip...
Attempt an image build
sudo apt-get install -y git git clone https://github.com/eiffel-community/eiffel-gerrit-herald.git cd eiffel-gerrit-herald sudo podman build . ...snip... STEP 8: WORKDIR /app --> e1d8ccf9597 STEP 9: COPY --from=build /temp/build/libs/eiffel-gerrit-herald-*.jar /app/eiffel-gerrit-herald.jar STEP 10: COMMIT --> 83cd5a06249 83cd5a06249eff1c4df79edfbd7ef5ee03ea36cd0fd7c0d8e5e835f4d67ab08a ...snip...
You should have the image built
sudo podman images REPOSITORY TAG IMAGE ID CREATED SIZE <none> <none> 83cd5a06249e 31 seconds ago 507 MB <none> <none> 77401bfcd466 59 seconds ago 950 MB docker.io/library/hello-world latest bf756fb1ae65 9 months ago 20 kB docker.io/library/openjdk 12 e1e07dfba89c 13 months ago 477 MB
Instructions for Ubuntu 20.04
These instructions are valid for podman version 2.1.1.
It is assumed that you have right to execute sudo commands.
Please note that these instructions are for running rootless Podman.
Install podman
. /etc/os-release echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/Release.key | sudo apt-key add - sudo apt-get update sudo apt-get -y upgrade sudo apt-get -y install podman fuse-overlayfs
Verify podman installation
podman version podman info --debug
Configure podman to use Nordix Harbor as proxy to Docker Hub
mkdir -p ~/.config/containers cat << EOF > ~/.config/containers/registries.conf unqualified-search-registries = ["docker.io"] [[registry]] prefix = "docker.io" location = "registry.nordix.org/docker-hub-proxy" insecure = false EOF
Change storage driver to overlayfs
cat << EOF > ~/.config/containers/storage.conf [storage] driver = "overlay" runroot = "/run/user/$(id -u)" graphroot = "$HOME/.local/share/containers/storage" [storage.options] mount_program = "/usr/bin/fuse-overlayfs" EOF
Verify podman pulls images through Docker Hub proxy set up on Nordix Harbor. You should be seeing references to registry.nordix.org in the output and image getting pulled from the Nordix Harbor. (Please note that IMAGE IDs may be different in the command output)
podman pull hello-world --log-level debug ...snip... DEBU[0000] GET https://registry.nordix.org/v2/ DEBU[0000] Ping https://registry.nordix.org/v2/ status 401 DEBU[0000] GET https://registry.nordix.org/service/token?scope=repository%3Adocker-hub-proxy%2Flibrary%2Fhello-world%3Apull&service=harbor-registry DEBU[0000] GET https://registry.nordix.org/v2/docker-hub-proxy/library/hello-world/manifests/latest DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json" DEBU[0001] Using blob info cache at /home/ubuntu/.local/share/containers/cache/blob-info-cache-v1.boltdb DEBU[0001] Source is a manifest list; copying (only) instance sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042 for current system DEBU[0001] GET https://registry.nordix.org/v2/docker-hub-proxy/library/hello-world/manifests/sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042 DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" DEBU[0001] IsRunningImageAllowed for image docker:docker.io/library/hello-world:latest DEBU[0001] Using default policy section DEBU[0001] Requirement 0: allowed DEBU[0001] Overall: allowed DEBU[0001] Downloading /v2/docker-hub-proxy/library/hello-world/blobs/sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b DEBU[0001] GET https://registry.nordix.org/v2/docker-hub-proxy/library/hello-world/blobs/sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b ...snip... podman images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/library/hello-world latest bf756fb1ae65 9 months ago 20 kB podman run hello-world ...snip... Hello from Docker! This message shows that your installation appears to be working correctly. ...snip...
Attempt an image build. (Please note that IMAGE IDs may be different in the command output)
sudo apt-get -y install git git clone https://github.com/eiffel-community/eiffel-gerrit-herald.git cd eiffel-gerrit-herald podman build . ...snip... STEP 8: WORKDIR /app --> 2a94a693a48 STEP 9: COPY --from=build /temp/build/libs/eiffel-gerrit-herald-*.jar /app/eiffel-gerrit-herald.jar STEP 10: COMMIT --> 9b98923f3a6 9b98923f3a614348b065ee438a0e135ecc2bf137e0c503916f5b8665bd82c1d1 ...snip...
You should have the image built. (Please note that IMAGE IDs may be different in the command output)
podman images REPOSITORY TAG IMAGE ID CREATED SIZE <none> <none> 9b98923f3a61 19 seconds ago 507 MB <none> <none> 893fce3eb5bd About a minute ago 950 MB docker.io/library/hello-world latest bf756fb1ae65 9 months ago 20 kB docker.io/library/openjdk 12 e1e07dfba89c 13 months ago 477 MB
How Does It Work?
When Podman is installed and configured to use Nordix Harbor as proxy to Docker Hub, podman commands to pull images from Docker Hub, Podman rewrites reference to Nordix Harbor.
This then results in queries for images hitting to proxy on Nordix Harbor first.
If the requested image (and tag) is available on Nordix Harbor, it is served to the client.
If the requested image (and tag) is not available on Nordix Harbor, it gets cached to Nordix Harbor and then served to the client.
And the rest is business as usual.