Page tree
Skip to end of metadata
Go to start of metadata

Introduction

Docker Hub recently introduced changes to their policy which impacts communities working with and using Docker Hub for sharing container images they produce.

Summary of policy changes are

  • Image Retention Period: Inactive images that are not pushed/pulled within last 6 months is candidate for removal. (Reference)
  • Image Pull Rate Limitation: Anonymous pulls are rate limited which means that images pulls will be blocked once the limit is reached. (Reference)

In order to limit impacts on the community, Nordix Infra team made Docker Hub proxy available through Nordix Container Image Registry (Nordix Harbor).

This document describes how this works and what steps to follow in order to use proxy provided by Nordix.

Considerations

Docker client has docker.io hardcoded in its code base and there is no straightforward way to configure it to use proxy to access the images available on Docker Hub.

Whenever you attempt to pull images using unqualified names (such as docker pull hello-world instead of docker.io/library/hello-world), docker client ends up looking at Docker Hub by default which may result in hitting image pull rate limitation over time if the user is frequently pulling images with free Docker Hub plan.

Another limitation is with the Dockerfiles. Majority of existing Dockerfiles created and used for projects tend to use unqualified names for container images like the one shown below.

FROM openjdk:12 AS build

COPY . /temp
WORKDIR temp
RUN ./gradlew clean jar

FROM openjdk:12

# Use the shell form of ENTRYPOINT to allow environment variable expansion.
# exec is used to make sure that the JVM gets pid 1 so that it receives
# SIGTERM et al and gets an opportunity to shut down cleanly.
ENTRYPOINT exec java ${JAVA_OPTS:-} -jar /app/eiffel-gerrit-herald.jar ${HERALD_OPTS:-}

RUN mkdir /app
WORKDIR /app

COPY --from=build /temp/build/libs/eiffel-gerrit-herald-*.jar /app/eiffel-gerrit-herald.jar

What happens when a user attempts to build the image using docker build command is same as pulling image using docker client, thus increased possibility of hitting rate limits.

With docker client, it is challenging to provide none-intrusive proxy support for docker pull or docker build since the client will fall back to docker.io even with fully qualified image names.

Using the image registry which is where the proxy is setup as part of fully qualified name for the images is not preferred either since it will then not possible to access such registry in all cases due to different reasons. 

Various alternatives are investigated to address the issues summarised above and provide proxy support for the community without touching Dockerfiles.

The most promising alternative is the open source project Podman.

Podman provides straightforward way to use proxy registries in none-intrusive and transparent way without making any changes to Dockerfiles or how people use docker client on a daily basis.

Even though there are limitations with Podman comparing to docker client itself, Podman can easily be used as drop in replacement for most frequently used container image operations/commands such as pushpullbuild, and so on.

Podman has various advantages over docker itself. The first and perhaps the most important one is the rootless interaction with containers.

The other advantage is that Podman is daemon-less. (No Big Fat Daemons)

Please consult Podman documentation for more details.

Rest of the guide will demonstrate how Podman can be installed, configured, and used to allow users to utilise Nordix Harbor as proxy to Docker Hub.

As usual, please send any questions and/or issues you have with this to Nordix maillist discuss@lists.nordix.org since this is a community maintained service. (You need to subscribe to maillist in order to send & receive mails to/from this maillist.)

Installation and Configuration Guide

Installation and configuration instructions for Ubuntu 18.04 and Ubuntu 20.04.

Please consult Podman Installation Guide for instructions for other operating systems.

Instructions for Ubuntu 18.04

These instructions are valid for podman version 2.1.1.

It is assumed that you have right to execute sudo commands.

It is not straightforward to configure Podman on Ubuntu 1804 to interact with containers in a rootless mode. 

There are multiple reasons for this such as old Linux Kernel version (4.15) or lack of fuse-liboverlayfs debian package. 

Because of this, there are differences while configuring and using Podman on Ubuntu 18.04. The biggest difference is that all Podman commands used while interacting with images and containers are executed as root on Ubuntu 18.04.

  1. Install podman

    . /etc/os-release
    echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
    curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/Release.key | sudo apt-key add -
    sudo apt-get update
    sudo apt-get -y upgrade 
    sudo apt-get -y install podman
  2. Verify podman installation

    podman version
    podman info --debug
  3. Configure podman to use Nordix Harbor as proxy to Docker Hub 

    mkdir -p ~/.config/containers
    cat << EOF > ~/.config/containers/registries.conf
    unqualified-search-registries = ["docker.io"]
    
    [[registry]]
    prefix = "docker.io"
    location = "registry.nordix.org/docker-hub-proxy"
    insecure = false
    EOF
  4. Verify podman pulls images through Docker Hub proxy set up on Nordix Harbor. You should be seeing references to registry.nordix.org in the output and image getting pulled from the Nordix Harbor.

    sudo podman pull hello-world --log-level debug
    
    ...snip...
    DEBU[0000] GET https://registry.nordix.org/v2/
    DEBU[0000] Ping https://registry.nordix.org/v2/ status 401
    DEBU[0000] GET https://registry.nordix.org/service/token?scope=repository%3Adocker-hub-proxy%2Flibrary%2Fhello-world%3Apull&service=harbor-registry
    DEBU[0000] GET https://registry.nordix.org/v2/docker-hub-proxy/library/hello-world/manifests/latest
    DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json"
    DEBU[0001] Using blob info cache at /home/ubuntu/.local/share/containers/cache/blob-info-cache-v1.boltdb
    DEBU[0001] Source is a manifest list; copying (only) instance sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042 for current system
    DEBU[0001] GET https://registry.nordix.org/v2/docker-hub-proxy/library/hello-world/manifests/sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042
    DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json"
    DEBU[0001] IsRunningImageAllowed for image docker:docker.io/library/hello-world:latest
    DEBU[0001]  Using default policy section
    DEBU[0001]  Requirement 0: allowed
    DEBU[0001] Overall: allowed
    DEBU[0001] Downloading /v2/docker-hub-proxy/library/hello-world/blobs/sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b
    DEBU[0001] GET https://registry.nordix.org/v2/docker-hub-proxy/library/hello-world/blobs/sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b
    ...snip...
    
    sudo podman images
    REPOSITORY                     TAG     IMAGE ID      CREATED       SIZE
    docker.io/library/hello-world  latest  bf756fb1ae65  9 months ago  20 kB
    
    sudo podman run hello-world
    
    ...snip...
    Hello from Docker!
    This message shows that your installation appears to be working correctly.
    ...snip...
  5. Attempt an image build

    sudo apt-get install -y git
    git clone https://github.com/eiffel-community/eiffel-gerrit-herald.git
    cd eiffel-gerrit-herald
    sudo podman build .
    
    ...snip...
    STEP 8: WORKDIR /app
    --> e1d8ccf9597
    STEP 9: COPY --from=build /temp/build/libs/eiffel-gerrit-herald-*.jar /app/eiffel-gerrit-herald.jar
    STEP 10: COMMIT
    --> 83cd5a06249
    83cd5a06249eff1c4df79edfbd7ef5ee03ea36cd0fd7c0d8e5e835f4d67ab08a
    ...snip...
  6. You should have the image built

    sudo podman images
    REPOSITORY                     TAG     IMAGE ID      CREATED         SIZE
    <none>                         <none>  83cd5a06249e  31 seconds ago  507 MB
    <none>                         <none>  77401bfcd466  59 seconds ago  950 MB
    docker.io/library/hello-world  latest  bf756fb1ae65  9 months ago    20 kB
    docker.io/library/openjdk      12      e1e07dfba89c  13 months ago   477 MB

Instructions for Ubuntu 20.04

These instructions are valid for podman version 2.1.1.

It is assumed that you have right to execute sudo commands.

Please note that these instructions are for running rootless Podman.

  1. Install podman

    . /etc/os-release
    echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
    curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/Release.key | sudo apt-key add -
    sudo apt-get update
    sudo apt-get -y upgrade 
    sudo apt-get -y install podman fuse-overlayfs
  2. Verify podman installation

    podman version
    podman info --debug
  3. Configure podman to use Nordix Harbor as proxy to Docker Hub 

    mkdir -p ~/.config/containers
    cat << EOF > ~/.config/containers/registries.conf
    unqualified-search-registries = ["docker.io"]
    
    [[registry]]
    prefix = "docker.io"
    location = "registry.nordix.org/docker-hub-proxy"
    insecure = false
    EOF
  4. Change storage driver to overlayfs

    cat << EOF > ~/.config/containers/storage.conf
    [storage]
      driver = "overlay"
      runroot = "/run/user/$(id -u)"
      graphroot = "$HOME/.local/share/containers/storage"
      [storage.options]
        mount_program = "/usr/bin/fuse-overlayfs"
    EOF
  5. Verify podman pulls images through Docker Hub proxy set up on Nordix Harbor. You should be seeing references to registry.nordix.org in the output and image getting pulled from the Nordix Harbor. (Please note that IMAGE IDs may be different in the command output)

    podman pull hello-world --log-level debug
    
    ...snip...
    DEBU[0000] GET https://registry.nordix.org/v2/
    DEBU[0000] Ping https://registry.nordix.org/v2/ status 401
    DEBU[0000] GET https://registry.nordix.org/service/token?scope=repository%3Adocker-hub-proxy%2Flibrary%2Fhello-world%3Apull&service=harbor-registry
    DEBU[0000] GET https://registry.nordix.org/v2/docker-hub-proxy/library/hello-world/manifests/latest
    DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json"
    DEBU[0001] Using blob info cache at /home/ubuntu/.local/share/containers/cache/blob-info-cache-v1.boltdb
    DEBU[0001] Source is a manifest list; copying (only) instance sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042 for current system
    DEBU[0001] GET https://registry.nordix.org/v2/docker-hub-proxy/library/hello-world/manifests/sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042
    DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json"
    DEBU[0001] IsRunningImageAllowed for image docker:docker.io/library/hello-world:latest
    DEBU[0001]  Using default policy section
    DEBU[0001]  Requirement 0: allowed
    DEBU[0001] Overall: allowed
    DEBU[0001] Downloading /v2/docker-hub-proxy/library/hello-world/blobs/sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b
    DEBU[0001] GET https://registry.nordix.org/v2/docker-hub-proxy/library/hello-world/blobs/sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b
    ...snip...
    
    podman images
    REPOSITORY                     TAG     IMAGE ID      CREATED       SIZE
    docker.io/library/hello-world  latest  bf756fb1ae65  9 months ago  20 kB
    
    podman run hello-world
    
    ...snip...
    Hello from Docker!
    This message shows that your installation appears to be working correctly.
    ...snip...
  6. Attempt an image build. (Please note that IMAGE IDs may be different in the command output)

    sudo apt-get -y install git
    git clone https://github.com/eiffel-community/eiffel-gerrit-herald.git
    cd eiffel-gerrit-herald
    podman build .
    
    ...snip...
    STEP 8: WORKDIR /app
    --> 2a94a693a48
    STEP 9: COPY --from=build /temp/build/libs/eiffel-gerrit-herald-*.jar /app/eiffel-gerrit-herald.jar
    STEP 10: COMMIT
    --> 9b98923f3a6
    9b98923f3a614348b065ee438a0e135ecc2bf137e0c503916f5b8665bd82c1d1
    ...snip...
  7. You should have the image built. (Please note that IMAGE IDs may be different in the command output)

    podman images
    REPOSITORY                     TAG     IMAGE ID      CREATED             SIZE
    <none>                         <none>  9b98923f3a61  19 seconds ago      507 MB
    <none>                         <none>  893fce3eb5bd  About a minute ago  950 MB
    docker.io/library/hello-world  latest  bf756fb1ae65  9 months ago        20 kB
    docker.io/library/openjdk      12      e1e07dfba89c  13 months ago       477 MB

How Does It Work?

When Podman is installed and configured to use Nordix Harbor as proxy to Docker Hub, podman commands to pull images from Docker Hub, Podman rewrites reference to Nordix Harbor.

This then results in queries for images hitting to proxy on Nordix Harbor first.

If the requested image (and tag) is available on Nordix Harbor, it is served to the client.

If the requested image (and tag) is not available on Nordix Harbor, it gets cached to Nordix Harbor and then served to the client.

And the rest is business as usual.