Remotely Push and Pull Container Images to OpenShift

There are a variety of use-cases that call for the movement of container images between different environments.

Example 1 – Managing Images Across the SDLC

An OpenShift deployment may be divided into non-production and production clusters. In order to promote images between the clusters it is necessary to pull images from the Image Registry of the non-production cluster, and push them to the production cluster.

image01

Example 2 – Distributing Images to Multiple Geos

OpenShift may be deployed anywhere RHEL can be deployed. This opens up the ability to create hybrid private, public cloud environments across a number of different geographical locations. Customer may wish to leverage public cloud availability in specific geos to provide low-latency resources to their customers living and working in those locations, for example.

image00

Trying it out with OpenShift

In preparing this guide I developed the following examples using OpenShift Enterprise 3.1.1, and the Red Hat Container Development Kit beta 5, with my local Docker instance running on Fedora 23. Where possible I’ve added notes regarding differences you may encounter on other versions or platforms.

There are a few steps needed to get this working:

  1. Expose OpenShift’s Docker Registry, to make it available to external systems
  2. We may need to allow our local Docker daemon to access insecure registries
  3. Create, or identify, a service account with sufficient access rights
  4. Prepare local images for pushing to OpenShift

Making OpenShift’s Docker Registry Externally Accessible

Prepare the docker registry, by default, the registry is not externally accessible

$ oc expose service docker-registry -n default

 

Note: if you’re using the CDK the docker-registry service should already be exposed as hub.openshift.10.1.2.2.xip.io

 

Make sure the route name is resolvable by the external system. For example:

docker-registry-default.apps.example.com

 

Tip: use xip.io to create resolvable routes. Find the IP address of the router, create a route using this IP address and xip.io, for example:docker-registry.192.168.121.113.xip.io

 

Your local docker registry needs to be configured to accept communication with this registry, by default it will be listening on port 80 and be insecure (you may be required to provide a secured registry in which case I recommend following the OpenShift documentation on Accessing The Registry Directly). To allow Docker to communicate with an insecure registry add the –insecure-registry option to your docker daemon service configuration, and include the port specifier. For example on Fedora, Centos or RHEL, edit /etc/sysconfig/docker on your local machine.

# /etc/sysconfig/docker


# Modify these options if you want to change the way the docker daemon runs

OPTIONS='--add-registry=docker.io --log-driver=journald --insecure-registry docker-registry.192.168.121.113.xip.io:80'

#OPTIONS='--selinux-enabled --log-driver=journald'

DOCKER_CERT_PATH=/etc/docker

...

 

Tip: If you’re using Docker Machine on Windows, or Mac, you will find a way to configure an insecure registry in the  configuration file here:  <USER_HOME>/.docker/machine/machines/<MACHINE_NAME>/config.json

Under EngineOptions key

 

Once you’ve updated your local docker configuration, you will need to restart the local docker daemon.

$ systemctl restart docker

Preparing an OpenShift Project to Accept Pushes

In order to access the Docker Registry remotely it is necessary to provide access credentials. These credentials could be based on your personal user account, but for automation approaches, for example using an external Jenkins service, it’s recommended that a Service Account (SA) is used.

You could use the ‘builder’ SA within a project, but let’s create a new SA for this role.

$ oc new-project pushed

Creating a new SA in OpenShift Origin, and OpenShift 3.2 has been simplified to the following command:

$ oc create serviceaccount pusher

For older versions of OpenShift, use the following method:

$ oc create -f - << API

apiVersion: v1

kind: ServiceAccount

metadata:

  name: pusher

API

 

To pull images users need to get imagestreams/layers

To push images users need to update imagestreams/layers

The system:image-puller role will just provide pull capability.

The system:image-builder role allows both pull and push capability.

Note: the system:image-builder role can only be given by cluster-admins, for project admins, the edit role will provide sufficient access.

 

Add the system:image-builder, or edit role to the SA.

 

For cluster admins

$ oc policy add-role-to-user system:image-builder 

system:serviceaccount:pushed:pusher

 

For project admins

$ oc policy add-role-to-user edit 

system:serviceaccount:pushed:pusher

 

We will need the SA’s token

$ oc describe sa pusher

Name: pusher

Namespace: pushed

Labels: <none>


Mountable secrets:  pusher-token-fxg2k

                    pusher-dockercfg-vwddo


Tokens:             pusher-token-98rix

                    pusher-token-fxg2k



Image pull secrets: pusher-dockercfg-vwddo

 

Select one of the token names, and discover its value

$ oc describe secret pusher-token-98rix

Name: pusher-token-98rix

Namespace: pushed

Labels: <none>

Annotations: kubernetes.io/service-account.name=pusher,kubernetes.io/service-account.uid=0bd3b6e0-f655-11e5-aca3-525400723ce5


Type: kubernetes.io/service-account-token


Data

====

ca.crt: 1066 bytes

token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiw…

ETq0W3G3wHiO1Rdg4nsk7R8k-VJ8Qk-2V7elYVz8LYsNpkd1PGQGE5Jtegnr5GoFDk5wKxzA7GT1zXt2vVg

 

You will also need an image stream to accept the image to be pushed.

 

A fully qualified docker image name should conform to the following format:

<registry>/<repo>/<image>:<tag>

 

When mapping to OpenShift,

<registry>/<project>/<imagestream>:<tag>

 

Create the image stream

$ oc create -f - <<API

apiVersion: v1

kind: ImageStream

metadata:

  annotations:

    description: Keeps track of changes in the application image

  name: myimage

API

 

The project is now ready to accept pull and push requests from the “pusher” service account.

Pushing an Image

You will need to login to the remote repository in order to push to it.

 

$ docker login --username=anything --email=anyone@anywhere.com --password=<token> <docker registry route>

 

For example:

$ docker login --username=ed --email=eseymour@redhat.com 

 docker-registry.192.168.121.113.xip.io:80

password: <paste SA token value>

 

Tip: if you prefer to use your personal account instead of a Service Account, you can discover your token using the oc whoam -t command.

 

Once logged in, you will need to tag your image.

$ docker images

REPOSITORY                                            TAG                 IMAGE ID            CREATED             VIRTUAL SIZE

<none>                                            latest              445f4be0f46f        6 days ago          392.1 MB


$ docker tag 445f4be0f46f 

docker-registry.192.168.121.113.xip.io:80/pushed/myimage:latest

 

And once tagged, the image can now be pushed to OpenShift.

$ docker push docker-registry.192.168.121.113.xip.io:80/pushed/myimage:latest

 

You may then check the status of the associated imagestream in OpenShift

$ oc describe is myimage

Name: myimage

Created: About an hour ago

Labels: <none>

Description: Keeps track of changes in the application image

Annotations: openshift.io/image.dockerRepositoryCheck=2016-03-30T10:28:30Z

Docker Pull Spec: 172.30.55.172:5000/pushed/myimage

 

Tag Spec Created PullSpec Image

latest <pushed> 17 seconds ago 172.30.55.172:5000/pushed/myimage@sha256:4ed6572d87aa07e9ac044421c30e6e6bd47349c6391ea6f2efdde9a2c2e9fcba

Pulling an Image

Pulling an image to a local Docker instance is simply a task of using the fully qualified image name from the remote OSE repository, for example:

$ docker pull docker-registry.192.168.121.113.xip.io:80/pushed/myimage

As you would expect you can also reference remote images in OpenShift to deploy as applications. In the diagram below the images are pulled directly into the OpenShift Production Cluster, rather than being source from the cluster’s own Image Registry.

image02

Normally we would create an ImageStream to present the image to OpenShift Production Cluster projects, however, in this example we will use a Deployment Configuration to reference the image directly.

Note: in order for this to work, the remote docker registry must be secured under SSL and the target OpenShift Cluster must trust the SSL certificate.
apiVersion: v1
kind: DeploymentConfig
metadata:
  name: myapp
spec:
  replicas: 1
  ...
  template:
    ...
    spec:
      containers:
        ...
        image: docker-registry.192.168.121.113.xip.io:80/pushed/myimage:v1.1
        imagePullPolicy: IfNotPresent
        name: myimage
        ...

We still need authorise the OpenShift Production Cluster project so that it can access the remote Image Registry. We do this by created pull secrets for the service account in the destination project.

$ oc secrets new-dockercfg <pull_secret_name> 

    --docker-server=<registry_server> --docker-username=<user_name> 

    --docker-password=<password> --docker-email=<email>
  • <pull secret name> – an arbitrary label
  • <registry_server> – the remote registry server, for example: docker-registry.192.168.121.113.xip.io:80
  • <user_name> – an appropriate username, if accessing an OpenShift Image Registry this can be anything.
  • <password> – the associated password, or if accessing an OpenShift Image Registry this is the authorised service account token from the source project.
  • <email> – any valid email address.

 

The secret then needs to be attached to the service account that will deploy container, in OpenShift this is normally the default service account.

$ oc secrets add serviceaccount/default secrets/<pull_secret_name> --for=pull

For more information on this see Image Pull Secrets in the OpenShift documentation.

In OpenShift 3.2 it will be possible to use the oc import-image command to automatically create an ImageStream and sync with an external authenticated repository. The ImageStream will be able to use the docker pull secret created above.

$ oc import-image myimage --from=docker-registry.192.168.121.113.xip.io:80 --confirm --insecure-repository=true

 

If, as in this example, we are using an unsecured registry, it was necessary to add the –insecure-repository option. If you view the resulting ImageStream configuration, you’ll notice the added annotation.

apiVersion: v1
kind: ImageStream
metadata:
  annotations:
    openshift.io/image.insecureRepository: "true"
  name: myimage
...
Categories
OpenShift Container Platform, OpenShift Origin
Tags
,

6 Responses to “Remotely Push and Pull Container Images to OpenShift”

  1. Srinivas Kotaru

    Instructions could have better using live examples. This is very important topic and lot of confusion on how to promote images across clusters. I’m expecting little more description and detailed step by step instructions.

  2. Jason Baik

    Thx! This helped me a lot when working with my company’s cluster. One question I have is, is there a documentation that lists out all the permissions and roles for OpenShift? It would’ve been impossible for me to figure out on my own that I’d need the permissions below to do the image push/pull.

    “To pull images users need to get imagestreams/layers
    To push images users need to update imagestreams/layers
    The system:image-puller role will just provide pull capability.
    The system:image-builder role allows both pull and push capability.”

  3. Amir

    Thanks Ed.
    I used your post for a rather simple use case:
    pushing images to the internal registry as I have a disconnected/offline installation.
    The docker login–> docker push path did not work out for me.
    oc import-image did the job.

    Many Thanks, Amir

  4. Кирилл Семаев

    Guys, specified steps work on standalone configuration, when we have only one master (and node at same server). When I try to do this in openshift cluster – only master is able to pull images, while nodes are waiting for me to manually push needed images with docker pull.
    How can I make nodes to pull images from remote registry automatically (as my master does)?

  5. erikb85

    First you start creating a user “pusher” and then you suddenly login as “anyone” or “ed”?

  6. Jaseem Abid

    Did this API change recently? On Origin v3.6 I get the error “error: services “docker-registry” not found”

Comments are closed.