There are a variety of use-cases that call for the movement of container images between different environments.
Example 1 – Managing Images Across the SDLC
An OpenShift deployment may be divided into non-production and production clusters. In order to promote images between the clusters it is necessary to pull images from the Image Registry of the non-production cluster, and push them to the production cluster.
Example 2 – Distributing Images to Multiple Geos
OpenShift may be deployed anywhere RHEL can be deployed. This opens up the ability to create hybrid private, public cloud environments across a number of different geographical locations. Customer may wish to leverage public cloud availability in specific geos to provide low-latency resources to their customers living and working in those locations, for example.
Trying it out with OpenShift
In preparing this guide I developed the following examples using OpenShift Enterprise 3.1.1, and the Red Hat Container Development Kit beta 5, with my local Docker instance running on Fedora 23. Where possible I’ve added notes regarding differences you may encounter on other versions or platforms.
There are a few steps needed to get this working:
- Expose OpenShift’s Docker Registry, to make it available to external systems
- We may need to allow our local Docker daemon to access insecure registries
- Create, or identify, a service account with sufficient access rights
- Prepare local images for pushing to OpenShift
Making OpenShift’s Docker Registry Externally Accessible
Prepare the docker registry, by default, the registry is not externally accessible
$ oc expose service docker-registry -n default
|Note: if you’re using the CDK the docker-registry service should already be exposed as hub.openshift.10.1.2.2.xip.io|
Make sure the route name is resolvable by the external system. For example:
|Tip: use xip.io to create resolvable routes. Find the IP address of the router, create a route using this IP address and xip.io, for example:docker-registry.192.168.121.113.xip.io|
Your local docker registry needs to be configured to accept communication with this registry, by default it will be listening on port 80 and be insecure (you may be required to provide a secured registry in which case I recommend following the OpenShift documentation on Accessing The Registry Directly). To allow Docker to communicate with an insecure registry add the –insecure-registry option to your docker daemon service configuration, and include the port specifier. For example on Fedora, Centos or RHEL, edit /etc/sysconfig/docker on your local machine.
# /etc/sysconfig/docker # Modify these options if you want to change the way the docker daemon runs OPTIONS='--add-registry=docker.io --log-driver=journald --insecure-registry docker-registry.192.168.121.113.xip.io:80' #OPTIONS='--selinux-enabled --log-driver=journald' DOCKER_CERT_PATH=/etc/docker ...
|Tip: If you’re using Docker Machine on Windows, or Mac, you will find a way to configure an insecure registry in the configuration file here: <USER_HOME>/.docker/machine/machines/<MACHINE_NAME>/config.json
Under EngineOptions key
Once you’ve updated your local docker configuration, you will need to restart the local docker daemon.
$ systemctl restart docker
Preparing an OpenShift Project to Accept Pushes
In order to access the Docker Registry remotely it is necessary to provide access credentials. These credentials could be based on your personal user account, but for automation approaches, for example using an external Jenkins service, it’s recommended that a Service Account (SA) is used.
You could use the ‘builder’ SA within a project, but let’s create a new SA for this role.
$ oc new-project pushed
Creating a new SA in OpenShift Origin, and OpenShift 3.2 has been simplified to the following command:
$ oc create serviceaccount pusher
For older versions of OpenShift, use the following method:
$ oc create -f - << API apiVersion: v1 kind: ServiceAccount metadata: name: pusher API
To pull images users need to get imagestreams/layers
To push images users need to update imagestreams/layers
The system:image-puller role will just provide pull capability.
The system:image-builder role allows both pull and push capability.
Note: the system:image-builder role can only be given by cluster-admins, for project admins, the edit role will provide sufficient access.
Add the system:image-builder, or edit role to the SA.
For cluster admins
$ oc policy add-role-to-user system:image-builder system:serviceaccount:pushed:pusher
For project admins
$ oc policy add-role-to-user edit system:serviceaccount:pushed:pusher
We will need the SA’s token
$ oc describe sa pusher Name: pusher Namespace: pushed Labels: <none> Mountable secrets: pusher-token-fxg2k pusher-dockercfg-vwddo Tokens: pusher-token-98rix pusher-token-fxg2k Image pull secrets: pusher-dockercfg-vwddo
Select one of the token names, and discover its value
$ oc describe secret pusher-token-98rix Name: pusher-token-98rix Namespace: pushed Labels: <none> Annotations: kubernetes.io/service-account.name=pusher,kubernetes.io/service-account.uid=0bd3b6e0-f655-11e5-aca3-525400723ce5 Type: kubernetes.io/service-account-token Data ==== ca.crt: 1066 bytes token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiw… ETq0W3G3wHiO1Rdg4nsk7R8k-VJ8Qk-2V7elYVz8LYsNpkd1PGQGE5Jtegnr5GoFDk5wKxzA7GT1zXt2vVg
You will also need an image stream to accept the image to be pushed.
A fully qualified docker image name should conform to the following format:
When mapping to OpenShift,
Create the image stream
$ oc create -f - <<API apiVersion: v1 kind: ImageStream metadata: annotations: description: Keeps track of changes in the application image name: myimage API
The project is now ready to accept pull and push requests from the “pusher” service account.
Pushing an Image
You will need to login to the remote repository in order to push to it.
$ docker login --username=anything --email@example.com --password=<token> <docker registry route>
$ docker login --username=ed --firstname.lastname@example.org docker-registry.192.168.121.113.xip.io:80 password: <paste SA token value>
|Tip: if you prefer to use your personal account instead of a Service Account, you can discover your token using the oc whoam -t command.|
Once logged in, you will need to tag your image.
$ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE <none> latest 445f4be0f46f 6 days ago 392.1 MB $ docker tag 445f4be0f46f docker-registry.192.168.121.113.xip.io:80/pushed/myimage:latest
And once tagged, the image can now be pushed to OpenShift.
$ docker push docker-registry.192.168.121.113.xip.io:80/pushed/myimage:latest
You may then check the status of the associated imagestream in OpenShift
$ oc describe is myimage Name: myimage Created: About an hour ago Labels: <none> Description: Keeps track of changes in the application image Annotations: openshift.io/image.dockerRepositoryCheck=2016-03-30T10:28:30Z Docker Pull Spec: 172.30.55.172:5000/pushed/myimage Tag Spec Created PullSpec Image latest <pushed> 17 seconds ago 172.30.55.172:5000/pushed/myimage@sha256:4ed6572d87aa07e9ac044421c30e6e6bd47349c6391ea6f2efdde9a2c2e9fcba
Pulling an Image
Pulling an image to a local Docker instance is simply a task of using the fully qualified image name from the remote OSE repository, for example:
$ docker pull docker-registry.192.168.121.113.xip.io:80/pushed/myimage
As you would expect you can also reference remote images in OpenShift to deploy as applications. In the diagram below the images are pulled directly into the OpenShift Production Cluster, rather than being source from the cluster’s own Image Registry.
Normally we would create an ImageStream to present the image to OpenShift Production Cluster projects, however, in this example we will use a Deployment Configuration to reference the image directly.
|Note: in order for this to work, the remote docker registry must be secured under SSL and the target OpenShift Cluster must trust the SSL certificate.|
apiVersion: v1 kind: DeploymentConfig metadata: name: myapp spec: replicas: 1 ... template: ... spec: containers: ... image: docker-registry.192.168.121.113.xip.io:80/pushed/myimage:v1.1 imagePullPolicy: IfNotPresent name: myimage ...
We still need authorise the OpenShift Production Cluster project so that it can access the remote Image Registry. We do this by created pull secrets for the service account in the destination project.
$ oc secrets new-dockercfg <pull_secret_name> --docker-server=<registry_server> --docker-username=<user_name> --docker-password=<password> --docker-email=<email>
- <pull secret name> – an arbitrary label
- <registry_server> – the remote registry server, for example: docker-registry.192.168.121.113.xip.io:80
- <user_name> – an appropriate username, if accessing an OpenShift Image Registry this can be anything.
- <password> – the associated password, or if accessing an OpenShift Image Registry this is the authorised service account token from the source project.
- <email> – any valid email address.
The secret then needs to be attached to the service account that will deploy container, in OpenShift this is normally the default service account.
$ oc secrets add serviceaccount/default secrets/<pull_secret_name> --for=pull
For more information on this see Image Pull Secrets in the OpenShift documentation.
In OpenShift 3.2 it will be possible to use the oc import-image command to automatically create an ImageStream and sync with an external authenticated repository. The ImageStream will be able to use the docker pull secret created above.
$ oc import-image myimage --from=docker-registry.192.168.121.113.xip.io:80 --confirm --insecure-repository=true
If, as in this example, we are using an unsecured registry, it was necessary to add the –insecure-repository option. If you view the resulting ImageStream configuration, you’ll notice the added annotation.
apiVersion: v1 kind: ImageStream metadata: annotations: openshift.io/image.insecureRepository: "true" name: myimage ...