OpenShift and “manifest unknown”

Docker is a surprisingly fast-moving open source project. As with all open-source projects, at some point a backwards-incompatible changes becomes necessary. The OpenShift community recently finds itself facing just such a change. On the surface, the problem can be confusing. Here are some more details.

A New Release, A New Schema

Around the Docker 1.10 release, a schema change was introduced in how metadata is provided and stored in the Docker registry (v2 schema). At present, OpenShift is built around the v1 schema, which causes some issues. Images that are built with Docker 1.10 and pushed to newer registries will store data with the v2 schema by default. When OpenShift tries to inspect a remote registry, for example, Docker Hub, it sometimes receives v2 schema data that is incompatible with what it is expecting. This usually shows up as an error in OpenShift’s events that say something like “manifest unknown”.
For those who are trying to consume publicly-available images from the Docker Hub on OpenShift, this can be a little frustrating. Fortunately, there is a relatively simple workaround until the next release of OpenShift fixes this issue. The following assumes you are using a Linux-like operating system that can redirect CLI output and so on and so forth. For this exercise we will use Docker’s official memcached image.

Generating the Objects

The “new-app” subcommand in the “oc” CLI on OpenShift is very handy for quickly getting an existing Docker image up and running. “new-app” by default will simply create all of the objects immediately and show you what happened. However, if we tell the subcommand to generate different output with the flag “-o yaml” we actually are returned a serialized set of the objects that would be created. Nothing actually gets created besides text output to the screen.
We can redirect the “new-app” output to a temporary local file:

oc new-app -o yaml > memcached.yml

Then, we can open the file in a text editor. If you are familiar with the YAML syntax, you’ll notice that the file is simply a list (array) of OpenShift data objects — DeploymentConfig, Service, and so on and so forth.

Remove the ImageStream

ImageStreams are a unique feature of OpenShift. They essentially are a way for OpenShift to keep track of the stream of changes to a Docker image. For example, when new tags are created, or when existing tags are updated, the ImageStream is keeping track of those things. Unfortunately, this is where the “break” occurs with Docker’s new schema. The ImageStream expects metadata in a certain way and doesn’t get it.

ImageStreams are not required to use OpenShift. Their primary function is so that OpenShift can then automatically act when the tracked image changes. For example, triggering a new deployment. In this case, we are trying to work around getting the image running, so we’ll have to forego automatic deployments on new images for now.

In the YAML file, remove the ImageStream definition. The block of YAML to remove starts with:

- apiVersion: v1
  kind: ImageStream
    dockerImageRepository: ""

You want to remove everything up until (NOT INCLUDING) the next “- apiVersion” line that is associated with another object — the dockerImageRepository stanza is the last line to remove.

Update the DeploymentConfig

The DeploymentConfig object tells OpenShift the configuration for how to deploy something. One of the things in the DeploymentConfig is the template for the Pod, which details the containers that will be run and from what image they are derived. In your text editor, you will see the DeploymentConfig has a section that talks about image change:

    - imageChangeParams:
        automatic: true
        - memcached
          kind: ImageStreamTag
          name: memcached:1.4.30
      type: ImageChange

As we discussed above, ImageStreams are used to track images in registries, and, because of the schema problem, we can’t do that. So, simply disable the image change trigger by removing this entire section.

Save, Exit, Create

At this point you can simply save the file, exit your editor, and then use “oc create” to create all of the objects:

oc create -f memcached.yml

There’s just two objects:

deploymentconfig "memcached" created
service "memcached" created


That’s really all there is to it. There’s a simple workaround to the schema change. The upcoming release of OpenShift should include both the newer registry (that supports the V2 schema) as well as an updated Docker daemon along with the fixes for ImageStreams to be able to consume that V2 schema data.
If, at this point, you wanted this “application” to be externally accessible, you could use “oc expose” to create a Route. But that’s a story for a different day.

Coming Soon: In a future update we will add all the required steps to repeat this process for for builder images.

News, OpenShift Origin

One Response to “OpenShift and “manifest unknown””

  1. Preston Gilchrist

    We are having this problem except that is manifests itself with an error about a missing signature. If you no longer use image streams, what is the recommended way to trigger an update of the containers when a new image is available?

Comments are closed.