As part of my exploration of Kubernetes, while working on a project I wanted to execute commands inside a pod. Rather then forcing the container to have some specific behaviour, I wanted to utilize the API mechanism exposed as the
kubectl exec subcommand. While investigating, I found that
exec doesn’t yet sport extensive documentation, and hopefully this post will help those who find themselves in a similar situation.
The Kubernetes docs do not mention the
exec endpoint, but OpenShift’s documentation does offer some basic information. From that, we know where the endpoint lives and what parameters we need to pass.
We need to issue POST requests to this path.
the two strings that need to replaced in the path are fairly obvious, and the query string parameters are described correctly in the table in the OpenShift documentation, with a single exception: The
command parameter can be included multiple times.
First, let’s take a look at what it looks like for a single
command parameter, one that will simply execute
/bin/bash in the pod:
Now for multiple command parameters:
which gives us something like [‘/bin/bash’, ‘-c’, ‘/bin/bash’] which could be logically transcribed as
/bin/bash -c “/bin/bash”.
oc use the
SPDY protocol at the moment, which is being deprecated . The second option is to use Websockets, which seems to be the best way. Anyway one of these two protocols, SPDY or WebSockets, is required for communication with this endpoint, and the API will refuse requests without
To provide all the necessary information, the request needs to contain the set of headers required by the API. Some of them will be handled by your WebSockets client (e.g.
Upgrade, etc.), but there are two that need to be provided by the user.
The first one is
Authorization, with a value of
Bearer <token> that authenticates the request. For Kubernetes, follow this guide. With OpenShift, simply get the token for your user:
oc whoami -t
The other header is
Accept, with the value
*/*. Any other value will be rejected with
406 Not Acceptable, even though the example shown above in the documentation shows the incorrect value of
application/json. There is an issue in progress to make the documentation accurate.
With all the information in place, the WebSocket should be able to establish a connection and the API will start communicating. When you write to the WebSocket, the data will be passed to standard input (
stdin) and on the receiving end of the WebSocket will be standard output (
stdout) and error (
stderr). The API defines a simple protocol to multiplex
stderr over a single connection. Every message passed through the web socket is prefixed by a single byte that defines which stream the message belongs to.
|Code|Meaning | |----|--------| |0 | stdin | |1 | stdout | |2 | stderr |
So for every message received over the socket, you need to get the first byte and decide whether it is
stderr. In Ruby, this would look something like:
data = [1, 27, 91, 63, 49, 48, 51, 52, 104, 98, 97, 115, 104, 45, 52, 46, 50, 36, 32] case data.shift when 1 $stdout << data.pack('C*').force_encoding('utf-8') when 2 $sterr << data.pack('C*').force_encoding('utf-8') else unknown_data(data) end
To send data to the API, you need to convert to bytes and prepend
0 to indicate the message belongs in the
data = ‘ls -la\n’ data = data.unpack(‘C*’) # [108, 115, 32, 45, 108, 97, 13] socket.send(data.unshift(0))
One last problem is that there may be proxies and other “roadblocks” on the way to the API, or you may simply reach the TCP timeout. To get around that, send an empty message every once in a while to keep the connection busy:
Thread.new loop do socket.send() sleep(30) end end
With this information, it should be possible to write your own application to communicate through the Kubernetes API with processes running inside your Kubernetes clusters. The sample Ruby excerpts have been tested on OpenShift 3.7.1, using minishift.
$ oc version openshift v3.7.1+282e43f-42 kubernetes v1.7.6+a08f5eeb62
While the examples use Ruby, it should be straightforward to translate them into your favourite language.
If you can read Go, you can check how the endpoint is used by
kubectl itself in the [upstream source code] (https://github.com/kubernetes/kubernetes/blob/release-1.7/pkg/kubectl/cmd/exec.go).