When explaining Kubernetes to people new in the space I noticed that the concept of services is often not well understood. To help you better understand what services are and how you can troubleshoot them, we will have a look at a concrete setup and discuss the inner workings of services in this post.

In a nutshell, Kubernetes services are an abstraction for pods, providing a stable, virtual IP (VIP) address. As pods may come and go, for example in the process of a rolling upgrade, services allow clients to reliably connect to the containers running in the pods, using the VIP. The virtual in VIP means it's not an actual IP address connected to a network interface but its purpose is purely to forward traffic to one or more pods. Keeping the mapping between the VIP and the pods up-to-date is the job of kube-proxy, a process that runs on every node, which queries the API server to learn about new services in the cluster.

Enough theory, let's have a look at a concrete example: in the following I'm using a simple containerized HTTP service that exposes endpoint0/ at port 9876, returning a JSON payload echoing the server it is serving from along with some other infos.

Initially, let's deploy a pod under the supervision of a ReplicationController along with a Service, resulting in:

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
sise-bkvhs 1/1 Running 0 9m

$ kubectl describe pod sise-bkvhs
Name: sise-bkvhs
Namespace: namingthings
Security Policy: restricted
Node: 192.168.99.100/192.168.99.100
Start Time: Tue, 18 Apr 2017 09:45:22 +0100
Labels: app=sise
Status: Running
IP: 172.17.0.2
Controllers: ReplicationController/sise
...

You can now, from within the cluster, access the pod directly via its assigned IP 172.17.0.2:

$ curl 172.17.0.2:9876/endpoint0
{"host": "172.17.0.2:9876", "version": "0.4.0", "result": "all is well"}

This is however, as mentioned above, not advisable since the IPs assigned to pods may change. Enter services:

$ kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
simpleservice 172.30.40.155 80/TCP 1m

$ kubectl describe svc simpleservice
Name: simpleservice
Namespace: namingthings
Labels: name=simpleservice
Selector: app=sise
Type: ClusterIP
IP: 172.30.40.155
Port: 80/TCP
Endpoints: 172.17.0.2:9876
Session Affinity: None
No events.

From within the cluster we can now access simpleservice like so:

$ curl 172.30.40.155:80/endpoint0
{"host": "172.30.40.155", "version": "0.4.0", "result": "all is well"}

So far so go. But what makes the VIP 172.30.40.155 forward the traffic to the pod? The answer is: IPTables. Think of IPTables as a long list of rules that tells the Linux kernel what to do with a certain IP package.

Looking at the rules that concern our service (executed on a cluster node) yields:

$ sudo iptables-save | grep simpleservice
-A KUBE-SEP-ASIN52LB5SMYF6KR -s 172.17.0.2/32 -m comment --comment "namingthings/simpleservice:" -j KUBE-MARK-MASQ
-A KUBE-SEP-ASIN52LB5SMYF6KR -p tcp -m comment --comment "namingthings/simpleservice:" -m tcp -j DNAT --to-destination 172.17.0.2:9876
-A KUBE-SERVICES -d 172.30.40.155/32 -p tcp -m comment --comment "namingthings/simpleservice: cluster IP" -m tcp --dport 80 -j KUBE-SVC-IKIIGXZ2IBFIBYI6
-A KUBE-SVC-IKIIGXZ2IBFIBYI6 -m comment --comment "namingthings/simpleservice:" -j KUBE-SEP-ASIN52LB5SMYF6KR

Above you can see the four rules that kube-proxy has thankfully added to the routing table, essentially stating that TCP traffic to 172.30.40.155:80 should be forwarded to 172.17.0.2:9876, which is our pod.

Let's now add a second pod by scaling up the ReplicationController supervising it:

$ kubectl scale --replicas=2 rc/sise
replicationcontroller "sise" scaled
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
sise-bkvhs 1/1 Running 0 11m
sise-vncz4 1/1 Running 0 8s

When we now check the relevant parts of the routing table again we notice the addition of a bunch of rules:

$ sudo iptables-save | grep simpleservice
-A KUBE-SEP-ASIN52LB5SMYF6KR -s 172.17.0.2/32 -m comment --comment "namingthings/simpleservice:" -j KUBE-MARK-MASQ
-A KUBE-SEP-ASIN52LB5SMYF6KR -p tcp -m comment --comment "namingthings/simpleservice:" -m tcp -j DNAT --to-destination 172.17.0.2:9876
-A KUBE-SEP-RP53IYKEFRDLQANZ -s 172.17.0.4/32 -m comment --comment "namingthings/simpleservice:" -j KUBE-MARK-MASQ
-A KUBE-SEP-RP53IYKEFRDLQANZ -p tcp -m comment --comment "namingthings/simpleservice:" -m tcp -j DNAT --to-destination 172.17.0.4:9876
-A KUBE-SERVICES -d 172.30.40.155/32 -p tcp -m comment --comment "namingthings/simpleservice: cluster IP" -m tcp --dport 80 -j KUBE-SVC-IKIIGXZ2IBFIBYI6
-A KUBE-SVC-IKIIGXZ2IBFIBYI6 -m comment --comment "namingthings/simpleservice:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-ASIN52LB5SMYF6KR
-A KUBE-SVC-IKIIGXZ2IBFIBYI6 -m comment --comment "namingthings/simpleservice:" -j KUBE-SEP-RP53IYKEFRDLQANZ

In above routing table listing we see rules for the newly created pod serving at 172.17.0.4:9876 as well as an additional rule -A KUBE-SVC-IKIIGXZ2IBFIBYI6 -m comment --comment "namingthings/simpleservice:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-ASIN52LB5SMYF6KR which causes the traffic to the service being equally split between our two pods by invoking the statistics module of IPTables.

Summing up, this is our final setup:

Notice in above figure the different IP ranges for the pods, the service VIP as well as the node IP (192.168.99.100 in our example).

If you'd like to try out the above steps yourself, you can simply use the same setup I've used, that is, Minishift running locally on your machine. Ah, and before I forget it: kubectl on my machine is actually a symlink to oc, the OpenShift CLI client:

$ stat kubectl ; ls -l kubectl
File: "kubectl"
Size: 2 FileType: Symbolic Link
Mode: (0755/lrwxr-xr-x) Uid: ( 501/mhausenblas) Gid: ( 20/ staff)
Device: 1,4 Inode: 3685078 Links: 1
Access: Fri Apr 14 10:21:59 2017
Modify: Fri Apr 14 10:21:59 2017
Change: Fri Apr 14 10:21:59 2017
lrwxr-xr-x 1 mhausenblas staff 2 14 Apr 10:21 kubectl -> oc

Should you be interested in digging deeper into the topic of Kubernetes services and how to overcome scalability issues with large-scale service deployments—due to the inherent limitations of IPTables—check out the following resources:

I hope this post sheds some light on the topic of services in Kubernetes, what they do and how they work. Let me know what you think and which other Kubernetes objects we should have a look at in future posts.