Use Network Policy Objects to restrict traffic flow between application components or microservices with a demonstration from this video and post. Network Policy Objects allow you to define a policy to determine when traffic is allowed to flow to specific services and prevent traffic to other services.
As of the writing of this article, this feature is in “Technology Preview” in OpenShift 3.6.
To test this concept, you have to install an OpenShift cluster with
os_sdn_network_plugin_name in the Ansible hosts file or enable network policy in an existing cluster as explained in https://docs.openshift.com/container-platform/3.6/admin_guide/managing_networking.html#admin-guide-networking-networkpolicy
Understanding the Sample Application
To look at a practical example that showcases Network Policy Objects, we’ll use a sample application that is made up of a group of microservices from this Git repository: https://github.com/VeerMuchandi/microservices-on-openshift.
Deploying these microservices will provide you a use case for understanding how Network Policy Objects are useful.
This application has four microservices written using different technologies just to showcase how polyglot microservices talk to each other. Some of these microservices have application logic and a database. All the components run as containers on OpenShift.
- Front-end Service that provides a UI to register users, allows you to login, and displays the tweets for all the registered users, written in PHP.
- Email Service that can send emails to a registered user, written in Python, that saves data in MySQLdb.
- Twitter Service that pulls tweets for a registered user, written in Java running on Tomcat. This service is called by the Front-end HTML directly.
- User Registration Backend that registers the users, saves the registered user in a MongoDB, and is written in Node.Js. This service is called by the Front-end HTML and does the job of user registration. Once the user is registered, it calls the Email service to send an email confirmation.
This is all explained in this application’s Git repository. For our network policy use case, we will deploy these microservices into three different projects:
msclientproject hosts the front-end service.
msservicesproject hosts the user-registration back-end service and the Twitter service.
msinfraproject hosts the email service.
The deployment model for these microservices and the expected interations (in order) are shown in the figure below. I recommend spending some time on this model to get a better understanding.
Deploy the application
- For Email Service to work, you will need a Gmail account to send emails. You will also have to turn on access for less secure apps on this Gmail account so that the application can log in and send emails. Caution: Don’t use your personal Gmail. Create a new one that you don’t use so that you don’t care if it is hacked.
- For Twitter Service to work, you will need access to keys and tokens to work with Twitter. Refer to the documentation at https://dev.twitter.com/oauth/overview/application-owner-access-tokens for more information.
Step 1: Clone the repository to your local machine:
git clone https://github.com/VeerMuchandi/microservices-on-openshift
and change to the
Step 2: Create three projects (after you have already logged into the OpenShift cluster):
oc new-project msclient oc new-project msservices oc new-project msinfra
Step 3: Set environment variables
Edit 1.setVariable.sh script to set the following values:
OSE_DOMAINto your OpenShift domain name
FROM_GMAILto your Gmail account from where you want to send Gmails
FROM_GMAIL_PASSWORDpassword for the above Gmail account
- And values for
Run the script as follows
Step 4: Deploy microservices by running the following scripts:
This script will build and deploy Email Service and the MySQLdb in the
This script will build and deploy Twitter Service in the
This script will build and deploy User Registration Back-end Service in the
This script will build and deploy Front-end Service in the
In a few minutes, you will notice all the services up and running. You should see the following:
$ oc get pods -n msclient NAME READY STATUS RESTARTS AGE userreg-1-build 0/1 Completed 0 1m userreg-1-tm3x8 1/1 Running 0 1m $ oc get pods -n msservices NAME READY STATUS RESTARTS AGE mongodb-1-2zx66 1/1 Running 0 2m twitter-api-1-build 0/1 Completed 0 2m twitter-api-1-srvvm 1/1 Running 0 1m userregsvc-1-b4xq3 1/1 Running 0 1m userregsvc-1-build 0/1 Completed 0 2m $ oc get pods -n msinfra NAME READY STATUS RESTARTS AGE emailsvc-1-build 0/1 Completed 0 3m emailsvc-1-grb00 1/1 Running 0 2m mysql-2-ffj05 1/1 Running 0 3m $ oc get route -n msclient NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD userreg userreg-msclient.apps.devday.ocpcloud.com userreg 8080-tcp None
Step 5: Test your application
Access the UI for the frontend and register a few users. In my case, the URL is http://userreg-msclient.apps.devday.ocpcloud.com. If email is configured correctly the user that is registered should receive an email. Also, you should be able to pull the tweets for the configured user.
So far, we’ve deployed a few microservices and used them on OpenShift.
Now we will see why we need Network Policy Objects
We have a hack in the Front-end Service code. If you access your <>/hack.php, it talks to the MySQLdb in the
msinfra project and pulls all the data and shows it as below.
$ curl http://userreg-msclient.apps.devday.ocpcloud.com/hack.php Here is the list of emails sent: From Address: firstname.lastname@example.orgTo Address: email@example.comSubject: New registrationWhen: 2017-08-22 From Address: firstname.lastname@example.orgTo Address: email@example.comSubject: New registrationWhen: 2017-08-22 End of the list
Issue: Ideally UI should not reach a database behind a microservice, but instead should access it via the APIs exposed by the microservice. It is a well-known architecture practice that if you are running an application in a multi-tiered architecture, your UI layer should not be talking to th database directly but go via APIs exposed by business logic. In fact, the database is generally firewalled, and only ports from the services that are supposed to call the database are allowed. If you are using a flat network such as
ovs-subnet, these fine-grained network controls between services are not possible.
This post explains what Network Policy Objects are and how they can be used to control network traffic in theory: https://blog.openshift.com/whats-new-in-openshift-3-5-network-policy-tech-preview/.
Step 6: Network Policy Objects to the rescue
Let us now add Network Policy controls to our microservices so that the above kind of hack is not allowed. We’ll add policies that allow specific traffic to your services.
Let’s protect the MySQLdb behind the Email Service so that it is only accessible from the Email Service. Create a file with the following content:
$ cat allow-3306.yaml kind: NetworkPolicy apiVersion: extensions/v1beta1 metadata: name: allow-3306 spec: podSelector: matchLabels: app: mysql ingress: - from: - podSelector: matchLabels: app: emailsvc ports: - protocol: TCP port: 3306
Here we are allowing traffic coming from pods that have labels matching
app: emailsvc to reach the MySQL pod that has label
app:mysql. Look at the labels assigned to pods and services to understand how this works.
$ oc get svc -n msinfra --show-labels NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE LABELS emailsvc 172.30.42.184 8080/TCP 1h app=emailsvc,microservice=emailsvc mysql 172.30.43.224 3306/TCP 1h app=mysql,microservice=emailsvc $ oc get pods -n msinfra --show-labels | grep Running emailsvc-1-grb00 1/1 Running 0 1h app=emailsvc,deployment=emailsvc-1,deploymentconfig=emailsvc,microservice=emailsvc mysql-2-ffj05 1/1 Running 0 1h app=mysql,deployment=mysql-2,deploymentconfig=mysql,microservice=emailsvc
Now create this policy in the
msinfra project as follows:
$ oc create -f allow-3306.yaml -n msinfra networkpolicy "allow-3306" created
As soon as the policy is added, if you test the hack again, it should fail with
504 Gateway Time-out as shown below:
$ curl http://userreg-msclient.apps.devday.ocpcloud.com/hack.php
504 Gateway Time-out
The server didn’t respond in time.
Step 7: Fully protect your microservices
Note You can clone this Git repository to download all these policy files: https://github.com/VeerMuchandi/usingnetworkpolicyobjects
Let’s add more policies now.
Completely deny all traffic to
$ cat default-deny.yaml kind: NetworkPolicy apiVersion: extensions/v1beta1 metadata: name: default-deny spec: podSelector:
Create this policy:
$ oc create -f default-deny.yaml -n msclient networkpolicy "default-deny" created
Now if you test your UI, it should say
Application is not available.
You can see that the route (from loadbalancer) is reaching the service at port 8080:
$ oc get route -n msclient --show-labels NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD LABELS userreg userreg-msclient.apps.devday.ocpcloud.com userreg 8080-tcp None app=userreg,microservice=userreg
So let’s allow port 8080 for the the
userreg service to enable communication with UI as below:
$ cat allow-8080-frontend.yaml kind: NetworkPolicy apiVersion: extensions/v1beta1 metadata: name: allow-8080-frontend spec: podSelector: matchLabels: app: userreg ingress: - ports: - protocol: TCP port: 8080 $ oc create -f allow-8080-frontend.yaml -n msclient networkpolicy "allow-8080-frontend" created
Now your front-end URL should work again!
Next, Default Deny all traffic to
$ oc create -f default-deny.yaml -n msservices networkpolicy "default-deny" created
Now allow 8080 to User Registration Backend and Twitter Services:
$ cat allow-8080-userregsvc.yaml kind: NetworkPolicy apiVersion: extensions/v1beta1 metadata: name: allow-8080-userregsvc spec: podSelector: matchLabels: app: userregsvc ingress: - ports: - protocol: TCP port: 8080 $ cat allow-8080-twitter.yaml kind: NetworkPolicy apiVersion: extensions/v1beta1 metadata: name: allow-8080-twitter spec: podSelector: matchLabels: app: twitter-api ingress: - ports: - protocol: TCP port: 8080 $ oc create -f allow-8080-userregsvc.yaml -n msservices networkpolicy "allow-8080-userregsvc" created $ oc create -f allow-8080-twitter.yaml -n msservices networkpolicy "allow-8080-twitter" created
Allow MongoDB to be accessed only by User Registration Service:
$ cat allow-27017.yaml kind: NetworkPolicy apiVersion: extensions/v1beta1 metadata: name: allow-27107 spec: podSelector: matchLabels: app: mongodb ingress: - from: - podSelector: matchLabels: app: userregsvc ports: - protocol: TCP port: 27017 $ oc create -f allow-27017.yaml -n msservices networkpolicy "allow-27107" created
Deny all traffic to
$ oc create -f default-deny.yaml -n msinfra networkpolicy "default-deny" created
We already have only the Email Service accessing the MySQldb. We will now have to allow connection to the Email Service in the
msinfra project from the User Registration Service in a
msservices project. While you can select the destination in this case, you cannot select a specific pod on the source side when the namespaces are different. So we will have to open up the namespace
msservices to be able to reach the Email Service in
First, we will need a label to identify the (traffic) source project
msservices. Your cluster-admin may have to do this.
# oc label namespace msservices project=userregservices namespace "msservices" labeled
Now let’s add the policy that allows services in
msservices project to talk to the Email Service in the
msinfra project. Note the
namespaceSelector here, as we are opening traffic flow from
msservices namespace here.
$ cat allow-8080-emailsvc.yml kind: NetworkPolicy apiVersion: extensions/v1beta1 metadata: name: allow-8080-emailsvc spec: podSelector: matchLabels: app: emailsvc ingress: - from: - namespaceSelector: matchLabels: project: userregservices ports: - protocol: TCP port: 8080 $ oc create -f allow-8080-emailsvc.yml -n msinfra networkpolicy "allow-8080-emailsvc" created
Now we are all set. We have policies in place that prevent services crossing boundaries. Test the application to make sure the calls between microservices are all going through.
You should now understand:
- Why Network Policy Objects are important.
How to create policies that deny all the traffic.
How to open up traffic to specific destinations based on labels.
How to open up traffic that crosses across projects.
Dan Winship helped me at every stage of this testing Network Policy Objects and got me up to speed on using this feature. Many thanks to Dan.