As an OpenShift cluster grows by helping developers deliver features and applications faster, the efficiency of resource use becomes a central concern. Many OpenShift rollouts start with at least two physical clusters, one for development and one for production. While this has the perceived advantage of simplifying the mental model and some administrative aspects, OpenShift’s automated resilience, enhanced access controls, and advanced Routing features can logically isolate dev and prod environments on a single cluster. The Kubernetes scheduler at OpenShift’s core maximizing the density of applications running on each node. With additional configuration, further density and compute savings are possible through features like Quality of Service QoS.
This post shows how to deploy isolated development and production versions of an application on a single cluster by redirecting requests to the appropriate environment. If engineers are benchmarking and load testing a development application, it should not impact production traffic or production performance. For this reason, we will deploy “sharded” routing in an OpenShift cluster, creating multiple routers for particular purposes.
Cluster and Router Architecture
This is our high level cluster and router architecture:
In this cluster, we do not dedicate nodes to the different production and dev workloads. We just split traffic by routing requests appropriately.
Global Load Balancer: HAProxy
The global load balancer, at the top of the architecture diagram, runs haproxy with the following key lines of configuration:
acl host_router_dev req.ssl_sni -m sub -i apps-dev.example.com
acl host_router_prod req.ssl_sni -m sub -i apps-prod.example.com
use_backend atomic-openshift-router-dev if host_router_dev
use_backend atomic-openshift-router-prod if host_router_prod
(The complete haproxy configuration can be found here.)
OpenShift Routers
As shown in the architecture chart, different subdomains are sent to different OpenShift Routers. We deploy 3 routers: 2 for Production, and 1 for Development. Routers are deployed with the OpenShift command-line API tool, oc. The oc adm router
subcommands are used to manipulate router deployments:
oc adm router router-prod --replicas=2 --force-subdomain='${name}-${namespace}.apps-prod.example.com'
oc adm router router-dev --replicas=1 --force-subdomain='${name}-${namespace}.apps-dev.example.com'
We use --force-subdomain
to force separate subdomains for separate routers.
Next, we will dedicate each router deployment to serve traffic to only a subset of namespaces:
oc set env dc/router-prod NAMESPACE_LABELS="router=prod"
oc set env dc/router-dev NAMESPACE_LABELS="router=dev"
We ensure that our routers are running on their dedicated nodes, by specifying node labels for the router deployments to select:
oc label node infra1.example.com "router=prod"
oc label node infra2.example.com "router=prod"
oc label node infra3.example.com "router=dev"#patch deployments of the router:
oc patch dc router-dev -p "spec:
template:
spec:
nodeSelector:
router: dev"
oc patch dc router-prod -p "spec:
template:
spec:
nodeSelector:
router: prod"
OpenShift Namespace, Project, and Application Labels and Selectors
With router and node labels set, we create OpenShift projects for prod
and dev
, labeling them accordingly.
oc new-project prod
oc label namespace prod router=prod
Now, let's test our configuration with a new OpenShift application in the prod
project we just created and labeled:
# oc new-app cakephp-mysql-example
[...]
# oc get route
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
cakephp-mysql-example cakephp-mysql-example-prod.apps-prod.example.com cakephp-mysql-example web edge None
Notice that the route already uses the apps-prod
subdomain, for production workloads.
Let’s test the same routing pattern for development environments.
# oc new-project dev
# oc label namespace dev router=dev
# oc new-app cakephp-mysql-example
# oc get route
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
cakephp-mysql-example cakephp-mysql-example-dev.apps-dev.example.com cakephp-mysql-example web edge None
Next, we test both URLs as a client, using curl
:
# curl -sSLk -D - https://cakephp-mysql-example-dev.apps-dev.example.com -o /dev/null
HTTP/1.1 200 OK
Date: Wed, 11 Apr 2018 13:27:54 GMT
Server: Apache/2.4.27 (Red Hat) OpenSSL/1.0.1e-fips
Content-Length: 64467
Content-Type: text/html; charset=UTF-8
Set-Cookie: 1b15022d32bdaf178e4bb662559c535f=9b517482994c000cd2b19fe8ca6174e2; path=/; HttpOnly; Secure
Cache-control: private
# curl -sSLk -D - https://cakephp-mysql-example-prod.apps-prod.example.com -o /dev/null
HTTP/1.1 200 OK
Date: Wed, 11 Apr 2018 13:28:46 GMT
Server: Apache/2.4.27 (Red Hat) OpenSSL/1.0.1e-fips
Content-Length: 64484
Content-Type: text/html; charset=UTF-8
Set-Cookie: 2e2307f4645f03dde968155c002d6b44=8316f5abdc2526e8edcb1b110e430325; path=/; HttpOnly; Secure
Cache-control: private
Conclusion
By splitting traffic, we ensure that we can meet our application Service Level Agreements (SLAs) while reducing the number of clusters we have to manage and the resources consumed. By having everything in one cluster, we can save on hardware, people resources, and maintenance costs.
Before you move toward a "one cluster” architecture, ask yourself: Are you and your organization are ready for this?
Acknowledgements
This post is a simplified version of the OpenShift router documentation.
Special thanks to @noeloc.
About the author
More like this
Browse by channel
Automation
The latest on IT automation that spans tech, teams, and environments
Artificial intelligence
Explore the platforms and partners building a faster path for AI
Open hybrid cloud
Explore how we build a more flexible future with hybrid cloud
Security
Explore how we reduce risks across environments and technologies
Edge computing
Updates on the solutions that simplify infrastructure at the edge
Infrastructure
Stay up to date on the world’s leading enterprise Linux platform
Applications
The latest on our solutions to the toughest application challenges
Original shows
Entertaining stories from the makers and leaders in enterprise tech
Products
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Cloud services
- See all products
Tools
- Training and certification
- My account
- Developer resources
- Customer support
- Red Hat value calculator
- Red Hat Ecosystem Catalog
- Find a partner
Try, buy, & sell
Communicate
About Red Hat
We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.
Select a language
Red Hat legal and privacy links
- About Red Hat
- Jobs
- Events
- Locations
- Contact Red Hat
- Red Hat Blog
- Diversity, equity, and inclusion
- Cool Stuff Store
- Red Hat Summit