Building Microservices with WildFly Swarm and Netflix OSS on OpenShift

There are lots of discussions about microservices these days and the technology stacks that are suitable for this architecture style. In this post we explore how WildFly Swarm can be used for building microservices and how to integrate Netflix Hystrix and Turbine in order to build resilience into WildFly Swarm microservices.

What is WildFly Swarm?

WildFly is a lightweight, flexible, feature rich, Java EE 7 compatible application server. WildFly has even introduced a lean 28MB Servlet-only distribution. The most recent version of WildFly is 10.0.0 which will be used as the foundation for JBoss EAP 7.0, Red Hat’s commercial Java EE server offering.

WildFly Swarm departs from a monolithic approach to application servers and offers a way to package and run Java EE applications by packaging them with just enough of the Java EE platform to "java -jar" your application. It allows you to build your own, custom feature Java EE runtime as a single executable jar (fat-jar) including all the dependencies of your application and run it directly on the JVM. In addition, it offers support for microservice patterns with Java EE by integrating with the Netflix OSS stack, metrics and health endpoints, LogStash,  Swagger, single sign-on, and more.

What are Turbine and Hystrix?

Netflix Hystrix is a latency and fault tolerance library which implements the Circuit Breaker design pattern. It isolates points of access to remote services and prevents cascading failures in order to provide resiliency in distributed system when these services fail. It also generates a series of operational metrics regarding the number of current successful and failed calls to a particular service.

Netflix Turbine is a data aggregator engine that aggregates the data generated by Hystrix and other Netflix components into a single stream of data for visualization.

A Microservices Application

We have detailed microservices and how to build microservices with WildFly Swarm in previous posts. In this example, we focus on how to integrate Hystrix and Turbine in a WildFly Swarm microservices application running on OpenShift.

This example is composed of four components:

• Employee Service: A simple JAX-RS service using WildFly Swarm which returns the list of employees. This service randomly generates error and timeout responses.

• Payroll Service: A simple JAX-RS service using WildFly that invokes Employee service using Hystrix and returns the payroll data of employees.

• Turbine: A Netflix OSS component for aggregating streams of json data

• Hystrix Dashboard: A Netflix OSS component for visualizing aggregated data streams

Hystrix in a WildFly Swarm Service

In order to use Hystrix in a WildFly Swarm service, you need to add Hystrix as a dependency in pom.xml:

<dependency>

    <groupId>com.netflix.hystrix</groupId>

    <artifactId>hystrix-core</artifactId>

    <version>1.5.1</version>

</dependency>
<dependency>
 <groupId>com.netflix.hystrix</groupId>
 <artifactId>hystrix-metrics-event-stream</artifactId>
 <version>1.5.1</version>
 </dependency>
 

In order to take advantage of Hystrix, each outbound call to a service needs to be wrapped in a Hystrix command. The following snippet is from the PayrollController in the Payroll service:

@GET

@Produces(MediaType.APPLICATION_JSON)

public List findAll() {

List employees = new FindEmployeesCommand().execute();

    ...

}
class FindEmployeesCommand extends HystrixCommand<List> {
 public FindEmployeesCommand() {
 super(HystrixCommandGroupKey.Factory.asKey("EmployeesGroup"));
 }
 @Override
 protected List run() {
 String url = Utils.getEmployeeEndpoint("/employees");
 Builder request = ClientBuilder.newClient().target(url).request();
 try {
 return request.get(new GenericType<List>(){});
 } catch (Exception e) {
 ...
 }
 }
 }
 

The command FindEmployeesCommand makes a REST call to the Employee service and returns the response. Hystrix executes the command and returns the response. If the Employee services fails or times out for a number of times within a time window, Hystrix opens the circuit and returns calls immediately for a period of time. All these settings are configuration through HystrixCommandProperties.Setter().

Hystrix also exposes the metrics collected from the services calls. In order to make the metrics available to Turbine, HystrixMetricsStreamServlet should be configured in web.xml:

<servlet>

    <display-name>HystrixMetricsStreamServlet</display-name>

    <servlet-name>HystrixMetricsStreamServlet</servlet-name>

    <servlet-class>com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet</servlet-class>

</servlet>
<servlet-mapping>
 <servlet-name>HystrixMetricsStreamServlet</servlet-name>
 <url-pattern>/hystrix.stream</url-pattern>
 </servlet-mapping>
 

That’s it! Payroll service is enabled to use Hystrix for out-band calls to Employee service and it exposes the metrics to be able to monitor the state of the circuit.

It’s worth mentioning that WildFly Swarm is currently enhancing its Hystrix support by including the hystrix-metrics-event-stream and automatically setting up the above servlet definition in web.xml.

Deploying on OpenShift

All source codes for the above example are available on this GitHub repository:

https://github.com/siamaksade/wildfly-swarm-hystrix-example

To run this example on OpenShift, first you should build the Docker images for all components and have them available locally on OpenShift master. You can use Red Hat Container Development Kit or the OpenShift Origin all-in-one VM for a local OpenShift environment. Run the following on OpenShift master, CDK or the all-in-one VM to build all components and package them as Docker images:

$ git clone https://github.com/siamaksade/wildfly-swarm-hystrix-example.git

$ cd wildfly-swarm-hystrix-example

$ mvn install docker:build

The wildfly-swarm-plugin creates a fat-jar for the WildFly-Swarm service including all the dependencies required. Afterwards, the docker-maven-plugin builds a Docker image based on the fat-jar and starts it up directly on JDK using java -jar:

<entryPoint>

    <exec>

        <arg>java</arg>

        <arg>-jar</arg>

        <arg>/opt/${project.artifactId}-swarm.jar</arg>

    </exec>

</entryPoint>

Now that the images are ready, create a new project in OpenShift, import the template wildfly-swarm-microservices-example.json and create an application based on the imported template:

$ oc new-project wildfly-swarm --display-name="Wildfly Swarm Microservices"

$ oc create -f wildfly-swarm-microservices-example.json

$ oc new-app wildfly-swarm-microservices-example

A number of objects for the services and components are created within the project:

--> Deploying template wildfly-swarm-microservices-example for "wildfly-swarm-microservices-example"

--> Creating resources ...

Service "employee-app" created

Service "payroll-app" created

Service "turbine" created

Service "hystrix-dashboard" created

Route "employee-app" created

Route "payroll-app" created

Route "turbine" created

Route "hystrix-dashboard" created

DeploymentConfig "employee-app" created

DeploymentConfig "payroll-app" created

DeploymentConfig "turbine" created

DeploymentConfig "hystrix-dashboard" created

--> Success

Run 'oc status' to view your app.

Turbine uses a custom discovery mechanism in this project which makes API calls to OpenShift in order to discover Hystrix-enabled endpoints for aggregation. This is done through querying OpenShift for all services that are labeled with "hystrix.enabled": "true". You can also customize the Turbine cluster name for your service through the “hystrix.cluster” label.

Run the following commands as root on the master to give cluster privileges to the service account in your project:

$ oc login -u system:admin

$ oadm policy add-cluster-role-to-user cluster-reader system:serviceaccount:wildfly-swarm:default

You can verify that the deployed Wildfly Services working using the curl command:

$ curl http://employee-app-wildfly-swarm.10.1.2.2.xip.io/employees

[{"id":1,"name":"John"},{"id":2,"name":"Sarah"},{"id":3,"name":"Matt"},{"id":4,"name":"Linda"}]
$ curl http://payroll-app-wildfly-swarm.10.1.2.2.xip.io/payroll
 {"employee":{"id":1,"name":"John"},"salary":1500,"employeeName":"John"},{"employee":{"id":2,"name":"Sarah"},"salary":3000,"employeeName":"Sarah"},{"employee":{"id":3,"name":"Matt"},"salary":4500,"employeeName":"Matt"},{"employee":{"id":4,"name":"Linda"},"salary":6000,"employeeName":"Linda"}]
 

If you get an "Internal Server Error" for some of the Employee requests, it's by design! The service simulates a certain ration of errors and time-outs.

Monitor Services with Hystrix Dashboard

Hystrix Dashboard is a web application for monitoring the Hystrix commands being executed through the data collected at each endpoint. It shows the number of successful and failed requests among other things and if the state of the circuit (open or close).

Enter Turbine's stream url in Hystrix Dashboard and then click Add Stream and Monitor Streams:

Note that currently the FindEmployeesCommand circuit is open in the Payroll service and there are no commands being executed.

Generate some load on the Payroll service and monitor the result on the dashboard:

ab -n 100 http://payroll-app-wildfly-swarm.10.1.2.2.xip.io/payroll

Since the Employee service generates random errors, the circuit closes after a number of failed and timed out requests:

Conclusion

WildFly is a lightweight, flexible, feature rich, Java EE 7 compatible application server. WildFly Swarm is the marriage of WildFly with microservices approach of building services in order to create single self-sufficient executable jars (fat-jar) including all the dependencies of your service and run it directly on the JVM. WildFly Swarm combined with Netflix OSS on OpenShift allows building feature-rich, resilient and flexible Java EE 7 microservices which are packaged and orchestrated as containers through OpenShift on the infrastructure of your choice.

Red Hat plans to productize and provide support for WildFly Swarm in the future. This in essence allows you to use WildFly Swarm to create lean self-contained microservices and at the same time stay on a fully supported platform with enterprise-grade quality.