Removing ANSI Color Characters in Kibana

Visual cues are part of everyday life, from arrows indicating the rise and fall of the stock market, to the colors on traffic signals ensuring the smooth flow of goods, services, and individuals. When used with technology, colors help provide an indication of the ongoing health of systems. Green typically indicates normal operation, while red indicates an erroneous state. For developers and system maintainers, application and server logs are absolutely essential to determining the overall state of a system. Many of these same tools also add coloring queues to help express the fields being emitted. In order to produce colorized output, these systems add in a series of ANSI escape sequences as part of the data stream, which are not rendered as part of the normal output. First introduced in the 1970s, ANSI characters proliferated through the mainframe era and are found in many popular technologies today, such as build tools like Apache Maven, and application servers including JBoss Enterprise Application Platform (EAP) as shown below:

ANSI codes are great when viewing the realtime console output. However they do cause extraneous entries to show when the console output emitted from an application is captured in a file:

[0m[0m00:22:31,942 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: JBoss EAP 7.0.6.GA (WildFly Core 2.1.15.Final-redhat-1) started in 18243ms - Started 518 of 848 services (505 services are lazy, passive or on-demand)

The [0m[0m at the beginning of the log entry is the ANSI escape character and indicates that any existing colors that may be present should be removed.

Capturing console output that has been written to a file is a common pattern in the container ecosystem. OpenShift includes an aggregated logging framework built on the Elasticsearch, Fluentd, and Kibana stack that makes searching container logs running on disparate hosts a breeze. Fluentd captures the output emitted by running containers on each host and stores the resulting output in Elasticsearch. Kibana provides the user interface for searching for entries stored in Elasticsearch.

When analyzing the results in Kibana, ANSI codes can pose a challenge for determining the actual content as the embedded codes could make the original entry unclear (such as the server timestamp in the example above). Fortunately, Kibana contains a feature called Field Formatters that allows data to be displayed based on user preference regardless of how it is stored in Elasticsearch. Introduced in Kibana 4.1, the out of the box set of features includes the ability to format string, dates, and numbers. Also available is the functionality for end users to define their own set of custom field formatters. The use of custom field formatters is ideal for removing extraneous ANSI codes that are found within entries and Kibana makes it easy to create and apply them.

Let’s walk through how a custom formatter for removing ANSI codes can be implemented within the aggregated logging framework in the OpenShift Container Platform.

The assets are found within the following Git repository: https://github.com/sabre1041/kibana-ansi

Kibana uses a plugin model to support custom enhancements. This Git repository is organized in a way that the field formatter can be deployed to either a standalone Kibana instance or OpenShift. The instructions within the README file detail the steps for either method. For the purpose of this discussion, the OpenShift method will be described in the subsequent sections.

The first step is to clone the repository to your local machine and change into the repository directory:

$ git clone https://github.com/sabre1041/kibana-ansi

$ cd kibana-ansi

To avoid modifications to the Kibana docker image, the assets needed by the plugin can be stored within the OpenShift platform as ConfigMaps and injected into the image at runtime as a volume mount.

Login to the target OpenShift environment as a user with access to the project containing the EFK stack (logging by default).

$ oc login

$ oc project logging

A ConfigMap called kibana-ansi will be used to contain the required files to support Kibana’s plugin architecture and the necessary custom field formatter files from the repository:

• index.js – Describes the plugin content
• package.json – NPM package dependency descriptor file
• public/ansi.js – The field formatter

Create the ConfigMap:

$ oc create configmap kibana-ansi --from-file=index.js --from-file=package.json --from-file=public/ansi.js

User defined plugins can be installed either through the use of a CLI tool or by placing files in the $KIBANA_HOME/installedPlugins folder. The latter of these options will be used within this use case and the contents of the ConfigMap will be mounted into this location through a series of volumes. Typically, volumes overwrite any existing content in the directories that they are mounted within and the use of overlapping mount points as would be needed in this use case would cause additional conflicts. However, a fairly unknown option of volumes known as a subPath provides that flexibility to mount individual files within the existing image and provide the necessary plugin file structure supported by Kibana.

Execute the following command to modify the Kibana DeploymentConfig to mount the files contained within the ConfigMap:

oc patch dc/logging-kibana -p '{"spec":{"template":{"spec":{"volumes":[{"name":"kibana-ansi-type","configMap":{"name":"kibana-ansi","items":[{"key":"ansi.js","path":"ansi.js"}],"defaultMode":420}},{"name":"kibana-ansi-index","configMap":{"name":"kibana-ansi","items":[{"key":"index.js","path":"index.js"}],"defaultMode":420}},{"name":"kibana-ansi-package","configMap":{"name":"kibana-ansi","items":[{"key":"package.json","path":"package.json"}],"defaultMode":420}}],"containers":[{"name":"kibana","volumeMounts":[{"name":"kibana-ansi-index","mountPath":"/usr/share/kibana/installedPlugins/kibana-ansi/index.js","subPath":"index.js"},{"name":"kibana-ansi-type","mountPath":"/usr/share/kibana/installedPlugins/kibana-ansi/public/ansi.js","subPath":"ansi.js"},{"name":"kibana-ansi-package","mountPath":"/usr/share/kibana/installedPlugins/kibana-ansi/package.json","subPath":"package.json"}]}]}}}}'

Because the configuration of the DeploymentConfig was modified, a new deployment of Kibana will occur. Once the deployment completes, navigate and login to Kibana.

To demonstrate an application that emits ANSI codes as part of its normal operation as depicted earlier, deploy a JBoss EAP 7 image to OpenShift in a new project called kibana-ansi:

$ oc new-project kibana-ansi

$ oc new-app –image-stream=jboss-eap70-openshift

A new deployment of JBoss EAP 7 will occur and the resulting application logs can be found in Kibana under the kibana-ansi index. The contents of the container logs themselves are found in the message field of the index. If the field is not currently present in displayed columns, locate message under the Available Fields section on the left-hand side and click Add.

With the column present, notice how each line contains an ANSI character.

This is what we are hoping to remove though the use of the custom formatter.

Custom formatters can be applied within the Indices configuration tab of the Settings section. Click on Settings and then Indices. The list of index patterns is presented on the left-hand side of the page and uses the pattern project.<project_name>.<project_uid>. Select the index beginning with project.kibana-ansi and the page will update with the available fields that have been indexed thus far. Locate the row containing the message field. Notice how Kibana has indicated it as a string type, but also notice that there is no indicator under the format column.

Click on the pencil icon under the control column to modify the configuration of the message field.

The Type dropdown is locked as string, but selecting the Format dropdown reveals the available formatting options. If ANSI is present as an available option, then the plugin installation and configuration was successful.

Select ANSI from the format dropdown and then click the Update Field button to apply the changes.

Click on the Discover tab to once again reveal the log entries for the application. Because the custom field formatter was applied for the message field, the ANSI escape codes should no longer be present, thus fulfilling the intended goal.

Nice clean log entries!

Repeat this process of applying a formatter for each project where you need to remove ANSI codes from fields.

The ability to remove the ANSI escape characters from fields in the Kibana console through the use of custom field formatters gives both platform maintainers and application users the ability to fine tune how data is displayed and ultimately interpreted for analyzing the current state of applications.