Asynchronous Bind with the Automation Broker

Introduction

In June 2017, Todd Sanders & John Matthews introduced readers to the Ansible Playbook Bundles (APB) and the Automation Broker. Back then, the broker was named Ansible Service Broker but it serves the same function today, in conjunction with APBs, deliver simple to complex multi-container services to OpenShift or Kubernetes.

The Ansible Playbook Bundles are short-lived, lightweight container images consisting of a simple directory structure with named “action” playbooks, an Ansible runtime, and some metadata.

In the metadata, you can specify the parameters required to be specified to run your service, as well as any optional parameters you want to allow to be overridden. It also specifies the APB specification version, used by the Automation Broker for parsing APBs of different generations. The metadata also contains any supported service plans, i.e. a developer plan with limited resources, or a production plan with high availability services backed by storage.

The APBs are designed to orchestrate pre-existing containerized application images. For example, you could create a hosted blog service by pulling in WordPress and PostgreSQL images into a bundle to be deployed as a hosted WordPress service for your users.

As part of the Ansible Playbook Bundle development, we realized that we could extend the contract between the Automation Broker and the APB into a more generic Service Bundle contract. The Service Bundle contract is a lightweight, executable image. The image expects 3 parameters: the action, i.e. provision, bind; --extra-vars, an Ansible specific value, and a JSON document of information that is useful to the bundle.

Currently, the JSON is useful for populating variables in Ansible that can be used by roles. But Service Bundles are not limited to Ansible alone, you can use whatever you want inside as long as you abide by the 3 parameters. Want to use a simple shell script? a Helm chart? Just about anything can be used, it depends on your needs.

While you can use anything you want, we have developed tooling to help make Ansible Playbook Bundle development easier. Steps in creating an APB using the apb tool looks akin to:

1. Run apb init in your project directory
2. Edit the resulting Ansible playbooks/roles
3. Edit the apb.yml to expose any parameters and plans
4. Run apb prepare to generate the metadata
5. Build the container with apb build (or docker build)
6. Push to a registry so the image can be used by the Automation Broker

Because of the simplicity of the Ansible Playbook Bundle concept, combined with the Service Bundle contract, it allows images to be available in OpenShift console for your users. Our user base has grown as well. We have several community created example APBs available now such as Keycloak, Elasticsearch, Kibana, Prometheus, RocketChat and many others.

Actions

The Automation Broker implements the Open Service Broker API, which defines a set of actions used to provision, deprovision, update, bind to and unbind from services in cloud native environments. In the case of the Automation Broker, we’ve targeted OpenShift and Kubernetes.

Bind Action

Most authors will likely use provision and deprovision to deploy their services to the cluster. But there is also a class of applications that have the need for a binding. What a binding represents may vary by service. In general creation of a binding generates credentials necessary for accessing the resource.

Problems with synchronous bind

In OpenShift 3.7, the Automation Broker supported the 2.13 version of the Open Service Broker API which only defined a synchronous bind. Before we created the Automation Broker, many brokers were tied to a single resource. For example, there would be a PostgreSQL broker handling the resources for a PostgreSQL database. In these simple brokers having a synchronous bind was not a huge problem since they would more than likely return a simple set of credentials that were pre-determined. Not a lot of work was being put into the bind methods.

But with a more powerful and general purpose broker, like the Automation Broker, the managing of multiple and differing services became possible. With these differing services, came diverse use cases. Let’s use PostgreSQL as an example. Let’s say you want to create a hosted PostgreSQL service where each binding creates a new set of user credentials and their own database. Because the Open Service Broker API spec has a performance expectation, this limits what you can do in a synchronous call. Launching a meta-container to perform actions or attempting to run any other long-running action is not feasible in such a time.

Because of this limitation, the Automation Broker was only able to allow bind creation tasks in the provision action. Our synchronous bind would simply return the credentials created by the previous provision call. It limited what one could accomplish, making the hosted scenario described above almost impossible. What would be great is if we had an asynchronous version of bind and unbind to allow for more flexibility in what broker authors could do.

Introducing Asynchronous bind

The other actions defined in the Open Service Broker API spec have asynchronous variants: provision, deprovision, and update, can all be run asynchronously. These actions can perform complex tasks like launching several pods, creating storage, etc. The tasks can be monitored using the last_operation API.

Adding asynchronous bind to the Automation Broker was an easy task from an engineering perspective. The Automation Broker already had the existing synchronous bind action, and the infrastructure to perform asynchronous tasks. The biggest hurdle is getting these changes into the upstream Open Service Broker API specification because writing a specification is not trivial. Working with other community members from Google, IBM, Pivotal and Red Hat we were able to get the asynchronous bind spec to a place where we could start to implement it.

As of this writing, the asynchronous bind feature is in what we call the “validation through implementation” phase. This phase allows broker authors to implement the proposed feature to work out any possible kinks in the specification. As of this writing, asynchronous bind is planned to be released in the next version of the Open Service Broker API spec.

Having an asynchronous bind method allows a broker, especially the Automation Broker, perform some complex tasks that were previously only available during the other phases. With this new flexibility comes new opportunities from the trivial case of returning a pre-populated set of credentials or retrieving credentials from a deployed vault to generating certificates signed by a self-signed CA certificate created in the provision action. Maybe we want to support two-factor authentication with our services, where the bind operation waits for some external action, like a query on your phone, to confirm that the bind should happen.

None of the above would be possible with a synchronous bind, at least not without possibly timing out the call.

Enabling Asynchronous Bind in the Broker

In OpenShift 3.9, the asynchronous bind feature, while unsupported because it is still in the early access phase, can be enabled in the Service Catalog and the Automation Broker for testing. First, the AsyncBindingOperations feature gate needs to be enabled in the Service Catalog:

# oc edit daemonset controller-manager -n kube-service-catalog

containers:

- args:

- controller-manager

- -v

- "5"

- --leader-election-namespace

- kube-service-catalog

- --broker-relist-interval

- 5m

- --feature-gates

- OriginatingIdentity=true

- --feature-gates

- AsyncBindingOperations=true

Then the launch_apb_on_bind feature needs to be enabled on the Automation Broker. This setting tells the broker that it can launch an Ansible Playbook Bundle container during an asynchronous bind operation. To enable the feature edit the broker’s configmap:

# oc get configmap broker-config -o yaml

apiVersion: v1

data:

broker-config: |

...

broker:

dev_broker: True

bootstrap_on_startup: true

refresh_interval: "600s"

launch_apb_on_bind: True

...

After updating the configmap, rollout a new broker:

# oc rollout latest dc/asb

deploymentconfig “asb” rolled out

Once the features are enabled on both the Service Catalog and the Automation Broker, all binds will now be performed asynchronously.

Things to come

Be on the lookout for some more powerful APB examples in the coming months leveraging this new asynchronous bind feature, for example:

• retrieve credentials from a deployed Vault
• retrieve a previously generated credential from a Clevis/Tang deployment
• return credentials encrypted with a GPG public key that was passed in as a parameter
• create a self-signed CA cert using the “provision” action, then use the above “bind” operation to generate certificates signed by the provisioned CA
• two-factor authentication, where the bind operation waits for some external action, like a query on your phone, to confirm that the bind operation should continue

We already have a demo showing TLS certificates being issued by a deployed Vault service returned as credentials using the asynchronous bind feature. You can watch the demo on the Automation Broker’s YouTube channel: https://www.youtube.com/watch?v=4jUajlg811Q

For those of you interested in packaging your applications/services with Ansible Playbook Bundles, there is an excellent tutorial entitled “Ethereum APB Development and Testing Demonstration” by one of our engineers, Dylan Murray. Also, check out the Getting Started Guide.

To keep up with developments or to learn more:

• Website:
  • http://automationbroker.io/
• Trello:
  • https://trello.com/b/50JhiC5v/ansible-service-broker
• Github:
  • Automation Broker
    • https://github.com/openshift/ansible-service-broker
  • Ansible Playbook Bundle tooling and examples
    • https://github.com/ansibleplaybookbundle
  • Service Bundle library
    • https://github.com/automationbroker/bundle-lib
• IRC:
  • Freenode #asbroker
• Mailing list:
  • https://www.redhat.com/mailman/listinfo/ansible-service-broker
• YouTube:
  • https://www.youtube.com/channel/UC04eOMIMiV06_RSZPb4OOBw