You may have noticed OpenShift made the top 5 list in pull requests on GitHub last year. I was a bit surprised when I found out, especially considering origin-server is just one of the OpenShift repositories. As I thought about what separates OpenShift from most other projects on GitHub and it all started to make more sense.
As projects grow in number of contributors, it becomes more difficult to maintain day to day stability. Test cases are essential, but getting everyone to run test cases before pushing changes is unreliable. Systems like Travis CI and the GitHub status API help add more reliablity, however they lack the ability to test complex integration scenarios, particularly with multiple repos, and serial gating of merges.
Keeping master stable
So how does the OpenShift development process work? Three repositories make up the core of the OpenShift source: origin-server, origin-dev-tools, and rhc. Our main requirements are that we can serially gate reviewed changes on passing tests and the ability to simultaneously test and merge multiple repos with dependent changes. We do this with a little scripting around GitHub APIs combined with our Jenkins server. The script (test-pull-requests), which is run by our Jenkins server, polls OpenShift GitHub repos for pulls tagged by authorized users with a comment containing
It’s worth noting that almost no one working on OpenShift has direct access to push to OpenShift repos. At most, trusted reviewers have permission to apply a
[merge] do essentially the same thing (run all the gating tests) except that
[test] reports test results back to the pull request while
[merge] merges the pull request if all the tests pass.
test-pull-requests also pulls together multiple git repositories with a simple linking system (comments from trusted users with links to the related pulls). Pull requests from multiple repos are tested and merged together if the tests pass and none of the linked pulls have changed since the tests started.
What about complex tests?
OpenShift is a fairly complex project with several tiers involved. As you can imagine our integration tests need to be fairly elaborate. As a result we limit which tests are included in the gating tests. We tend to focus on fast unit tests and targeted functional and integration tests to get reasonable test coverage without taking a tremendous amount of time. This leaves us with a suite of extended tests we run after builds instead of gating tests. We also have flexibility to run extended tests before merges and so reviewers can choose to add them as prereqs to a
[merge] tag being awarded.
What about really complex tests?
Since OpenShift relies on automated tests to keep a stable code base the test matrix of scenarios and external components makes some manual testing a necessity. We found a fairly simple solution to enable manual testing without pushing changes to master where we simply leave our changes on a development fork(s), build an image with the changes on the fork using our normal build process (just pointing at the forked source rather than master), run tests to make sure the fork is stable, and then register the image for others to test. This ability has been very useful for large, long running, or generally risky changes.
This gives developers the ability to rewrite large sections of code and test it as if it were on master without any of the risk of pushing their code upstream. When the tests pass the developer knows in advance that they will also pass when the code is pushed upstream.
A repeatable packaging process
Packaging for OpenShift is always front and center in the development process. Most developers don’t have to worry much about it but every change made is constantly repackaged for testing. This is accomplished by
[merge] rebuilding every OpenShift maintained rpm from scratch. Doing so adds fairly minimal overhead and guarantees the same thing that worked during development once, will be setup the same way for the next developer, and again when our QE (Quality Engineering) team gets a hold of it. It also guarantees that the same code eventually ends up in production and OpenShift Enterprise.
Continuous build and images
To simplify installation and testing of OpenShift, we maintain a single development and test environment that installs all the OpenShift components into one instance. An image of each instance is created following every merge. These images are used by developers for their development and by QE for testing. Historical images are kept from previous releases making it easily possible to go back in time or test upgrades.
Incremental development with git
We also have a streamlined process for making changes to an existing OpenShift instance we refer to as syncing. The benefit of this process is fast and consistent updates. To accomplish sync, we use git to push a diff of the content from a local development environment to an instance running OpenShift. Using git allows us to easily push a diff making remote development to the cloud from high latency connections possible.
Next on the remote instance we launch a diff process, using Tito. Tito finds any source that has changed since the last sync and build and install the rpms for the affected packages. Note that this process doesn’t stop local development or syncing individual files while providing a simple way make changes across the entire project, or rebase in other people’s changes, and update an instance to a state as if it were installed from scratch.
And hence the pull request total
The combination of OpenShift being a very active project combined with a process that enables a stable master has enabled a lot of contributors to be continuously productive. All day, every day, we create new builds which are guaranteed to function at a high level regardless of how many people are contributing code.
Now it’s your turn. Get involved in OpenShift’s upstream community: