Imagine a bacon-wrapped Ferrari. Still not better than our free technical reports.

Conquering job hell and multiple app branches using Jenkins & Mercurial


In the past, I’ve written a lot about Jenkins, Mercurial (see all the cool links below) and other tools that we use at ZeroTurnaround. Today, I thought I’ll share with you how we manage multiple branches in Mercurial while still enabling a Continuous Integration experience for our development teams. I’ll concentrate on one of our products, LiveRebel (which, incidentally, we actually use to release over 30 applications of our own, twice a week, all without impacting our users. Dogfooding it baby!)

Our approach uses tools freely available on the market and some custom scripts. If you have multiple branches and struggling with CI, then this article might give you some ideas.

How we use the Branching Model


To set the stage, I really do believe that true Continuous Integration is with a single mainline that developers commit their code to. In reality, I doubt if there are many teams out there with the luxury of having a single mainline that everything can be merged to. At one point in any long-term project, there will be multiple version-based mainlines in existence, and branching is really the way to deal with this.

Let’s use the ZeroTurnaround product LiveRebel as an example, for which we have versions 2.6, 2.7, 3.0 and 3.1. Some of them are in active development. Others are not. But all of these branches get at least some development and releases, but at the same time only share roughly 70% of the functionality.

To make matters a bit more difficult, we also have a separate staging branch for each version branch. So, LiveRebel 3.0 also happens to have a 3.0 staging branch, and it’s this staging branch that we do a release off of. These staging branches came to be a couple of years ago when we were feeling that whenever we started a release we couldn’t commit anything to that branch.

Releases take time. Especially if your project is multi-module Maven project with huge artefacts that bring release time itself to about an hour. Besides the release time, we also don’t want to see any commits to the source tree during that time because they might blow something up and delay the release even more. So, to avoid a code freeze we introduced the staging branch, which is what we release from while we continue to code away on the version’s mainline. Simple, and works really well for us.

Playing nice with Jenkins and a gazillion jobs

Of course, when you have many branches in Mercurial (or Git, SVN, etc.), you will need a lot of jobs in Jenkins, naturally gaining the benefits of CI for all of them. Managing all this is another question, however, and I’ve discussed this at some length in my StackOverflow question about handling multiple branches in CI.

The first thing was to install the Jenkins Nested View Plugin. This plugin gives you multiple levels of views, i.e. the first level is the project/product name, the second level is mainly branch names and only then you see the job listing.

We opted to custom scripts to manage the inevitable job hell and we prepared scripts that copy the jobs from view to another. So, for example, when we kick off a new branch we also create a new set of jobs.


We also have scripts that “sync” these views so that when you change some jobs in a view you can get those changes propogated to another view.

Of course, this all seems a lot of work for such a typical problem. Jenkins & Git (compared to Mercurial) can manage multiple branches within a single job, but this might not be a fit if your project has evolved and you need to build some branches differently. In any case, here are a couple of similar approaches:

  • Job DSL Plugin – manage job definitions and creation with a DSL and templates
  • Jenkins Auto Jobs – automatically create jobs from templates and branches in SCM.


Multiple branches and CI is not a solved problem. For the purists, it is easy; just use a single mainline or special integration branch. We’ve tried and failed at that and started using multiple branches and multiple sets of CI jobs for the branches.

The solution with Jenkins and these scripts is a bit nasty, and as the guys build LiveRebel we certainly don’t suggest adding more manual scripting to any process, but this is our own mess to deal with. Now that there are plugins available to manage your jobs it makes even more sense to try to scale Jenkins jobs to your branching habits.

Of course, I do hope there will be a cleaner solution to this problem in the following years!

Related Posts

Here are a bunch of links for great content that we’ve written in the past on Jenkins plugins, Mercurial extensions, job chaining and more. Check them out!