9 September 2019

Why trunk-based development isn’t for everybody

Trunk-based development makes absolute sense to anybody who has lived through the merge hell associated with long-lived branches.

The approach shuns anything that resembles a long-lived code branch. The recent “Accelerate: State of DevOps” report suggests that trunk-based development is characterized by “fewer than three active branches”, with branches having lifetimes of “less than a day” before they are merged to master. In general, this means developers commit straight to the trunk.

This frequent integration means that the trunk is always ready to release. It also encourages a more cohesive and collaborative development team as everybody works on the same branch. The catch is that it’s not that easy to implement. You need a mature development shop, a loosely coupled code base and a high level of test automation to make it stick.

Can your code base support it?

Trunk-based development is more than just a branching strategy. It’s a cluster of techniques that can help to enable more efficient development.

Teams use branching by abstraction to separate work on large-scale features. Abstraction layers isolate work in progress and allow multiple implementations of functionality to exist in the same code base. A system can be released from a trunk safely despite containing multiple incomplete features.

This does depend on having a code base that allows for neat, feature-orientated abstractions. A legacy monolith with poorly defined layers or thousands of stored procedures isn’t always that amenable to new abstraction layers.

There are also dangers in adding abstractions purely for the sake of development workflow. You might be adding structures to the code base that don’t make a lot of sense after a feature has been implemented. This can increase the general level of abstraction in the code base, creating an elaborate mess that is hard to follow.

Given that incomplete or unreleased features are in the main code branch, access to them can be controlled in production with feature flags. These can be used to control the features that are available either at build or run-time, but they need careful management.

You need to ensure that feature flags are named sensibly, are well-documented, never re-used (see Knight Capital), and have a short life span. They can add to the test burden by increasing the number of permutations and execution paths that need to be tested. There is also an operational burden as you need to ensure controls over how they are accessed and monitor any changes.

Although trunk-based development can reduce merging conflicts, it does not necessarily reduce complexity of managing concurrent feature development. In a sense, you’re just redistributing some of this complexity downstream to other processes, such as the code design and application configuration.

Process and confidence

Development techniques are only part of the story. Trunk-based development also needs a mature build infrastructure, streamlined processes and disciplined development teams.

Above all, it requires confidence. Given that commits to the trunk should be coming thick and fast, developers need to be very certain that they will not break the build. Winding back the trunk after a failed commit can be painful, particularly if other developers have pulled the update onto their local copies.

A build and review process should be able to detect problems with commits as they happen. This depends on the speed of your build infrastructure and frequency of commits. If you have a large, monolithic architecture or a complex suite of tests then your build may not be quick enough to meet the demand of rapid-fire commits.

The team also need iron discipline. You need to make sure that feature branches only live for a day or two. Developers should resist the urge to pull updates for feature branches rather than waiting for them to be accepted on the trunk. If you have release branches you should not be committing features to them. You need to stick to your review processes for even as you approach a deadline.

You also need to ensure that releases can be executed quickly without getting bogged down in over-long preparation phases. Many teams prepare for a deployment by cutting a specific release branch. You can cherry pick the occasional commit from the trunk, but this becomes untenable if the release branch lives for more than a few days.

Pre-requisites for trunk-based development

There’s more to adopting trunk-based development than deciding to do away with branching and commit to the trunk. You need to have the right pre-requisites in place to make it work, i.e.

  • A team of experienced and confident developers who trust each-other’s work
  • A loosely-coupled code base that can support feature abstractions
  • A robust solution for feature flags (e.g. Launch Darkly or Flagr)
  • A continuous build and test infrastructure that can keep up with a high rate of commits.
  • Streamlined processes around review and release that don’t hold up the rate at which commits can be ingested.

Filed under Agile, Development process.