27 October 2016

The problem with using maturity models to describe technology solutions

Maturity models were originally developed as organisational assessment tools. They describe an evolution through a set of stages in order help objectively assess current state and provide clear solutions for improvement.

One of the better known and earliest examples is the Capability Maturity Model (CMM) developed by the US Department of Defense in the early 1990s. It was used in assessing the capacity of software contractors to deliver on projects. Since then there has been a proliferation of maturity models each describing different aspects of organisational capability, from project management to e-learning.

These models normally follow a consistent pattern that describes an anticipated evolutionary path. The first stage has little or no capabilities while the highest stage represents a complete state of maturity. The levels between describe a continuous progression where each subsequent level can be identified by specific criteria and characteristics.

Maturity models as a false continuum

The maturity model approach is often extended to defining software architecture or describing technology solutions. The problem is that technology does not always fit the consistent linear progression implied in a maturity model. Real world solutions tend to be subject to obstacles, tangents and detours rather than following an ordered transition.

The Richardson Maturity Model is an example of this kind of false continuum. The model attempts to analyse REST APIs in terms of increasing levels of maturity, ranging from lowly HTTP based APIs to the full “glory” of HATEOAS-enabled REST. The intention is to help promote understanding by implying explaining REST in stages, but the progression it implies may not always be valid.

Is the fully mature, HATEOAS-enabled REST described in the final stage of the Richardson model always an appropriate solution? A more optimal solution can involve a more pragmatic approach based on resources that do not go the whole hog on hypermedia. On the other hand, strict readers of the original thesis by Roy Fielding would argue that REST is an “all-or-nothing” proposition with no progression involved at all.

This example reveals two weaknesses of the maturity model approach when allied to software architecture.

Firstly, there are dangers in making too many assumptions of what constitutes the “final state” of maturity. This tends to vary between different systems in the real world. Even critics of the CMM pointed out that process maturity is not necessarily mandatory for successful software development.

Secondly, problems cannot always be defined in terms of a linear progression. A maturity model should be able to demonstrate meaningful evolution by describing the value that is added with each stage. This can become quite contrived as systems don’t always evolve along a clear, linear path.

This is particularly the case in an agile context. Systems acquire new capabilities in an order that should respond to evolving circumstances. Maturity models imply a fixed progression that can set the wrong motivation. Teams end up celebrating the achievement of level two rather than focusing on genuine product improvements.

Maturity models can also cloud debate and obscure understanding of technical nuance. They frame development in terms of an arbitrary level to aim for, where the most appropriate solution often involves a more nuanced blend of capabilities.

Creating arbitrary models

Jörg Becker et. al. pointed in out that the ongoing proliferation of maturity models indicates a certain arbitrariness in the way they are developed. The authors of maturity models rarely reveal their motives for creating them or explain what process they use. In the absence of any credible, published design process a maturity model risks being relegated to the status of “a mere marketing tool for business consultants”.

Most models have very little empirical support in terms of real-world observation or comparison. They are rarely backed with evidence that clearly demonstrates the relevance of the problem or defines the target audience. This lack of rigor makes it difficult to tell whether a model can reliably help to deliver any improved solutions.

Maturity models inevitably become obsolete over time due to changing conditions and technology. If a model is not backed by a coherent definition process then it will struggle to adapt to changing circumstances. The arbitrary nature of its creation will only undermine its continued relevance.

A weak analysis?

Describing something as a “model” implies a degree of rigor or scientific method. Most maturity models don’t have any formal theoretical basis and are built on arbitrary decision-making and untested assumptions. They may be based on the judgement of “very knowledgeable people” but they can provide a weak analysis, particularly when applied to technology solutions.

When it comes to software architecture, “maturity” may not be a meaningful or desirable target. After all, you could be described as mature when you are about to die. This is unlikely to be a feature of a system that has reached a perfect state.

Filed under Architecture, Design patterns, Rants.