27 October 2016

The problem with using maturity models to describe technology solutions

Maturity models were originally developed as organisational assessment tools. They describe an evolution through a set of stages in order help objectively assess current state and provide clear solutions for improvement.

One of the better known and earliest examples is the Capability Maturity Model (CMM) developed by the US Department of Defense in the early 1990s. It was used in assessing the capacity of software contractors to deliver on projects. Since then there has been a proliferation of maturity models each describing different aspects of organisational capability, from project management to e-learning.

These models normally follow a consistent pattern that describes an anticipated evolutionary path. The first stage has little or no capabilities while the highest stage represents a complete state of maturity. The levels between describe a continuous progression where each subsequent level can be identified by specific criteria and characteristics.

Maturity models as a false continuum

The maturity model approach is often extended to defining software architecture or describing technology solutions. The problem is that technology does not always fit the consistent linear progression implied in a maturity model. Real world solutions tend to be subject to obstacles, tangents and detours rather than following an ordered transition.

The Richardson Maturity Model is an example of this kind of false continuum. The model attempts to analyse REST APIs in terms of increasing levels of maturity, ranging from lowly HTTP based APIs to the full “glory” of HATEOAS-enabled REST. The intention is to help promote understanding by implying explaining REST in stages, but the progression it implies may not always be valid.

Is the fully mature, HATEOAS-enabled REST described in the final stage of the Richardson model always an appropriate solution? A more optimal solution can involve a more pragmatic approach based on resources that do not go the whole hog on hypermedia. On the other hand, strict readers of the original thesis by Roy Fielding would argue that REST is an “all-or-nothing” proposition with no progression involved at all.

This example reveals two weaknesses of the maturity model approach when allied to software architecture.

Firstly, there are dangers in making too many assumptions of what constitutes the “final state” of maturity. This may vary depending on circumstances so the progression described in a model will not always be valid. Even critics of the CMM pointed out that process maturity is not necessarily mandatory for successful software development.

Secondly, problems cannot always be defined in terms of a linear progression. A maturity model should be able to demonstrate meaningful evolution by describing the value that is added with each stage. This can become quite contrived when describing a software architecture where there isn’t always a meaningful progression, only a binary flip between “delivered” and “in progress”.

If you want to describe an evolving technology solution, you are probably better off calling it a “roadmap”.

Maturity models can cloud debate and obscure understanding of technical nuance. They frame debate in terms of the level to aim for, where the most appropriate solution often involves a blend of one or more of levels. It can also be more appropriate to tackle the levels in a different order to the ones proscribed, or exclude some entirely.

Creating arbitrary models

Jörg Becker et. al. pointed in out that the ongoing proliferation of maturity models indicates a certain arbitrariness in the way they are developed. The authors of maturity models rarely reveal their motives for creating them or explain what process they use. In the absence of any credible, published design process a maturity model risks being relegated to the status of “a mere marketing tool for business consultants”.

Most models have very little empirical support in terms of real-world observation or comparison. There is often very little evidence to clearly demonstrate the relevance of the problem or define the target audience. This lack of rigor makes it difficult to tell whether a model can reliably help to deliver any improved solutions.

Maturity models inevitably become obsolete over time due to changing conditions and technology. If a model is not backed by a coherent definition process then it will struggle to adapt to changing circumstances. The arbitrary nature of its creation will only undermine its continued relevance.

A weak analysis?

Describing something as a “model” implies a degree of rigor or scientific method. Most maturity models don’t have any formal theoretical basis and are built on arbitrary decision-making and untested assumptions. They may be based on the judgement of “very knowledgeable people” but they can provide a weak analysis, particularly when applied to technology solutions.

Filed under Architecture, Design patterns, Rants.