19 August 2011

What value does Gartner’s Magic Quadrant bring to a WCMS selection?

Gartner’s Magic Quadrant is frequently used as a reference point when evaluating WCMS platforms. Although the report may provide a strategic overview of a particular slice of the WCMS market, it does little to shed light on the real issues that underpin a WCMS selection.

What are we assessing here?

Gartner’s Magoc Quadrant process involves rating each platform on the basis of “completeness of vision” and “ability to execute”. The language that Gartner uses to categorise platforms – e.g. “visionaries” and “leaders” – seems designed to strike a chord with corporate decision makers. It provides a straightforward comparison that may be easy to digest at a high level, but does it actually mean anything to people who are having to make procurement decisions?

Having advised a number of clients on WCMS platform selection, I regard the distinctions offered by Gartner as a little simplistic. I am also suspicious of using a simple linear scale to compare platforms, particularly when Gartner do not publish the scoring decisions that underpin it.

Any procurement decision should take into account the complex trade-off between different requirements that is unique to each publishing scenario. These trade-offs are notoriously difficult to express through a scoring mechanism and cannot be represented by a pair of scores or points that are plotted on a grid.

Assessing the WCMS market on this scale inevitably runs into problems of not comparing like with like. The selection of platforms included by Gartner represents a fair cross section of the WCMS market and their target markets and use cases do not necessarily overlap.

A pair of scores will not let you draw any useful distinctions between an enterprise platform such as Percussion and a middle-market player such as EPiServer. The fact that one is platform is labelled a “visionary” whilst the other is a “niche player” does not provide any constructive support for decision making.

A skewed view of the market?

There have been accusations that Gartner’s ranking system is influenced less by objective study and more by consultancy fees. This may be a little cynical for me, though there is a clear conflict of interest if you do not openly declare the money that you are receiving from vendors.

Gartner is very open about the criteria it uses when selecting platforms to include, but there are a number of exclusions that give the magic quadrant a skewed view of the WCMS market.

Firstly, Gartner insists that a company must be generating at least $10m from WCMS before it can be included in the report. Not only does rule out many open source platforms but it creates a bias towards larger, commercial software organisations.

This bias has even been the subject of a lawsuit from a software vendor who claimed that Gartner favours vendors with large sales and marketing departments over smaller, more innovative suppliers. Gartner defended this lawsuit on the basis that their reports were based on “pure opinion” – i.e. are completely subjective – which doesn’t help to confirm the credibility of their scoring system.

Big is not necessarily best. You don’t necessarily have to reach for an expensive, proprietary platform to scale across an enterprise. It is surprising how many media organisations are implementing large-scale publishing platforms using a lightweight and inexpensive WCMS such as Drupal or even WordPress. This is a trend that does not fit well with the magic quadrant paradigm.

Gartner also have a geographic bias and it is slower to include platforms that have developed a regional stronghold outside of the USA. For example Ektron is included despite its limited penetration outside of North America while the more European-centric Magnolia does not get a look in. This limits the report’s usefulness outside of America.

Falling behind the curve

Gartner’s reports are often criticised for being reactive and conservative. Rather than shedding light on a constantly evolving market, they review a consistent group of mature corporate vendors whose products are on a similar two-year release cycle to each report.

Yves de Montcheuil – a marketing VP at Open source integration company Talend – outlined this position in a criticism of Gartner’s magic quadrant for data integration:

“More than a state of the market, the Magic Quadrant reflects past adoption of certain technologies by large accounts in the US, who are customers of Gartner. Updated every 18 to 24 months and reflecting the long cycles of traditional vendors, who used to take years before their could achieve a significant position on a market, this Magic Quadrant is no longer compatible with new development and adoption cycles such as the ones we can find in open source and SaaS.”

Gartner’s Andreas Bitterer defended Gartner’s focus on track record and denied that Gartner are “turning a blind eye on new developments”. However, Gartner’s WCMS report does seem to miss large areas of the WCMS market, both in terms of function and geography. They may struggle to remain relevant unless these deficiencies are addressed.

A strategic summary rather than a buyers’ guide

It’s been suggested that Gartner is directed more at potential investors rather than people making procurement decisions. Gartner’s tendency to concentrate on each platform’s marketing and messaging may be relevant to other WCMS vendors but means little to the people making procurement decisions. This lack of detailed assessment means that Gartner’s magic quadrant is less of a tool for WCMS selection and more of a guide for those wanting a strategic overview of a particular US-centric, well-established slice of the WCMS market.

Filed under CMS, Strategy.