Generative AI seems to be having something of a moment. ChatGPT has really captured the public imagination due to its ability to give detailed responses across a wide range of knowledge domains. The responses are fast, well-structured, and on point.

Scratch beneath the surface and the results can start to feel less impressive. The written outputs are what I would expect from a competent arts graduate who has learnt how to structure a response yet has no experience of the subject. The responses are consistently formulaic in both structure and style. Like all creatively sloppy writers, these models have also been known to fabricate results.

Despite these apparent shortcomings, these large language models might have arrived at a "spinning jenny" moment where they have the potential to transform how we approach creative tasks such as writing reports and cutting code. The problem is that our ability to understand the proper context and potential may be distorted by an unhelpful explosion of hype.

Vendors are falling over themselves to add an "AI" tag to their product names, much in the same way that the "e-" prefix was over-used during the first internet boom. There's a Cambrain explosion of new tools promising to change the way we communicate, collaborate, and create. Commentators who haven't heard of the luddite fallacy are heralding the imminent demise of any vaguely creative role, including software engineering.

In this increasingly hucksterish environment it's easy to be overcome with world-weary cynicism and roll your eyes whenever mention is made of ChatGPT. We've been here before, of course, with emerging technologies from blockchain to the metaverse failing to live up to expectations. The risk is that we may miss emerging opportunities due to the discouraging influence of this excessive hype.

Gartner's Hype Cycle

Gartner defined the "hype cycle" to describe the way that people tend to respond to new and emerging technologies. It's a pattern that should be familiar to anybody who's been around long enough to see technologies go from initial hype to eventual acceptance.

First comes the "peak of inflated expectation" as excitement spreads around a new technology's game-changing potential, often beyond the borders of what might be regarded as rational. This early, heady, enthusiasm can rapidly give way to the "trough of disillusionment" when it becomes clear that things aren't "different this time" after all.

This "boom and bust" can be damaging as it can give rise to inappropriate implementations by early adopters. The disillusionment and disappointment that accompanies any hype hangover can lead to potential value being overlooked. A technology may be discarded for failing to live up to the hype, regardless of the value it can provide.

Eventually, the reputation of a technology recovers and it passes into the "slope of enlightenment". A more sensible and nuanced understanding of the potential value for the technology starts to come into view, leading to the "plateau of productivity" where it assumes an accepted niche where its capabilities are well understood.

Dunning Kruger and the long road to wisdom

The analysis bears a similarity to a commonly shared analysis extrapolated from the Dunning Kruger effect. This is the idea that poor performers will tend to over-estimate their own abilities as they cannot recognise the qualitative difference between themselves and others.

Joseph Paris explored how Dunning Kruger can affect an individual's behaviour over time. The resulting analysis is very similar to the hype cycle. The individual may quickly reach the "peak of mount stupid", which can give way to the "valley of despair" as their misguided confidence becomes apparent. This is followed by salvation in the form of the "slope of enlightenment" that eventually leads to the "plateau of sustainability".

Both analyses are referring to how wisdom evolves over time. Where Paris's extrapolation of Dunning Kruger explains how an individual can reach a more sustainable balance, the hype cycle talks more in terms of a collective's response to new technology.

The lesson here is that utilising new technology effectively will inevitably involve some long, hard work. Early enthusiasm should be treated with caution until it can be better tempered by experience. We should also be wary of being too dismissive of a new technology as a reaction to unhelpful initial hype. The challenge here is to chart a reasonable course between these two extremes.

This has been summed up nicely by Roy Amara, whose adage about forecasting the effects of technology is known as Amara's Law, i.e.

We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run.

Hoping for a softer landing

Where does this leave our large language models? Very much on the "peak of inflated expectation". Technology often thrives on the new, and an implementation of AI that appears to be accessible to allcomers is worth getting excited about. The wider economic environment also has an influence here. It's no surprise that an industry suffering from layoffs and consolidation is focusing on GPT as a much-needed source of good news.

The problem is that these models may not able to live up to some of the hype that's been generated for them. This is a shame, as it could lead to widespread disappointment and cynicism that may undermine future applications.

The "plateau of productivity" is more likely to settle around using these models as labour saving tools for creative tasks and interpretting data sets. This will make us more productive, but may not usher in the brave new world that is currently being glimpsed on that "peak of inflated expectations"...

Filed under Architecture and Strategy.