Is LLM-based generative AI close to the point of diminishing returns?

• The meteoric growth is now slowing as “performance plateau” looms
• Fresh data is a quickly declining resource
• Are marginal improvements the new order of the day?
• A change of AI scaling strategy may be needed

The bros in Silicon Valley and the analysts in New York are getting twitchy at the news that the latest iteration of Open AI’s artificial intelligence model is showing only limited incremental improvements on earlier versions. There is talk of generative AI (GenAI) technology having hit a “performance plateau”. If that is the case, the AI bubble could begin to leak some of its superheated hype, stock prices could take a knock and the sky could fall.

The model under development is called “Orion”, named after the giant huntsman of ancient Greek mythology. When he grew up, Orion got a bit too big for his massive boots and, in a fit of pique, threatened to hunt down every living thing on earth. So, the gods, naturally, sent a giant scorpion to kill him. Which it did. Later, the Greek goddesses begged Zeus, king of the gods up on Mount Olympus, to place Orion in the heavens, which he did. Now Orion is a constellation and you can still see the line of stars that form his belt in the night sky.

So, why the ancient history lesson you may wonder? Well. Because in a posting on X, Sam Altman, OpenAI’s CEO, observed that “scaling laws [of AI] are decided by god” meaning that, according to a law laid down by an unknowable superhuman being, AI models will improve as they get bigger, more powerful and able to scrape in ever more data. However, Altman’s belief in such a supernatural formula may be mistaken and Orion’s comparatively modest performance increase over its predecessor, GPT-4, may be evidence that it is. 

OpenAI’s Orion, which has been under development for many months, was unveiled to the world over a year ago and hopes were high that its performance would be a huge improvement on what had gone before. There is little doubt that it will be better than GPT-4 but perhaps only to a limited, incremental extent. According to an exclusive report in The Information, the San Francisco, California- headquartered business newsletter that focuses on high-tech industries, ChatGPT, on testing Orion, found that some of its improvements were so slight (particularly where coding is concerned) as to be almost negligible. The question is, why would that be?

One answer is that data is a fast-disappearing resource. Just about every scintilla of information of every type ever created by humanity and ever put online has been remorselessly scraped (often in breach of copyright laws and often by brazen theft) to train AI large language models (LLMs). It is estimated that generative AI will have picked the world clean of textual data by 2028. Then what happens to the scaling model?

One, almost unbelievable, suggested solution is for AI to generate its own synthetic data and add that to the sum of all human knowledge. How’s that for hubris? It was back in 1986 that music journalist David Quantick wrote that “pop will eat itself”, meaning that pervasive pop culture in general and music in particular had become self-referential, taking from and then remaking itself to the point of meaninglessness. If AI starts to generate synthetic (i.e. unreal) data, it really will eat itself – and us along with it.

Another problem for AI is that, for the foreseeable future, the computing power that has been the engine of AI development, will remain limited and expensive, something Sam Altman has openly admitted. He has said that ChatGPT faces “a lot of limitations and hard decisions” about when, where and who to allocate computing resources. If he’s waiting for the time that commercial quantum computing becomes available to solve his problems, he won’t be a bro any longer, he’ll be a pensioner. 

Is data integration a viable solution?

The scaling of AI is challenging and the biggest problem, from the very start, has been the constant availability of high-quality data. Every AI model out there relies on it for training and for inference, which is the process a trained machine learning (ML) model uses to draw conclusions from brand new data. An AI model capable of making inferences can do so without examples of the desired result but, of course, the old maxim of ‘rubbish in, rubbish out’ still pertains, and organisations and enterprises alike face the perennial problem of incomplete and inaccurate data. 

So, AI systems rely heavily on vast amounts of high-quality and relevant data for training and providing accurate outputs, but the integration of data from different sources across an organisation, such as from computers, customer relationship management (CRM) systems, enterprise data warehouses and data lakes, business intelligence platforms, external systems, third-party data and so on is a daunting task. What’s more, AI and ML models are highly dynamic, they can never be static, and so require endless monitoring and maintenance to ensure performance and reliability but data is often compromised, inconsistent and contextless and low-quality data decays model performance.

One way to minimise the problem of AI scaling is data integration because data fragmentation makes it very difficult to aggregate the huge and diverse datasets needed to train accurate AI models. Data integration also permits organisations to automate data cleansing and ensure data validation because cleaned data used for AI training and decision-making reduces the risk of biassed or inaccurate models. Unsurprisingly, the scaling AI models require some powerful computing resources, particularly during the training and inference phases, and the complexity and workload of AI models can vary, requiring dynamic allocation and optimisation of resources. It’s all very tricky.

So a further major challenge is allocation of resources according to the varying needs of different AI models – and managing the operational costs associated with them. The best data integration platforms facilitate the uniform distribution of data across computer clusters and cloud infrastructure to ensure that AI models have the necessary resources for training and inference.

Obviously, the AI sector wants to be able to continue to make step function advances in performance and not meander aimlessly on a performance plateau, tweaking things here and there but failing to make the generational leaps and performance bounds that investors, analysts and users have come to expect. Just look at one example of costs: Sam Altman has confirmed that it cost ChatGPT $100m just to train its first AI model – and that’s peanuts compared to the astonishing forecast that, within the next few years, it could cost $10bn plus per company to train new AI models. Big returns will be expected as the quid pro quo for such costs but if financial figures fail to meet market presumptions, there’ll be trouble – including failures, mergers, acquisitions and even bankruptcies. There’s a lot at stake and it’s getting harder to keep the boom rumbling along.

– Martyn Warwick, Editor in Chief, TelecomTV