Are data and computing limits slowing AI progress?
Core argument: GPT-4 was trained on several thousand Nvidia chips in 2022, while estimates suggest GPT-5 was trained on hundreds of thousands of Nvidia's next-generation processors, with more powerful chips on the way.
For the past five years, companies such as OpenAI and Anthropic have been able to show consistent gains in the performance of their systems by using a simple formula: more data and more computing power equals bigger, better models. Many AI leaders still believe that these “scaling laws” can continue to hold for years to come. But the approach is starting to reach the limits of the available resources. First, AI companies have sucked up all of the available free training data on the internet. They are now seeking more fuel for their models by making data-sharing deals with publishers and copyright holders, but it is unclear if that is enough to push the frontier forward. AI labs are also constrained by computing power. It is very energy-intensive to train and run large AI models. Back in 2022, GPT-4 was trained on several thousand of Nvidia’s chips. Estimates suggest that GPT-5 was trained on hundreds of thousands of Nvidia’s next-generation processors, with even more powerful chips on the way.



