“From the moment GPT-4 came out in March 2023, OpenAI has been working on GPT-5…” reports the Wall Street Journal. [Alternate URL here.] But “OpenAI’s new artificial-intelligence project is behind schedule and running up huge bills. It isn’t clear when — or if — it’ll work.”
“There may not be enough data in the world to make it smart enough.”
OpenAI’s closest partner and largest investor, Microsoft, had expected to see the new model around mid-2024, say people with knowledge of the matter. OpenAI has conducted at least two large training runs, each of which entails months of crunching huge amounts of data, with the goal of making Orion smarter. Each time, new problems arose and the software fell short of the results researchers were hoping for, people close to the project say… [And each one costs around half a billion dollars in computing costs.]
The $157 billion valuation investors gave OpenAI in October is premised in large part on [CEO Sam] Altman’s prediction that GPT-5 will represent a “significant leap forward” in all kinds of subjects and tasks…. It’s up to company executives to decide whether the model is smart enough to be called GPT-5 based in large part on gut feelings or, as many technologists say, “vibes.”
So far, the vibes are off…
OpenAI wants to use its new model to generate high-quality synthetic data for training, according to the article. But OpenAI’s researchers also “concluded they needed more diverse, high-quality data,” according to the article, since “The public internet didn’t have enough, they felt.”
OpenAI’s solution was to create data from scratch. It is hiring people to write fresh software code or solve math problems for Orion to learn from. [And also theoretical physics experts] The workers, some of whom are software engineers and mathematicians, also share explanations for their work with Orion… Having people explain their thinking deepens the value of the newly created data. It’s more language for the LLM to absorb; it’s also a map for how the model might solve similar problems in the future… The process is painfully slow. GPT-4 was trained on an estimated 13 trillion tokens. A thousand people writing 5,000 words a day would take months to produce a billion tokens.
OpenAI’s already-difficult task has been complicated by internal turmoil and near-constant attempts by rivals to poach its top researchers, sometimes by offering them millions of dollars… More than two dozen key executives, researchers and longtime employees have left OpenAI this year, including co-founder and Chief Scientist Ilya Sutskever and Chief Technology Officer Mira Murati. This past Thursday, Alec Radford, a widely admired researcher who served as lead author on several of OpenAI’s scientific papers, announced his departure after about eight years at the company…
OpenAI isn’t the only company worrying that progress has hit a wall. Across the industry, a debate is raging over whether improvement in AIs is starting to plateau. Sutskever, who recently co-founded a new AI firm called Safe Superintelligence or SSI, declared at a recent AI conference that the age of maximum data is over. “Data is not growing because we have but one internet,” he told a crowd of researchers, policy experts and scientists. “You can even go as far as to say that data is the fossil fuel of AI.”
And that fuel was starting to run out.
Read more of this story at Slashdot.