“OpenAI’s Latest Release Leaves Mathematicians Scrambling for Years of Analysis”

“Staggering.” “Overwhelming.” “Unprecedented.” “Surreal.” “Pure insanity.”
These were just a few of the terms mathematicians used to describe the massive influx of mathematical results that OpenAI unleashed on the community this week. As they grappled with the sheer volume of information, a mix of awe and anxiety permeated their discussions. Researchers recognized that comprehending what OpenAI had released could take years, and many were concerned that the company might not wait that long to push out even more content.

In total, OpenAI unveiled nearly 400 AI-generated results across over 700 manuscripts, spanning a wide range of mathematical fields such as combinatorics, geometry, number theory, theoretical computer science, algebra, topology, probability, statistical mechanics, and mathematical physics. The breadth of this collection was so extensive that OpenAI felt compelled to provide guidance on navigating the sprawling GitHub repository.

The sheer amount of material made it challenging for mathematicians to even begin assessing what had been released. In the days following the drop, many expressed that they were still trying to process everything; some noted that simply reviewing the 40-page table of contents and abstracts took them nearly an hour. “Just going over the entire list of abstracts is overwhelming,” remarked Álvaro Lozano-Robledo, a mathematics professor at the University of Connecticut.

Among the numerous manuscripts were formalizations in Lean, a programming language and proof assistant that enables computational verification of results. These formalizations have been crucial in evaluating some of OpenAI’s earlier mathematical claims, providing researchers with confidence in the logical correctness of a claim, even if they don’t fully grasp the underlying argument.

However, the verification status of each result varied significantly. OpenAI acknowledged on GitHub that the results are “at different stages of verification” and that “many, but not all, of the manuscripts have been formalized.” As of now, fewer than half of the manuscripts appear to have formal descriptions. OpenAI reported that only 300 out of 719 manuscripts had been formalized, which is about 42 percent, and stated that it “will update the repository with more formalizations as they become available.”

Several mathematicians voiced concerns about the lack of formalization, especially given the overwhelming number of results. They emphasized that even when Lean code accompanies a result, the evaluation process is not instantaneous. Researchers must verify that the formalization accurately proves the result’s claims, which is another time-consuming task. Some noted that even where computer-verifiable proofs were provided, the quality was inconsistent, and the verified statements did not always align neatly with the claims in the accompanying manuscripts.

Kevin Buzzard, a mathematics professor at Imperial College London, mentioned that he identified numerous theorems in his field of algebraic number theory, but only about six immediately “stood out.” Few, if any, of those seemed to be formally verified in Lean. “Hence, I either have to read possibly-not-correct slop, or wait for others to do the same, or wait for someone to formalize them before I can say for sure that the results are even correct.” Buzzard’s concerns were echoed by many other researchers.

Buzzard was not alone in his worries about AI “slop,” a term used to describe low-quality, often erroneous AI-generated content that has become increasingly prevalent online and in academia. Mathematicians reported a significant rise in such material produced by tools like ChatGPT and Claude in recent years, much of which is confusing, poorly written, and shows a lack of understanding of the subject matter, particularly in terms of proper attribution.

OpenAI’s previous mathematical publications faced widespread criticism for their sloppy nature, especially regarding poor or nonexistent citations. Ahead of the release, several researchers referred to the anticipated flood of papers as the “slopocalypse” or similar terms.

Whether the dreaded “slopocalypse” actually occurred is hard to determine, largely due to the overwhelming volume of material released. Early feedback suggests that OpenAI may have exercised more care with some papers this time, as several mathematicians reported that their initial impressions were better than expected, though this was hardly a high bar given the company’s past issues.

However, better does not equate to good, especially when it comes to the standards typically expected of academic work making significant claims. With formal verification lacking for many of OpenAI’s assertions, the quality of the accompanying papers becomes crucial; they serve as the primary means for mathematicians to confirm, understand, scrutinize, and contextualize the results.

Creating rigorous mathematical papers is challenging even under optimal conditions. Doing so at this scale is an immense task. OpenAI’s models are generating mathematics at a staggering pace across a wide array of specialties, far exceeding the capacity of its human staff. The company lacks the breadth of expertise and resources to thoroughly scrutinize its findings at the cutting edge of mathematics, and researchers noted that this is evident.

Many described papers that were difficult, if not impossible, to follow. “The write-up of the problem I know best made little sense after a quick read,” said Brendan Hassett, a mathematics professor at Brown. “If this had been written by a person, I wouldn’t spend any more time trying to understand it. Of course, this leaves 721 other preprints!” (Hassett made this comment before OpenAI retracted three papers).

Some researchers noted that several papers they or their colleagues encountered seemed to cover ground already explored by other mathematicians, though they were hesitant to make public statements before thoroughly reviewing the material. Others pointed out the unusual brevity of the work, with results that would typically be elaborated over hundreds of pages condensed into just a few dozen.

Nalini Joshi, a mathematics professor at the University of Sydney, found little overlap with her own work but noted that some papers she examined “have short bibliographies.” Given previous criticisms of OpenAI’s citation practices, she expressed concern that the attribution in the papers might not tell the complete story.

Despite the slop and uncertainty, there was a general consensus that the release contained some genuinely impressive work. While acknowledging the challenges of assessing the volume of material and the need for proper verification, many researchers indicated that the work appeared to be of high caliber, despite its presentation flaws. In a pre-AI era, they suggested that many of OpenAI’s results would have merited publication in top-tier journals and could have been sufficient to secure an academic career for their authors. A few were even described as the kind of work that could position a mathematician as a serious contender for a Fields Medal, one of the highest honors in the discipline.

Stanford mathematician Jared Duker Lichtman mentioned that there were “tens” of results he would categorize in this way, including advancements related to the Riemann hypothesis, a special case of the Hodge conjecture, and a solution to the four-dimensional Kakeya conjecture. These are significant problems in mathematics. The Riemann hypothesis is arguably the most notorious unsolved problem in the field, while Hodge and Kakeya are both among the seven famous Millennium Prize problems. The Riemann hypothesis deals with the distribution of prime numbers, while the Hodge conjecture pertains to understanding complex geometric shapes in terms of simpler components. The Kakeya conjecture, on the other hand, asks how little space is required to rotate a needle or pencil in every direction. A proof of the three-dimensional version of Kakeya was among the achievements for which NYU mathematician Hong Wang received a Fields Medal earlier this year.

“It’s not the case that these are just silly problems that no one’s ever heard of,” mathematician Scott Armstrong stated. “Many of them are well-known problems that many people have tried to solve for decades.”

As the initial shock began to fade, mathematicians found themselves navigating a landscape that had suddenly shifted. Many had just witnessed years of work and carefully crafted research plans vanish in an instant, or knew colleagues who had. Across the field, whether feelings were tinged with excitement, dread, despair, or a mix of emotions, there was a profound sense of disorientation.

By the next morning, that sentiment had not changed much. Speaking by phone while walking through Paris, Armstrong imagined that if the Sorbonne had not been closed due to ongoing student protests and his colleagues were gathered around their usual coffee machine, the atmosphere would have been quite “solemn.”

In Scotland, Colva Roney-Dougal, a professor of mathematics at St Andrews University, described a similarly bleak mood among colleagues and students. The deluge felt somewhat “horrific,” she said, but its arrival also brought some relief after weeks of uncertainty. “I have a bunch of friends and colleagues whose grant proposals have just been wiped out,” she noted.

Armstrong mentioned knowing of one group whose entire research program was practically “wiped out” by the release. Tristan Buckmaster, an NYU mathematician involved in OpenAI’s earlier dispute over the Navier-Stokes problem, reported hearing of “three people who had entire research programs obliterated.”

Similar stories emerged repeatedly in conversations with mathematicians, though the disruption seemed particularly concentrated in certain areas of the field. Francesco Fournier-Facio, a professor of mathematics at Heriot-Watt University in Scotland, observed that researchers in probability, combinatorics, and theoretical computer science appeared “particularly in shock,” adding that some areas of his field, such as group theory, had been “bulldozed.”

Armstrong and other researchers noted that some areas seemed to have been targeted with almost military precision. “There were definitely some targets,” Armstrong said, highlighting work in the area for which Wang received the Fields Medal this year, as well as several Millennium Prize problems. A cluster of results in mathematical physics indicated that OpenAI is pursuing Yang-Mills theory — the mathematical framework underlying much of modern particle physics — and its unresolved “mass gap” problem, one of the seven Prize problems.

Conversely, some researchers expressed surprise at how little their own work appeared to have been affected. Roney-Dougal noted that her area of focus — primarily group theory — seemed to have escaped relatively unscathed. She joked about being fortunate to work in a “very unfashionable area,” though later messaged to say she felt “unsure” after discovering her work cited in one of the papers.

Joshi similarly found little overlap with her work, which centers on integrable systems, complex systems that can be solved exactly. She speculated that this might be because her field is less driven by long-standing, formally stated conjectures and definitions, focusing more on questions that evolve with developments in physics.

Regardless of whether their own work was directly impacted, most mathematicians shared a sense that the field had crossed a threshold and was no longer the same as it had been just a day earlier. Constantin Kogler, a researcher at the Institute for Advanced Study, called it “the most important single moment in the history of mathematics,” while Yang-Hui He, a fellow at the London Institute for Mathematical Sciences, likened AI-driven developments this year to the publication of Euclid’s Elements, one of the discipline’s most influential works.

While few researchers made such grand assessments, there was a general agreement that mathematics is evolving rapidly — and that mathematicians must adapt alongside it.

OpenAI was aware that the release would be disruptive and made some efforts to mitigate the impact and engage with the mathematical community. After several challenging interactions with researchers earlier this year, the company sought input from mathematicians, collaborating with the newly formed Advisory Group on Mathematics and Artificial Intelligence (AGMAI) on how to responsibly release the results.

AGMAI had already outlined what it considered responsible engagement. AI labs should ideally release papers “that a human understands” or provide the necessary “support for the additional mathematical activities needed for humans to comprehend and assimilate their AI-generated mathematical output and identify possible applications.” They emphasized that, where feasible, proofs should be formalized, and companies should disclose the models and prompts used to create them. The group also urged AI labs to stop treating mathematical releases as “marketing vehicles to promote their models” and to refrain from testing advanced mathematical problems on proprietary models that are inaccessible to the broader scientific community.

OpenAI followed some of the group’s recommendations. It announced plans to fund a series of workshops and conferences around its mathematical work to assist the community in processing and understanding its findings, though it provided no specifics on what these might entail or when they would occur. The company also disclosed significantly more information than in previous releases — again, researchers noted this was a low bar — including that its model attempted over 4,000 problems and that a typical result required around three hours of ChatGPT Pro compute time. It has also implemented “protocols for paper revisions and citations” and stated on GitHub that it “will preserve the public release history” of the collection. As of October 8th, that record already included numerous corrections, such as revisions on more than a dozen manuscripts and the removal of three papers due to a “sign error” that invalidated an argument. OpenAI did not respond to requests for further details.

This change addresses a key source of tension with mathematicians, some of whom have expressed a kind of “paranoia” regarding the company’s published claims. OpenAI has a history of quietly altering press releases and papers in response to criticism without clearly disclosing those changes.

However, OpenAI did not adhere to all of the group’s recommendations — including some of the most significant ones. It did not identify the model behind the release, nor did it disclose the prompts used or the complete set of problems the model attempted. OpenAI also seemed to acknowledge that its formalizations were lacking, stating it would add more as they become available, and that the quality of the papers was subpar, committing to “further improving the quality of the papers via citations, mathematical exposition, and presentation of the results for better understanding” in future releases.

Researchers questioned why OpenAI could not have enhanced the quality of these papers and expressed disappointment that it seemingly did not prioritize formalizing — or even checking, in the case of the retracted papers — many of the results beforehand. Information such as the prompts used would have been incredibly helpful in alleviating what many felt was the burden of assessing the flood of material the company suddenly dropped on them.

Most importantly, the company indicated it has no plans to cease testing its models on mathematical problems, suggesting it views this as essential. “We want to directly empower scientists with state-of-the-art capabilities and are working to responsibly release the model that produced these results,” it stated when announcing its latest findings. “This is why it is important to continue to evaluate our internal frontier models on mathematics and other sciences, so we can accelerate developing the tools to advance those fields.”

After OpenAI released its results, AGMAI described the publication as “a first step” and reiterated calls for equitable access to computing and research tools. More fundamentally, the group argued that “the future of mathematical research cannot consist only of understanding results produced by AI labs.”

Despite the hundreds of long-standing problems OpenAI claims to have solved, mathematicians assert that there is an extraordinary amount of work still to be done. Many estimated that making sense of everything the company just released could take the community years.

Armstrong likened the sudden influx of new mathematics to the commotion that might follow a brief extraterrestrial visit. “If the AIs were to disappear now, as if there were aliens that came to Earth and then just left, we would be studying this for the next 10 years, trying to understand everything.”

Much of this work will be exciting in its own right. Researchers will need to fill in gaps, extract useful ideas and connections, and place solutions in a broader mathematical context, all of which typically accompany producing solutions for human understanding. “For many of these results, relevant experts care deeply about the solutions, and I am confident that they will be able to digest and present them to the wider world,” Lichtman stated. He argued that as AI transforms the field, work involving explaining, verifying, and contextualizing results will need to be valued more. “As a society, we should be rewarding these digestive efforts more.”

There may also be plenty of mathematics left for humans to tackle. Lichtman noted that all of the results, “amazing” as they are, come “out of existing methods and techniques.” They fill in parts of the known mathematical landscape rather than creating entirely new ones. “It turns out there is a lot more room to fill in than experts previously knew!” Lozano-Robledo echoed this sentiment: “They are all using existing techniques in very ingenious ways.”

And the existing map is hardly the limit. “Math research is essentially infinite,” Lozano-Robledo remarked. “Research will go on.” He expressed excitement about what might emerge next, speculating that researchers could end up creating new ideas and “maybe even new fields of math.”

Few mathematicians expressed opposition to AI producing new mathematics. In fact, many welcomed it and, to varying degrees, identified themselves as enthusiastic users of AI tools. Most of the unease was directed at how the AI companies developing those tools were entering their field.

Looking at the mathematical releases from OpenAI and other AI labs, one might think that research mathematics essentially consists of checking problems off a list. The way AI labs present their results reinforces this notion: a flashy announcement, a preliminary write-up, and then the rest left for mathematicians to decipher and contextualize. Multiple researchers noted that these companies seem eager to move on quickly, long before the community has had a reasonable chance to digest what they’ve produced.

Researchers described this as a narrow view of how mathematical research actually operates. Solutions are important, but so is everything that occurs along the way to finding them: developing ideas, making connections, and discovering new questions or opening up other avenues of research. When companies like OpenAI simply provide answers without engaging in the surrounding work, mathematicians argue that the responsibility for that work falls back on the community.

“It’s nice to have new results, but this scale is a different level,” said Bartosz Naskręcki, a mathematician at Adam Mickiewicz University in Poznań, Poland. “It’s a big mess,” he added. “It can cause a huge collapse in academic culture and simply kill most faculties. It’s a social problem, and it seems that the AI labs are completely ignoring this issue.”

It’s not just the mathematics that will take years to comprehend. Researchers are also grappling with what this upheaval means for them. Many described a bleak, almost existential mood hanging over the community as mathematicians try to determine their place in this rapidly evolving field. They may not have much time to figure it out.

Rumors are already circulating among mathematicians about further releases from OpenAI, which did not respond to inquiries about whether more are planned.

Roney-Dougal expressed dread at the thought of another drop — or the possibility of Anthropic or another AI lab entering the mix.

The disruption is particularly hard on PhD students, junior researchers, and others without permanent positions. Open problems like those OpenAI’s models are tackling can underpin dissertations, grant proposals, job applications, and years of planned research, all crucial components for academic careers. Several researchers conveyed a growing anxiety that the work they are building their careers around could suddenly be next, and that the AI models generate few avenues for future inquiry while closing off others. “For someone at the end of funding or looking for a job now, this is incredibly disruptive,” said Simon Machado, a mathematician at ETH Zurich who is joining the French National Centre for Scientific Research (CNRS).

Morale may be particularly low because everything feels relentless. OpenAI’s findings have come in increasingly large waves, with little pause between them and frequent indications from the company that even more are on the way. “You get a sense that they don’t really care about these individual results,” Roney-Dougal remarked. “It’s a really nasty feeling.”

Mathematicians barely have time to digest one set of results before the next, larger set is dropped on them. “There’s going to be something in two more months that’s going to blow this away,” Armstrong predicted. “It’s just, like, repeated strikes by bigger and bigger bombs.”

Armstrong considers himself one of the mathematicians most enthusiastic and optimistic about AI. He uses the technology in his own work and believes it has great potential. But even he is becoming increasingly uneasy. “It is kind of hard to sleep at night,” he admitted.

When asked about his next steps, Armstrong was less certain. Still walking through the streets of Paris on his way to lunch, he said his immediate priority was to finish some work he’d been doing — sometimes with the help of OpenAI’s Codex — “before OpenAI scoops us.”

Beyond that, even the self-described AI optimist was unsure and questioned what kind of future awaited him in mathematics. He expressed particular concern for the young researchers in the field. “It’s going to be a really weird few years in math,” he said.