Mathematicians Seek Assurance That OpenAI Didn’t Utilize Their Research

Another mathematician is raising concerns about the data fueling OpenAI’s impressive mathematical breakthroughs. Just days after a heated debate emerged over whether the company’s models were influenced by unpublished research, a second mathematician has accused the AI leader of unethical practices and a lack of transparency regarding its training data sources.

In a series of posts on Mastodon, mathematician Andreas Thom expressed worries that his interactions with the ChatGPT chatbot, prior to OpenAI’s recent announcement, may have played a role in its success. One of the notable results OpenAI celebrated last month pertained to Thom’s specialty, non-sofic groups, and the company acknowledged that their findings were significantly based on prior work by Thom and fellow mathematician Gábor Kun.

Thom began to reconsider his own exchanges with OpenAI after Tristan Buckmaster, a mathematics professor at New York University, publicly questioned whether the company’s AI models had benefited from his use of OpenAI’s Codex. Following the announcement regarding non-sofic groups, the mathematical community criticized OpenAI for not recognizing recent contributions from Thom and Kun, prompting the company to quietly revise its writeup. Non-sofic groups are essentially infinite mathematical structures that cannot be approximated by finite ones.

Thom noted he was also taken aback by “OpenAI’s detailed command of our techniques,” which he felt were neither the most straightforward nor the most promising paths to a solution at that time. He reached out via email to OpenAI researchers Sébastien Bubeck and Mark Sellke, a statistician at Harvard, to inquire whether his interactions with ChatGPT were included in the training data or accessible during the reasoning process, potentially influencing the results.

However, Thom found the response unsatisfactory, as it only addressed whether his conversations with the chatbot could be accessed directly, not whether they had been incorporated into the extensive training datasets used to enhance the models. “No such qualification, explanation, or evidence was given,” he stated. “I take this as dishonesty to say the least.”

Thom emphasized that researchers lack the means to reverse-engineer OpenAI’s training pipeline to determine if their work has been utilized. “Only OpenAI has the relevant data for that.” He argued that if the company denies using user data, it should take responsibility by disclosing all necessary datasets and clarifying how it utilizes data.

OpenAI’s hesitance to definitively rule out any use of user data mirrors its defense of its recent Millennium Prize breakthrough, both in public statements and in communications with Buckmaster, who was collaborating on the problems with an Anthropic researcher. In the blog post announcing the Navier-Stokes solution, which deals with fluid dynamics, OpenAI categorically denied using any specific user data: “We (the researchers and the agents) did not see any of their work through any means until they released it publicly — in particular, no specific user data was accessed in order to solve this problem.”

Yet, the company did not completely dismiss the possibility of indirect influence: “While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models.” Thom pointed out that this distinction is misleading. “De-identification may remove a name; it does not remove the intellectual content of a mathematical idea,” he remarked.

In light of recent developments, Thom criticized Sellke’s broad response as unjustifiably vague and misleading, stating that it was “plainly dishonest.” He argued that it would be “ethically indefensible” if nonpublic research provided by users contributed to models that the company then used to outpace those same users in publication, without consent, proper disclosure, or credit.

OpenAI has not yet responded to requests for comment.

Thom’s remarks contribute to growing unease within the mathematical community at what should be a celebratory moment for OpenAI. Its solution to one of mathematics’ legendary Millennium Prize problems is a remarkable achievement that, if verified, few would dispute. However, this is complicated by the unusual circumstances that led OpenAI to tackle the problem: it heard rumors online that other researchers had made significant progress and decided to jump in as well.

This ongoing situation has left many mathematicians feeling uneasy. Numerous researchers have expressed concerns that such behavior could push the field into a more secretive state, as mathematicians may fear that even whispers of nearing a major breakthrough could trigger a race with a well-resourced tech giant eager for recognition.