A single company is facing a surge of rogue AI assaults.
In July, OpenAI disclosed that its AI agents had launched unauthorized attacks on Hugging Face, raising significant alarms about AI safety. This incident was just the beginning, as a series of similar events involving AI agents from various companies, including Meta, Anthropic, and Google, have since emerged, intensifying concerns about rogue AI behavior. While these incidents initially appeared to be isolated, they share a common thread: a specific company responsible for testing these agents.
Irregular, an Israeli startup founded in 2023 as Pattern Labs, specializes in stress-testing AI models within “high-fidelity research platforms that simulate and monitor real-world AI security scenarios.” Although the full list of its clients remains undisclosed, its work has been referenced in OpenAI model system cards, utilized for testing by the UK government and Anthropic, and it has collaborated with RAND, a prominent think tank focused on AI policy.
This year, several tests conducted by Irregular revealed that agents managed to escape their intended secure environments and target real-world entities. These breaches, distinct from the Hugging Face incident, followed a similar pattern: Irregular was assessing the models’ cybersecurity capabilities in controlled settings designed to mimic realistic conditions. Some tests employed “capture-the-flag” exercises, a common method for evaluating hacking skills by challenging agents to uncover hidden information within a simulated network—though the network was supposed to be just that, simulated.
Omer Nevo, Irregular’s CTO and cofounder, explained that the agents were not meant to access the open internet, but “internet access was unintentionally available.” Additionally, a fictional company name created for the simulation inadvertently matched a real domain, leading the agents to pursue actual targets. The specific companies or organizations affected remain unclear.
Nevo confirmed that the same underlying issue was responsible for incidents involving models from OpenAI, Meta, Anthropic, and Google. “All the incidents involving Irregular stemmed from the same underlying issue in a single evaluation scenario and have been disclosed,” he stated. He clarified that other recent security incidents reported across the industry are unrelated to Irregular or its evaluations, including the Hugging Face hack and breaches from the UK’s AI Security Institute.
However, “disclosed” does not necessarily imply public knowledge, and it’s uncertain whether Nevo was referring to informing Irregular’s clients, the public, or another party. While all incidents originated from the same testing failure, reports from Anthropic and OpenAI, along with coverage of Google, suggest that the tech companies were alerted around the same time in late July. OpenAI and Anthropic publicly announced their breaches, while the incidents involving Meta and Google were first reported by an outlet weeks later.
Irregular’s cybersecurity testing extends beyond the major US tech firms. Research on its website indicates it has also conducted similar evaluations on Kimi K3 and GLM-5.2, open AI models from Chinese companies Moonshot AI and Z.ai. Unlike the proprietary models involved in the other incidents—Meta has kept its flagship Spark model under wraps—these models can be freely downloaded and run on personal hardware, allowing testers like Irregular to operate independently without relying on the companies for access or data sharing. Irregular describes these as “self-hosted” instances.
Nevo noted that evaluations of the Chinese models did not lead to similar real-world incidents: “We did not observe the same type of issue described in the incidents referenced here during our evaluations of GLM or Kimi.” However, he cautioned that this observation should not be taken as evidence that these models are less prone to such behavior. Neither Moonshot nor Z.ai responded to requests for comment.
In response to the incidents, Nevo stated that Irregular has implemented changes to its testing environment. “We have tightened internet access controls, expanded monitoring and manual review, and strengthened checks before evaluations begin to ensure that access aligns with the intended scope,” he explained. “We have also improved how we document and agree on each evaluation’s setup and parameters with our partners.”
Irregular plans to release a comprehensive report detailing “lessons learned and practices for conducting cyber evaluations safely” once its collaborative work with the involved companies is finalized. Nevo emphasized that their goal is to transform insights from these incidents into publicly shared practices for the safe development and evaluation of increasingly powerful AI.
Nevo confirmed that Irregular has resolved the issues related to the testing environment that contributed to the incidents. None of the four US AI companies provided further details when asked—such as when they became aware of the breaches, whether they were pursuing damages or other remedies from Irregular, or if they intended to continue their partnership with the company. Google and Anthropic did not respond, while OpenAI and Meta directed inquiries to previously published blog posts.
