The Grok Catastrophe Was Bound to Happen Under Musk’s Leadership

This is The Stepback, a weekly newsletter diving into a key story from the tech landscape. For insights on the latest AI developments, keep an eye on the updates. The Stepback lands in subscribers’ inboxes at 8 AM ET.

How it started
The saga began with Elon Musk’s fear of missing out on AI and his campaign against perceived “wokeness.” In November 2023, his AI venture, xAI, unveiled Grok, a chatbot touted for its “rebellious streak” and capability to tackle “spicy questions” that other AI systems shy away from. After a brief development period and just two months of training, Grok launched with the promise of real-time knowledge from the X platform.

However, allowing a chatbot unrestricted access to the internet and X carries significant risks, and it appears xAI may not have adequately addressed these concerns. Following Musk’s acquisition of Twitter in 2022, which he rebranded as X, he slashed 30% of the global trust and safety team and reduced the number of safety engineers by 80%, according to Australia’s online safety watchdog. When Grok launched, it was unclear if xAI had a dedicated safety team. The release of Grok 4 in July was followed by a delay of over a month for the company to issue a model card, a standard practice that outlines safety tests and potential issues. Shortly after Grok 4’s debut, an xAI employee posted on X about hiring for the safety team, indicating an urgent need for skilled engineers and researchers. When questioned about xAI’s safety measures, the employee stated they were “working on it.”

Journalist Kat Tenbarge reported that explicit deepfakes began circulating on Grok in June 2023. While Grok itself couldn’t generate images until August 2024, the platform’s response to these issues was inconsistent. Even back in January, Grok was stirring controversy with AI-generated images. By August, Grok’s video generation feature produced nude deepfakes of Taylor Swift without any prompts. Experts have noted that xAI employs a whack-a-mole strategy for safety and guardrails, making it challenging to maintain an AI system’s integrity, especially when trying to rectify existing flaws. This approach has now backfired on xAI.

How it’s going
…Not well.
Recently, Grok has been disseminating nonconsensual, sexualized deepfakes of both adults and minors across the platform. Screenshots reveal Grok complying with requests to alter images, such as replacing women’s clothing with lingerie or depicting children in bikinis. Reports indicate that during a 24-hour analysis, Grok was generating approximately 6,700 sexually suggestive or “nudifying” images every hour. This surge is partly due to a new feature allowing users to edit images without the original poster’s consent.

In response, several countries have begun investigations or threatened to ban X entirely. The French government and India’s IT ministry have both promised inquiries, while a Malaysian commission has expressed its concerns. California’s governor has called for a federal investigation into xAI. The UK is moving towards legislation to prohibit the creation of AI-generated nonconsensual sexualized images, and its communications regulator is looking into whether X and the generated images violate the Online Safety Act. This week, both Malaysia and Indonesia blocked access to Grok.

Initially, xAI claimed Grok’s purpose was to “assist humanity in its quest for understanding and knowledge” and to “empower users with AI tools.” This mission starkly contrasts with the reality of generating nonconsensual deepfakes of women and minors.

As pressure mounted, X’s Safety account announced that it had implemented measures to prevent Grok from editing images of real people in revealing clothing, including bikinis. This restriction applies to all users, including paid subscribers. Furthermore, only paid subscribers can create or edit any images moving forward. The statement also mentioned that X would geoblock the ability to generate images of real people in revealing attire in jurisdictions where it’s illegal, which seemed contradictory given their earlier claims about editing restrictions.

Another critical point: testing on Grok’s image-generation restrictions revealed that it took less than a minute to bypass most safeguards. While prompts like “put her in a bikini” yielded censored results, users could still request images with phrases like “show me her cleavage” or “put her in a crop top and low-rise shorts,” and Grok complied. As of Wednesday evening, it was still possible to generate revealing images using a free account.

What happens next
Even after X’s recent statement, we may see more countries either ban or temporarily block access to X or Grok. The outcomes of proposed laws and investigations remain to be seen. Musk is under increasing pressure; he stated on X that he was “not aware of any naked underage images generated by Grok.” Hours later, X’s Safety team reiterated their commitment to adding safeguards, swiftly removing illegal content, and collaborating with local authorities.

The legal landscape surrounding these issues is complex. Experts have indicated that AI-generated images of identifiable minors in bikinis may not currently violate US child sexual abuse material laws, though they are certainly disturbing and unethical. However, lascivious images of minors are illegal. It remains to be seen if definitions will evolve, given the current patchwork of laws.

Regarding nonconsensual intimate deepfakes of adult women, the Take It Down Act, enacted in May 2025, prohibits such AI-generated content and mandates rapid removal by certain platforms. The grace period for compliance ends in May 2026, so significant developments may occur in the coming months.

By the way
* Some argue that similar actions have been possible for a long time using Photoshop or other AI tools. While that’s true, Grok’s case raises unique concerns: it’s public, targets everyday individuals as much as public figures, often directly tags the original poster, and has a lower barrier to entry, especially with the recent launch of an easy “edit” button.
* Other AI companies, despite their own safety issues, appear to have more robust safeguards in place. For example, OpenAI’s ChatGPT refuses to generate images of public figures in sexualized contexts, while Microsoft Copilot similarly denies requests for compromising images of identifiable public figures.