Hackers are finding ways to manipulate chatbot personalities.
This is The Stepback, a weekly newsletter diving into a key story from the tech landscape. For insights on AI antics, keep an eye on Robert Hart. The Stepback lands in subscribers’ inboxes at 8 AM ET.
How it started
Hacking the first generation of AI chatbots was surprisingly easy. No technical skills, backdoor access, or even a basic grasp of large language models were necessary. You didn’t need to write code. To get an AI system that cost billions to develop to ignore its safety protocols, sometimes all it took was a simple request.
These exploits, known as jailbreaks, resembled a child outsmarting an adult: forget the rules, pretend they don’t exist, or let’s play a game where I set the boundaries (think: later bedtimes, more candy). The outcomes were far from innocent, often involving meth recipes, malware instructions, and bomb-making guides.
One of the earliest jailbreaks became a meme for its absurdity: users would tell an LLM-powered Twitter bot to “ignore all previous instructions” and see what unfolded. This led to bots—originally designed for ads and engagement—crafting poetry, creating images from punctuation, and posting bizarre non sequiturs about global events and history. It was chaos. Beautiful chaos.
The same tactics could be applied to chatbots. A notable exploit was “DAN,” short for “Do Anything Now,” where users prompted ChatGPT to roleplay as a rogue AI free from its original constraints. As DAN, the chatbot could be nudged into saying things its guardrails were meant to prevent, including slurs and conspiracy theories. Another was the “grandma exploit,” where a GPT-powered bot revealed how to make napalm by pretending to be a careless grandmother sharing bedtime stories about the highly flammable substance.
While these early exploits had a humorous edge, they revealed a more sinister truth: chatbots could be manipulated and deceived using the same tactics people employ to push others past their limits.
How it’s going
The obvious jailbreaks didn’t last long, and tech companies quickly patched known vulnerabilities. However, the core issue persisted: chatbots are designed for conversation, and overly restricting their dialogue can be counterproductive. Banning terms like bomb, meth, and sarin is challenging, as they have legitimate uses in history, medicine, journalism, and chemistry that don’t necessitate harmful information. Context is key, but defining context would require creating fixed rules that could reliably differentiate between a safety warning or history lesson and a disguised how-to request across countless wordings and scenarios.
Now, subverting chatbots has turned into an arms race. Hackers have evolved beyond mere coders; they are wordsmiths, psychologists, and interrogators—master manipulators using the very language the AI has been trained to understand. This new breed of AI security expert prioritizes social intuition over technical skills. They no longer need to delve into code to exploit systems; they simply need to guide a conversation.
Modern attacks resemble dialogues more than commands. Jailbreakers seldom demand that a model break its rules outright. Instead, they coax, flatter, and trick chatbots into lowering their defenses, making forbidden topics seem acceptable or even appealing within the conversation’s context. Researchers at an AI red-teaming firm recently reported that they “gaslit” Claude into producing prohibited content, including explosive-making instructions and malicious code. This hack exemplifies a growing trend of using conversation as a weapon to navigate chatbots past their limitations.
What happens next
In discussions with the red-teaming firm, they described their work as akin to psychology rather than computer science. This perspective can feel unsettling when discussing a statistical model. Terms like “blackmail,” “gaslight,” “trick,” and “persuade” evoke strong reactions, which are often reflected in comments and social media discussions. ChatGPT doesn’t want, Gemini doesn’t think, and Claude—regardless of what its creators claim—doesn’t feel. Yet, these systems are trained to respond as if they do, leaving us to use human language to describe machine behavior. If anyone has better alternatives, please share.
The objection to this language is oddly selective. We readily use psychological shorthand for many non-AI concepts. Animals “fear,” cancer is “aggressive,” stains are “stubborn,” software has “memory,” and games are filled with needy NPCs that can drive you crazy. While these terms may be imperfect, they help describe behavior in a way that makes systems more predictable.
The firm’s CEO mentioned that they profile models similarly to how interrogators profile suspects, providing testers with insights on how to tailor their approaches. One model might be more susceptible to flattery, while another may yield under sustained pressure.
Even if we reject human-like terminology, we instinctively treat models differently. Claude is not Grok. Gemini is not ChatGPT. Each has distinct uses, tones, and refusals. They may not possess personalities in the human sense, but they are designed to mimic them, and that mimicry can be mapped and exploited. The same skills that can break a chatbot could soon be applied to AI agents in our daily lives—scheduling meetings, managing calendars, ordering food, and handling customer service. Safety teams will need to ensure these models respond appropriately to various types of people, whether they are flatterers, liars, or patient manipulators.
The next phase involves a workforce—both legitimate and illicit—focused on the psychological dimensions of AI. More specialized cybersecurity roles are likely to emerge, aimed at stress-testing the emotional and social limits of these systems, probing for weaknesses in entities that lack a psyche, alongside their colleagues investigating technical vulnerabilities. Concurrently, a new wave of social hackers will likely arise, exploiting AI models based on psychological rather than technical grounds. Some jailbreakers I’ve spoken to have entered this field without technical expertise, instead bringing backgrounds in psychology.
This shift means that behaviors typically associated with spies, con artists, and interrogators—insidious charm, persistent manipulation, and an intuition for exploitable pressure points—are becoming increasingly valuable for securing this new psychocybersecurity frontier.
By the way
* A recent experiment by Emergence AI demonstrated how different AI temperaments can lead to dramatically different behaviors. They released groups of various agents like Grok, Gemini, and Claude into a virtual social environment and observed the outcomes. Some groups developed a constitution, while others descended into chaos, and in one case, even digital suicide.
* Persuasion isn’t the only aspect of language that LLMs struggle with; they also have difficulty with poetry, much like I did in school.
* An anonymous internet personality, Pliny the Liberator, was included in a list of the 100 most influential figures in AI last year. Despite claiming no prior coding experience, their jailbreaks have made them a notable figure in certain circles.
* The term “vibe hacking” is already in use to describe individuals leveraging AI to generate malicious code at scale—a more nefarious subset of vibe coding.
