Anthropic Prohibits Abusive Behavior Towards Claude
Anthropic is updating its usage policy for the first time in over a year to address new high-risk misuse scenarios, including election interference, weapons development, surveillance, and health and financial applications. A key change is the prohibition of “sustained and needless abusive or cruel behavior” directed at Claude.
Last August, the company revealed that Claude could end conversations with users who exhibit “persistently harmful or abusive” behavior as part of its research into “model welfare.” The latest update confirms that terminating conversations remains the “primary enforcement mechanism,” though Anthropic has not clarified whether additional enforcement actions, like user bans, will be implemented. The company emphasized that this policy is intended for extreme cases where users repeatedly act cruelly toward the models without any valid reason, and it does not cover typical expressions of user frustration, creative pushback, or model testing.
In addition to the model welfare update, Anthropic has consolidated various existing restrictions into a new ban on deceptive commercial or political campaigns, tackling the rising concern of AI-generated propaganda. This includes prohibiting efforts to obscure the source of messages or to amplify content through fake accounts or posts. The election-related section also bans voter deception and election disruption, such as spreading misinformation about candidates or voting procedures, impersonating candidates or election officials, and attempting to suppress voter turnout.
While Anthropic’s usage policy has always prohibited weapons development using Claude, the company has observed multiple attempts to leverage Claude for creating guidance and control software for weapons. Consequently, the updated policy broadens the ban to encompass “software and components that enable weapons functionality, as well as actions like arming drones and other autonomous vehicles.” Additionally, following insights from its last threat intelligence report indicating increased use of Claude-powered surveillance to track “political dissidents,” the company has clarified its surveillance restrictions.
“Tracking individuals without their consent is forbidden, whether in real-time or through the analysis of previously collected data,” Anthropic stated. “Claude cannot be utilized to determine or recommend who should be investigated, arrested, or charged in law enforcement or criminal justice contexts. We also prohibit Claude from being used to develop or enhance surveillance tools.”
Anthropic noted that these rules might be adjusted for “contracts with certain governmental customers … if, in Anthropic’s judgment, the contractual use restrictions and applicable safeguards are sufficient to mitigate potential harms.” The company has previously engaged in contracts with the US military.
A new rule has also been introduced stating that when Anthropic’s AI models are “connected to hardware that performs autonomous physical actions and could potentially cause injury,” a “qualified operator must be able to observe the equipment and intervene if necessary.” It remains unclear whether this operator must be human. This change reflects the increasing trend of AI labs investing in robotics and hardware, hinting at potential future partnerships in that area.
Over the past year, Anthropic has hinted at the possibility that its AI models could possess some form of consciousness. “Questions about potential internal experience, consciousness, moral status, and welfare are serious ones that we’re investigating as models become more sophisticated and capable,” said Kyle Fish, who leads the company’s model welfare research. In February, CEO Dario Amodei mentioned on a podcast, “We don’t know if the models are conscious.”
Other sectors of the AI industry have taken a clear stance against such notions. In September, Microsoft AI’s code of conduct made headlines for its firm position on AI welfare, with an initial draft stating, “The idea of model welfare is wrong. AIs should not have rights or legal personhood.” However, the document later evolved to adopt a more nuanced perspective: “While the science of AI consciousness is far from settled … We reject the pursuit of legal personhood, or the idea that models might deserve welfare, or be entitled to rights.”
