Instagram and X Face Daunting Deadline for Deepfake Detection Technology

The latest developments in deepfake detection are about to face a significant challenge. India has announced new regulations requiring social media platforms to swiftly remove illegal AI-generated content and ensure that all synthetic materials are clearly labeled. Tech companies have long claimed they could manage this on their own, but now they have only days to comply with these legal requirements, which take effect on February 20th.

With a staggering 1 billion internet users, many of whom are young, India represents a crucial growth market for social media. This means that the obligations set forth could influence deepfake moderation efforts globally, either by pushing detection technologies to improve or by compelling tech companies to recognize the need for new solutions.

Under the revised Information Technology Rules, digital platforms must implement “reasonable and appropriate technical measures” to prevent users from creating or sharing illegal synthetic audio and visual content, commonly known as deepfakes. Any generative AI content that isn’t blocked must include “permanent metadata or other appropriate technical provenance mechanisms.” Social media platforms have specific responsibilities, such as requiring users to disclose AI-generated or edited content, using tools to verify these disclosures, and clearly labeling AI content so that users can easily identify it as synthetic, including adding verbal disclosures to AI-generated audio.

However, achieving this is no small feat, especially considering the current limitations of AI detection and labeling systems. One of the most effective systems available is C2PA (Content Credentials), which attaches detailed metadata to images, videos, and audio at the point of creation or editing, providing an invisible description of how the content was made or altered.

Despite the use of C2PA by major tech companies like Meta, Google, and Microsoft, the system has not proven effective enough. While platforms like Facebook, Instagram, YouTube, and LinkedIn do label content flagged by C2PA, these labels can be hard to notice, and some synthetic content that should have this metadata is slipping through. Additionally, social media platforms cannot label content that lacks provenance metadata, such as materials generated by open-source AI models or “nudify apps” that do not adhere to the voluntary C2PA standard.

India boasts over 500 million social media users, with 500 million on YouTube, 481 million on Instagram, 403 million on Facebook, and 213 million on Snapchat. It is also estimated to be X’s third-largest market.

Interoperability remains a significant challenge for C2PA, and while India’s new regulations may drive adoption, the metadata is not foolproof. It can be easily stripped away during file uploads, and the new rules mandate that platforms must not allow metadata or labels to be altered, hidden, or removed. However, time is running out for compliance, with platforms like X that have yet to implement any AI labeling systems facing a tight nine-day deadline.

Adding to the urgency, social media companies in India must now remove unlawful materials within three hours of discovery or reporting, a drastic reduction from the previous 36-hour window. This applies to deepfakes and other harmful AI content as well.

The Internet Freedom Foundation (IFF) has raised concerns that these rapid changes could lead platforms to become “rapid fire censors.” They argue that such short timelines eliminate meaningful human review, pushing platforms toward excessive automated removals.

Given that the amendments specify provenance mechanisms should be implemented to the “extent technically feasible,” it seems the officials behind India’s order recognize that current AI detection and labeling technology is not fully prepared. The organizations supporting C2PA have long asserted that the system will function effectively if widely adopted, making this a pivotal moment to demonstrate its capabilities.