Cloudflare Aims to Protect the Web from AI Threats Using AI Technology

Today, I’m diving into a conversation with Matthew Prince, the CEO of Cloudflare, as part of a two-part series exploring the future of business. Matthew last joined us about two and a half years ago, during what we thought was a pivotal moment for the internet. Fast forward to now, and things have become even more chaotic, largely due to AI, with Cloudflare right in the thick of it.

In June, Cloudflare reported that bots accounted for over half of all internet traffic, a figure that’s only climbing as AI companies increasingly scrape the web and deploy agents to perform tasks online. Cloudflare acts as a gatekeeper between websites and these agents, giving site owners the ability to control access—whether that means blocking all AI tools, allowing them, or selectively permitting those willing to pay for access.

We discussed how to manage these bots, the chaos they create online, and what information might hold value as payment models evolve. I asked Matthew directly whether the outcomes he describes are genuinely beneficial. His thoughtful response has lingered with me long after our chat.

AI is also transforming Cloudflare internally. Earlier this year, the company laid off over a thousand employees—20% of its workforce. Matthew penned an op-ed about this decision, titled “How I Choose Which Cloudflare Employees to Replace with AI.” It’s a compelling mix of AI, tough choices, and organizational dynamics.

Before we jump in, remember you can catch Decoder on YouTube, with new episodes dropping every Monday and Thursday.

Now, let’s get into it: Matthew Prince, CEO of Cloudflare. Here we go.

This interview has been lightly edited for length and clarity.

Matthew Prince, welcome back to Decoder.

Thanks for having me.

I’m thrilled to chat again. It’s been about two years since your last appearance. Looking back, I mentioned it was a momentous time for the internet, but I had no idea how much more significant things would become.

You’re at the forefront of reshaping how companies interact with AI, which is incredibly relevant. I want to discuss bots, the internet, publishers, Google, and everything else you’re involved with. But let’s start with the basics.

Cloudflare is a complex and vital company. Last time, you described it as a service that speeds up the internet and protects it from malicious actors. Is that still how you define Cloudflare?

That’s usually my go-to line at a cocktail party when I want to wrap up the conversation. If I’m interested in a deeper discussion, I’d say Cloudflare aims to rebuild the internet as it should have been constructed from the start, given its significance.

We operate a vast network, with a presence in over 350 cities globally and thousands of data centers. We provide various services to ensure that applications and content online are secure. For users and employees, we ensure safe online experiences. We empower developers to create scalable applications that can reach a global audience using our infrastructure, which is the fastest-growing segment of our business.

Recently, we’ve been contemplating the dramatic shifts in the internet’s business model and how we can help shape a healthier future for it.

Can you connect the dots between the internet’s business model, how it should function, and Cloudflare’s mission to build the internet as it ought to be?

That’s a shift from your previous description of Cloudflare as merely a protector against bad actors. It’s a notable evolution.

The internet’s business model has changed significantly due to AI, bot traffic, and AI scrapers. What do you think the current business model is, and what should it be?

For the past 30 years, advertising has been the primary business model of the internet. While it’s not the only model, it has driven web growth and created the experiences we enjoy today, which is quite remarkable.

Google has been the most influential player in this space, defining the online advertising ecosystem for nearly three decades. While we often think of Google in terms of search, they’ve built a comprehensive advertising infrastructure, acquiring companies like DoubleClick and developing tools like Google Analytics to track web traffic. They’ve been the heroes of what I call the first generation of the internet.

However, the landscape is rapidly changing regarding who or what is using the web. At the Web Summit last November, I was asked when non-human traffic would surpass human traffic. Analyzing our data, we predicted that by the second half of 2027, automated traffic would exceed human traffic for the first time in internet history.

When I was asked the same question at South by Southwest in March, our data indicated that this shift would occur in the first half of 2027. The growth of AI is staggering, and it’s driving a massive increase in traffic.

Just a few months later, in May, my team informed me that automated traffic had already surpassed human traffic online. This trend is accelerating. If we extrapolate, with the caveat that I’ve been wrong in every prediction I’ve made, we believe that in five years, automated traffic could be 1,000 times greater than human traffic. This isn’t because human traffic will decline; we expect it to remain stable, but the explosion of automated traffic is unprecedented.

The challenge is that with 1,000 times more traffic, someone has to fund the infrastructure to support it. This will require bandwidth, servers, and various resources.

The traditional advertising model doesn’t work for bots, as they don’t click on ads or respond to conventional branding. We need to develop a new model to meet the insatiable demand that will be placed on the internet moving forward. I’m dedicating a lot of time to envisioning what that might look like.

When you refer to a business model, it implies customers, revenue, and profits. Is that a realistic expectation?

There’s certainly a lot of revenue.

That may be true for companies like Cloudflare and Google, but what about others?

Or for companies like Anthropic! They generated $10 billion in revenue in a single month. There’s enormous demand for AI services. For an AI company, three key elements are essential.

First, you need top talent and researchers, which is currently scarce. However, every university is ramping up its AI programs, and we’re training people rapidly. Labor markets are efficient, and we’ll see an influx of talent in this space.

Second, you need chips—silicon to power these systems. Nvidia currently leads the market, but many others, including AMD and Qualcomm, are developing their own chips. While there’s a significant shortage of silicon now, that will change.

Third, you need data to feed into these models. Historically, we’ve kept the internet open, allowing bots access to all created content. However, I believe that’s starting to shift. More content creators are saying, “We’ll provide this for free to humans, but if a bot wants it, they need to pay because they don’t engage with ads in the traditional sense.”

This harkens back to some of the original internet protocols. The 402 protocol, established in the early days of Netscape, required payment for content access. We’re collaborating with companies like Coinbase and Stripe to explore how we can enable content creators to monetize the increased traffic they receive, even if traditional advertising revenue isn’t feasible.

Where does that fraction of a penny come from? It would be the fees paid by AI agents and systems, similar to how Spotify or Apple Music operates today.

It’s fascinating that we’re revisiting the 1996 dream of micropayments on the web. Let’s explore that further.

When I mention business models, I’m really addressing the idea of incentives. Perhaps there is a viable model for information where you publish new content, bots find it, and the 402 protocol is revived, leading to crypto micropayments powered by Cloudflare. That’s a future many have envisioned.

However, I’m concerned about the incentives for creating new information online. If I believe most of my audience will be bots rather than people, I might opt to create a TikTok channel instead and monetize that audience. This seems like a significant inflection point.

You mentioned Google, and I’ve raised this concern before. The incentives to publish new information online appear to be in decline. Conversely, the web as an application platform is thriving. Every new app is a web app, or if it’s a desktop app, it’s often just an Electron wrapper. There’s a remarkable shift happening in the web as an application platform, while the information platform is suffering. You’re trying to bridge that gap, right? You want to change the incentives for information creation.

What does a future look like—perhaps with microtransactions or another model—where the incentives for making the web an information platform are as strong as they are for YouTube?

You’ve highlighted a challenging dichotomy. Publicly accessible data on Wikipedia contributions has significantly declined, as contributors question the incentive to add information. In the past, contributors knew their work was being read. Now, people consume information through various answer engines, like OpenAI or ChatGPT, rather than directly on Wikipedia. This shift has led editors to reconsider their contributions.

On the flip side, the web has been plateauing since around 2012. It experienced explosive growth in the late ’90s and 2000s, but since 2012, it has stagnated and even declined until about 2025, when things began to change. Starting in October 2025, various platforms made it easier for anyone to create a website, leading to a surge in application development. As you noted, even desktop and mobile applications are increasingly just wrappers around web applications.

The overall growth of the web, with high-quality content and more contributors, is faster today than it has been since the early 2000s. That’s the tension we’re navigating.

What’s unique about Cloudflare is that we sit in front of over 20% of the web, giving us the ability to address some of these incentive issues. We can quickly implement a system where users pay a fraction of a penny for access to information, which could kickstart the necessary changes.

Without such a system, we face a tragedy of the commons. For instance, if I ask my AI agent where to go for lunch, it might scan every menu in the area, but only one restaurant will receive my business. If this continues unchecked, my agent could end up suggesting a place like Wendy’s after scanning every option, leading to a massive waste of resources. We need a mechanism to impose costs on webpage loads, as bandwidth and server resources are finite, and someone has to pay for that. If advertising and direct commerce aren’t viable, we need a solution to impose some constraints. We’re in a position to implement a minimal fee, perhaps a thousandth of a penny, to help fund the infrastructure.

For valuable content, like news or academic work, there could be a premium charge, allowing creators to monetize their work effectively. However, for this to happen, we need a player like Cloudflare to initiate the process. Once we do, we’ve seen that AI companies are willing to pay, and content creators are eager to participate. The missing piece has been the technology to connect these elements, which we’re actively developing.

The underlying theme of this conversation seems to be scarcity. As a former copyright lawyer, I recognize that copyright law historically imposed a scarcity mechanism—one physical CD meant one copy. The internet changed that, making duplication effortless and leading to the expectation that information should be free. The famous quote, “Information wants to be free,” often omits the second part: “Information also wants to be expensive because it’s hard to generate.” The internet flipped that dynamic.

The scarcity provided by physical media, which gave economic value, vanished, and we assumed information should be free, supported by advertising. This created a tension that was valuable in its own right. You’re discussing how to impose scarcity through technology.

Markets require supply and demand. While demand is crucial, you also want constrained supply. There’s no market for air where we are because it’s abundant. However, underwater, air becomes a constrained resource, creating a market.

Music serves as a relevant example. In the past, most music sales came from physical media. The internet, along with platforms like Napster, commodified music, making it widely available for free. Twenty-three years ago, the music industry was valued at around $8 billion, which, while significant, was not enough to sustain the industry.

Then came Steve Jobs with iTunes, offering songs for 99 cents, complete with cover art and high quality. This appealed to the notion that music should be paid for. While this wasn’t the ultimate winning model, it established that music had value, paving the way for platforms like Spotify. Last year, Spotify returned approximately $12 billion to the music creator ecosystem. We can debate the fairness of distribution, but it’s clear that the model has evolved.

I believe what happens in the music industry often foreshadows trends in other sectors. The shift you’re describing—accessing a vast catalog of music for a small fee—could fundamentally alter the business model of information.

Dan Ek from Spotify mentioned that the music industry has changed significantly. While streaming may not pay much per play, revenue from touring, sponsorships, and licensing has skyrocketed.

However, it’s important to note that private equity is acquiring music catalogs for substantial sums. This indicates that streaming is indeed lucrative, with Spotify alone sending $12 billion back to the music industry.

The comparison I’m making is about the shift in how access to information is monetized. The transition to a model where access is granted for a fee fundamentally changes the nature of information creation.

Dan pointed out that when you search for a popular song on Spotify, you get a relevant result. But if you search for something niche, like a disco song about dancing with a cat, the results may be lacking. Spotify uses this data to inform music creators about unmet demands, which is a fascinating approach.

Now, consider the implications of this for knowledge creation. For the first time, we have a mathematical model of human knowledge through LLMs. We know where they excel and where they fall short. AI companies are eager to fill those gaps with new knowledge.

I can share a personal example: my wife and I own a small local newspaper in Park City, Utah. This year, we expect to earn more from AI licensing deals than from digital advertising. Local media is becoming increasingly valuable in the evolving media landscape, which has been decimated by a focus on volume and scale. Local news is about sharing unique stories, like the opening of a new restaurant.

For AI companies aiming to be the best travel planners, they need access to local insights, like the latest restaurant openings. If they don’t have access to our newspaper, they miss out.

We won’t protect all media, just as the music industry has seen winners and losers. However, the new winners may be those who provide unique, local content—what many internet users crave. If we can align incentives correctly, we can encourage the creation of more original content rather than the clickbait-driven media landscape we see today.

To make this work, we need to find ways to block AI crawlers from accessing content without compensation. Recently, there’s been a lawsuit involving music companies suing an AI firm, and the White House has supported the idea that training AI should be considered fair use.

We’re navigating a complex legal landscape, with various regulations emerging worldwide. The best-case scenario from the Anthropic case suggests that if companies had purchased the content they used for training, it would have been acceptable.

As you build technology, you must consider the evolving market and the need to control access to content.

Different regulations will emerge globally. As a recovering intellectual property attorney, I often think about using IP law, but that’s a cumbersome approach compared to leveraging technology.

I recall discussions with media executives about preventing AI companies from scraping their content. I pointed out that while we face sophisticated hackers, the tech companies are easier to identify and regulate.

We’re seeing a willingness from AI companies to identify themselves when accessing sites, allowing content creators to control how their information is used.

This was a relevant question two years ago, but even Google, which has faced scrutiny, is recognizing the importance of a healthy ecosystem. They’ve struck deals to access content, but now they’re training on it without providing traffic back to the original sources. This shift in value exchange is significant.

I’ve been impressed by Google’s willingness to engage in these discussions. They understand the ecosystem’s value and are taking steps toward transparency regarding their crawlers. If Google commits to fair practices, it could encourage other AI companies to follow suit.

Do you think you need to take a hard stance and block Google’s crawlers to compel them to pay?

As of September 15th, we’re setting defaults for our free customers to block Google from AI training. If they can’t differentiate between AI training and search engine crawling, we’ll block them entirely.

Site owners can opt out if they prefer, but we’re establishing these defaults to encourage fair practices.

I’m optimistic that Google recognizes the need for a healthy ecosystem and that they must adhere to the same rules as other players in the AI market, rather than leveraging their search monopoly to dominate AI.

I’ve had numerous discussions with Sundar Pichai about this. He’s thoughtful and kind, but I sense he feels that publishers have become overly reliant on Google’s traffic without building their own audiences.

This feels like a turning point. We’re blocking Google’s search traffic and crawlers, and other companies are following suit. If Google can’t access this content, it creates a significant gap in their search ranking system.

Cloudflare’s position, covering over 20% of the internet, means that if we block access, it disrupts the entire ecosystem. We’re advocating for a fairer exchange of value, as the current model is unsustainable.

I’m proud of our role in helping the publishing industry shift from a state of despair to one of progress. The deals large publishers are making are improving significantly.

Now, the challenge is how to extend this model to the broader internet. Without it, we risk a tragedy of the commons, where content creation suffers due to a lack of incentives. If creators can’t sustain themselves, they won’t contribute, which harms everyone.

I’ve been critical of Google, but I believe their recent changes indicate a growing understanding of their role in the ecosystem and the need to support it.

What changes do you anticipate from Google?

They’ve committed to greater transparency regarding their crawler’s activities. We’ve encouraged them to separate their AI crawling from web crawling, but they’ve suggested a more efficient approach: announcing their crawler’s purpose when accessing a page and allowing site owners to control what’s permitted.

They’ve indicated that they’ll implement these changes broadly, not just in the UK. I believe this will lead to a stronger web and a better Google.

While we can debate the music industry’s evolution, it’s clear that changes in access and copyright law have transformed the business. The incentives for live performances and residencies have shifted due to changes in distribution.

You’re describing a future where the incentives for creating information may favor bots over people, potentially altering the nature of that information. For instance, local papers might produce content that’s more digestible for AI models.

Instead of reviewing a hotel, you could review every room. For every word in a New York Times article, there are likely hundreds of words that didn’t make the cut. This additional metadata, traditionally constrained by physical space, could be valuable to AI systems.

There’s a wealth of content that’s currently underutilized. I was recently in a meeting discussing the lead image for an upcoming story. We debated which image would elicit the best response from readers, ultimately choosing one while leaving others unpublished.

However, if we published all the images and provided them to an LLM, it would change the nature of our content and its intended audience. The distribution method inevitably alters the content we create.

Is the outcome you’re describing beneficial, where we generate an endless stream of information optimized for AI consumption rather than human engagement?

First, we should consider whether the current model is effective. Many media outlets focus on producing content as cheaply as possible, often resorting to sensational headlines to provoke emotional responses.

While that may be true for some, it’s not universal. Major publications like The New York Times and The Wall Street Journal have maintained their integrity, but the pressure to generate clicks has led to more inflammatory headlines over the years.

This reflects a shift in distribution, primarily driven by social media algorithms. The new distribution model you’re proposing could change the nature of content creation.

The phrase I often use is, “Everything wrong with the world today is Google’s fault.” While that’s an oversimplification, Google has played a significant role in shaping the current media landscape, which often prioritizes attention over meaningful discourse.

What we need to explore is how to shift the focus from sensationalism to genuine knowledge creation. AI companies are seeking new, untapped knowledge, not just the same stories that dominate the news cycle.

For instance, the leaders at major AI firms want to uncover unique insights that haven’t been widely reported. They’re not interested in rehashing the latest political news; they want to discover new narratives.

This is where local media can thrive. Our newspaper in Park City is valuable because it provides insights that AI companies need to enhance their offerings. If they lack access to local news, they miss critical information.

While not all media will survive this transition, there will be new opportunities for those who provide unique, localized content. If we can align incentives correctly, we can foster a media landscape that prioritizes original storytelling over clickbait.

To achieve this, we need to implement measures to prevent AI crawlers from accessing content without compensation. The ongoing legal battles highlight the complexities of this issue, with varying regulations emerging worldwide.

As we navigate this landscape, we must consider how to balance the need for access with the rights of content creators. The goal is to create a sustainable ecosystem that rewards knowledge creation.

Ultimately, the power dynamics in this space are shifting. Cloudflare has a significant influence on the internet, and we’re committed to using that power to foster a healthier online environment.

We’ve seen the consequences of allowing unchecked access to content, and we’re taking steps to ensure that creators are compensated fairly for their work.

As we look to the future, it’s essential to recognize the importance of building a better internet—one that values knowledge and creativity over sensationalism and outrage. That’s the vision we’re working toward.