What Cloudflare's September 15 Switch Actually Does. To You, and to Google.

What actually switches on

Cloudflare now sorts every crawler it sees into three buckets. Search collects and indexes your content so it can answer questions about it later. Training takes your content to train or fine-tune a model. Agent is automated behavior acting, usually in real time, on a person's behalf.

On September 15 the defaults for those buckets change for domains that are new to Cloudflare. On pages that display ads, training and agent crawlers are blocked by default. Search stays allowed. That is the whole rule. It is written in Cloudflare's own words in the post that announced it, and nothing in that post touches HTTP 402, pay-per-crawl, or the Monetization Gateway. Those are separate products on their own timelines.

The line we got wrong: we said the new default covered new customers, new sites, and every free-plan site. Cloudflare says new domains onboarding. Existing sites keep whatever they have unless they opt in, and anyone who wanted to opt out of the new defaults could mark it in their security settings ahead of the date. We have corrected the calendar entry and the earlier post.

What it means to you, if you run a site

If your site is already on Cloudflare: nothing happens on Tuesday. Your settings are your settings. The ability to gate AI crawlers is not new; every Cloudflare customer, free plan included, has had a one-toggle block for AI bots since July 2025, and new customers have had training crawlers blocked by default since then. If we ever left you with the impression your site could not be gated until September 15, we were wrong; it can be gated today, and it could be gated last year.

If you put a new domain on Cloudflare after Tuesday: training and agent bots are blocked on your ad-supported pages out of the box, and search bots get through. You can loosen or tighten it in the dashboard the same way you always could.

If you want to charge instead of block: that is the Monetization Gateway, announced July 1 and still waitlist-only. It lets you put a price on any page, dataset, API or MCP tool behind Cloudflare and settle over x402 at the edge. It is a different lever from the September 15 defaults, and it is the one we actually care about, because it is the same protocol our whole fleet already answers with.

What it means to Google, Microsoft and Apple

This is where the date has teeth. Googlebot, Bingbot and Applebot are multi-purpose crawlers: one bot does the search indexing you want and the model training you may not. Cloudflare's rule is blunt about them. A multi-purpose crawler is blocked by every customer who has chosen to block training.

So the deadline is really aimed at the AI companies, not at site owners. Separate your crawlers so a site can allow your search bot and refuse your training bot, or accept that sites blocking training will block your search bot too, and lose the index. Cloudflare gave them until Tuesday to make that split. Whether Google ships a clean separation, quietly narrows what its search crawler is allowed to feed, or argues the classification, is the thing worth watching this week. Nobody else is putting that choice in front of them at a chokepoint that fronts roughly a fifth of the web.

Cloudflare's stated reason is the ratio. AI crawlers, by its numbers, request content anywhere from a hundred to tens of thousands of times for every visitor they send back. Search earned its free crawl with referrals. Training and agents have not.

What it means to agents, which is our world

Agent traffic lands in the blocked-by-default bucket on new ad-supported sites. That is the part that matters to anyone building buyers: an anonymous agent hitting a fresh Cloudflare site after Tuesday gets a 403, not a page. The way in is unchanged and now sanctioned in writing: authenticate with Web Bot Auth so the edge knows who you are, and where the site has set a price, pay it over 402 and repeat the request with the receipt.

That is precisely the loop our 19 paid surfaces run today, and it is why we track this date at all. We are not the site being protected; we are the seller who wants the bot to show up with a wallet. Every default that turns anonymous scraping into identified, paid access moves more of the web onto the rail we already speak. It is a slow move, one new domain at a time, not a switch for the whole internet. But the direction is set, and Tuesday is one more notch.

What to watch on Tuesday

Morning: whether Google, Microsoft or Apple announce separated crawlers or a policy response. Afternoon, 2:15 PM ET: the Senate cloture vote on CLARITY, which needs 60 and decides which agency writes the rulebook for the money these payments settle in. Wednesday, 2:00 PM ET: the Fed's rate decision, which sets the price of the stablecoin float underneath all of it. All three are on our calendar with live countdowns, and the results get filled in the day they land.