Pay-per-Crawl for Cloudflare Workers · v0.1.2 · free, open sourceCharge AI crawlers with a Cloudflare Worker in front of any site
A Worker on your zone, a custom domain, or workers.dev in front of any origin. Deploys in five minutes, no change to the site behind it.
Works with: Cloudflare Workers, any origin, Cloudflare Pages
Install
npm install @forgemeshlabs/pay-per-crawl-worker
Add the gate
export { default } from "@forgemeshlabs/pay-per-crawl-worker";name = "pay-per-crawl"
main = "src/worker.js"
routes = [{ pattern = "example.com/*", zone_name = "example.com" }]
[vars]
PAY_TO = "0xYourWalletOnBase"
PRICE_USD = "0.01"
MODE = "monitor"
# npx wrangler secret put MPP_SECRETTurn it on
- 1Turn Cloudflare's own bot blocking off for the zone: "Block AI bots", AI Crawl Control and "Pay per crawl". Otherwise crawlers are stopped before they reach the Worker.
- 2Install the package, add the one-line worker file and the wrangler.toml, set PAY_TO and your route.
- 3npx wrangler secret put MPP_SECRET, then npx wrangler deploy.
- 4Leave MODE at monitor for a few days and read the logs. Then set MODE = "gate" and redeploy.
Settings
| PAY_TO | Your 0x wallet on Base. Required to charge. |
|---|---|
| PRICE_USD | Price per page fetch in USDC. Default 0.01, minimum 0.005. |
| MODE | monitor (default) logs what bots would have paid; gate answers 402. |
| FACILITATOR_URL | Any x402 facilitator that settles on Base. Default PayAI, no API key. |
| CLASS_POLICY | JSON per class: search, social, monitor, seo, ai, automation, bot → free, paid or block. |
| BOT_OVERRIDES | JSON per crawler name, wins over the class: {"GPTBot":"paid","Bytespider":"block"}. |
| ALLOW_PATHS | Comma list of path prefixes that are never gated. |
| TRUST_HEADER | Header carrying the real client IP behind your proxy. Right-most entry is used. |
| MPP_SECRET | Long random string signing MPP challenges. Set it so challenges survive restarts. |
| PRO_KEY | Optional Pay-per-Crawl Pro license for the hosted dashboard. |
| ORIGIN_URL | Rewrite requests to this origin (workers.dev setups). Otherwise the Worker fetches the zone origin. |
| PPC_KV | Optional KV binding: shared replay lock across isolates. Bind it for production gating. |
| PPC_ANALYTICS | Optional Workers Analytics Engine binding: one data point per bot request. |
Things to know
- TRUST_HEADER defaults to cf-connecting-ip, which Cloudflare sets. Leave it.
- Without a KV binding the replay lock is per-isolate memory, fine for monitor mode and low traffic. Bind KV before gating real traffic.
- The Worker appends discovery pointers to your robots.txt and serves the three discovery files itself.
- Humans never see a 402. Search engines are verified by reverse DNS; a request that only claims to be Googlebot is charged.
- Start in monitor mode. It logs every bot with what it would have paid and never blocks. Switch to gate when the numbers make sense.
Proof it settles
Real settlements on Base mainnet through the PayAI facilitator, $0.01 each, October 2026:
- Same core as the Next.js edition; mainnet hashes on that guide