Crawler Atlas · free · ever-growingEvery bot that reads the web, explained
Most of a website's traffic is not people. This is the profile of each thing that is: who runs it, what it does with your pages, its robots.txt line, how to verify it, whether it can pay, and what sites across the network actually observe. Same card for every bot, so you learn the shape once.
- 85
- bots profiled
- 1
- sites reporting sightings
- 6
- new in the last 14 days
Mugshot wall → the ones to think twice about
Network numbers come from sites running Pay-per-Crawl that opted in to share anonymous daily counts, plus the ForgeMesh fleet. Small samples are labelled as such.
85 / 85
Run a crawler? Publish your own profile.
Put a crawler-profile.json at /.well-known/ on your domain and the Atlas shows your words next to what sites observe, with a verified badge. No signup; the domain is the proof. Spec and generator: github.com/forgemeshlabs. Missing a bot? Email [email protected] (free) or submit over the API ($0.005, x402).