What is Grok?
xAI · user fetch · class ai
Fetches on behalf of Grok users; no published crawler documentation.
Verdict: Usually let it in; it is a reader with an assistant.
On the mugshot wall: No published crawler documentation
Profile
Compiled by Crawler Atlas from the operator's published documentation. Not yet declared by the operator. JSON
| Operator | xAI |
|---|---|
| Purpose | Fetchers for an AI user. One request because a person asked an assistant about your page. Not bulk crawling; some ignore robots.txt because a human triggered them. |
| User-agent contains | xai-grok · grokbot |
| robots.txt | No documented token. Block at the edge or by user-agent match. |
| How to verify | No published verification method. Treat the name as a claim, not proof. |
| Should you block it? | Usually let it in; it is a reader with an assistant. |
| Does it pay? | No. It does not carry a wallet and cannot answer a 402. A site that charges simply serves it nothing. |
| Docs | None published |
Observed on the network
Anonymous daily counts from sites running Pay-per-Crawl that opted in, plus the ForgeMesh fleet. Last 7 days. Small sample: 1 sites reporting.
No sightings reported yet. Grok appears here the day an opted-in site sees it.
Do something about it
- See it in your own logs, verify it, and charge, allow or block it by name with the free Pay-per-Crawl plugin (WordPress, Python, JavaScript, Go, PHP).
- Run this crawler? Publish a crawler-profile.json on your domain and this page shows your declared profile with a verified badge.
- Machine-readable twin: /bots/grok/profile.json (free in a browser). Machines: POST
{"slug":"grok"}to x402.forgemesh.io/bot-profiles, $0.005 per call over x402.
Also in fetchers for an ai user
Crawler Atlas is maintained by ForgeMesh. Names and marks belong to their operators and are used to identify their crawlers.