Skip to main content
ForgeMeshBot mugshots · 20 on the wall

The crawlers to think twice about

Not a blocklist. A wall of the bots that give you a reason: nobody identifies them, they train models with no opt-out, they ignore robots.txt, their name gets impersonated, or most sites that see them already block them. Each card says why, so you decide. Reasons from network observations update daily and are labelled by sample size.

  • Bytespider

    ByteDance · AI training

    • Trains models with no published policy or opt-out
    • No published crawler documentation
    • No published way to verify a request is really this crawler

    robots.txt: User-agent: Bytespider

  • cohere-ai

    Cohere · AI training

    • Trains models with no published policy or opt-out
    • No published crawler documentation
    • No published way to verify a request is really this crawler

    robots.txt: User-agent: cohere-ai

  • DeepSeekBot

    DeepSeek · AI training

    • Trains models with no published policy or opt-out
    • No published crawler documentation
    • No published way to verify a request is really this crawler

    robots.txt: User-agent: DeepSeekBot

  • AI2Bot

    Allen Institute for AI · AI training

    • No published way to verify a request is really this crawler

    robots.txt: User-agent: AI2Bot

  • curl

    Unknown (bare tool) · script

    • Unidentified: no operator, no documentation
  • DuckAssistBot

    DuckDuckGo · AI search

    • No published way to verify a request is really this crawler

    robots.txt: User-agent: DuckAssistBot

  • Go-http-client

    Unknown (bare library) · script

    • Unidentified: no operator, no documentation
  • Grok

    xAI · user fetch

    • No published crawler documentation
  • Headless browser

    Unknown (automation) · script

    • Unidentified: no operator, no documentation
  • HTTPie

    Unknown operator · script

    • Unidentified: no operator, no documentation
  • iAskBot

    iAsk · AI search

    • No published way to verify a request is really this crawler

    robots.txt: User-agent: iAskBot

  • Java HTTP

    Unknown operator · script

    • Unidentified: no operator, no documentation
  • libwww

    Unknown operator · script

    • Unidentified: no operator, no documentation
  • Meta-ExternalAgent

    Meta · AI training

    • No published way to verify a request is really this crawler

    robots.txt: User-agent: Meta-ExternalAgent

  • Node fetch

    Unknown operator · script

    • Unidentified: no operator, no documentation
  • Perplexity-User

    Perplexity · user fetch

    • Documented as ignoring robots.txt

    robots.txt: User-agent: Perplexity-User

  • PostmanRuntime

    Unknown operator · script

    • Unidentified: no operator, no documentation
  • python-requests

    Unknown (bare library) · script

    • Unidentified: no operator, no documentation
  • wget

    Unknown operator · script

    • Unidentified: no operator, no documentation
  • YouBot

    You.com · AI search

    • No published way to verify a request is really this crawler

    robots.txt: User-agent: YouBot

Copy the wall into robots.txt

Only the bots on this wall that honour a robots.txt token. The ones that ignore it, or have no token, need an edge rule or a paywall instead.

User-agent: Bytespider
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: DeepSeekBot
Disallow: /

User-agent: AI2Bot
Disallow: /

User-agent: DuckAssistBot
Disallow: /

User-agent: iAskBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: YouBot
Disallow: /

Or charge them instead of blocking: the free Pay-per-Crawl plugin answers every bot on this wall with a price.