← All posts

robots.txt: Allow rules for GPTBot, ClaudeBot, and PerplexityBot

Practical guide to robots.txt for AI crawlers: GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google vs Yandex split, Allow/Disallow, post-deploy checklist, and AISO linkage. Q2 2026.

Tags
aeoAIaisogeoseo
robots.txt: Allow rules for GPTBot, ClaudeBot, and PerplexityBot
Contents

In brief

robots.txt is a root-level text file that tells crawlers which URLs they may fetch. For AISO (AI Search Optimization) we add explicit Allow rules for GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot when we want LLM systems to read public HTML. This is not a GEO growth hack and does not replace strong content: robots opens the door; Information Gain and server-rendered HTML decide whether you get cited. On stuzhuk.page the file is built dynamically from code on each request — verify it with curl after every deploy, not only in the browser.

What robots.txt is — and what it is not

Robots.txt follows the robots exclusion standard. Browsers ignore it; bots read it before or during crawling. Limits worth remembering:

  • Not security. A disallowed URL is still reachable by direct link. Protect secrets with auth, not robots.
  • Not a direct index control. Search engines may learn about URLs from sitemaps or inbound links even when crawling is disallowed. Use noindex in meta or headers to keep pages out of SERPs.
  • A convention, not enforcement. Legitimate bots usually comply; malicious ones do not.

For engineering teams, robots.txt is crawler access policy versioned in the repo and served from the production origin.

AI crawlers you will see in 2026

Each vendor uses its own User-agent. Mixing up training vs search bots is a common reason teams “allowed GPTBot” but still see no citations.

User-agent Vendor Typical purpose
GPTBot OpenAI Training / indexing for ChatGPT and SearchGPT
OAI-SearchBot OpenAI Search crawler (separate from GPTBot)
ClaudeBot Anthropic Claude products
PerplexityBot Perplexity Answers with source citations
Google-Extended Google Optional signal for Gemini (distinct from Googlebot)
Bytespider ByteDance Often blocked by policy in some regions

On stuzhuk.page we explicitly allow GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot in search-indexing.ts. Extend the list when vendors publish new documented agent names.

robots.txt vs meta robots vs X-Robots-Tag

Mechanism Where Effect
robots.txt /robots.txt Allow or deny crawling of URLs
<meta name="robots"> HTML noindex, nofollow for a single page
X-Robots-Tag HTTP header Same, for APIs, PDFs, non-HTML assets

Typical public blog setup: Allow AI bots and User-agent: * / Allow: / in robots; use noindex on /admin, drafts, and previews even if the root is open.

How to read Allow and Disallow

Rules apply top to bottom per User-agent. Specific agents override wildcard *.

Minimal “open AI search” policy:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://stuzhuk.page/sitemap.xml

Optional Disallow for internal paths:

User-agent: *
Disallow: /admin/
Disallow: /api/

Do not Allow URLs that return 401 or empty shells — policy must match what bots actually receive.

Google vs Yandex on two domains

One repo, two indexing profiles:

  • stuzhuklab.ru — Yandex open; Googlebot gets Disallow: / to avoid duplicating the international site.
  • stuzhuk.page — Google and Bing open; Yandex disallowed.

AI crawlers still get Allow on both deployments — deliberate AISO policy: public content stays reachable for citation regardless of classic SEO split. See the SEO / AEO / GEO / AISO pillar for the full layer model.

Staging: PUBLIC_BLOCK_SEARCH_INDEXING=true returns Disallow: / for everyone and drops the sitemap — verify before client demos.

What to allow vs block for AI bots

Allow makes sense for:

  • blog posts with full text in server HTML;
  • services, portfolio, FAQ pages;
  • /llms.txt and /sitemap.xml as entry points.

Block or noindex:

  • /admin, account areas, pre-checkout carts;
  • UTM-only duplicates (fix with canonical, not robots alone);
  • drafts (draft: true) — excluded from sitemap, must not appear in public nav;
  • staging hosts — full Disallow: /.

Robots does not replace GEO: after Allow, bots still prefer pages with Information Gain over rewritten TOP-10 filler.

How we serve robots.txt in code

The file is not a static asset in public/. Route robots.txt.ts calls buildRobotsTxt() with the production origin from env so VPS builds never bake in localhost. After ./update.sh intl or ./update.sh ru:

curl -s https://stuzhuk.page/robots.txt
curl -sI https://stuzhuk.page/robots.txt | head -5

Expect: 200, text/plain, a Sitemap: line on the same host you use in the browser, and User-agent: GPTBot / Allow: / blocks.

Confirm articles render without JavaScript:

curl -s https://stuzhuk.page/blog/robots-txt-ai-crawlers-gptbot-claude | head -c 2000

If H1 and body paragraphs are missing, AISO fails regardless of Allow rules.

Linkage to llms.txt, sitemap, and the AISO audit

Robots is the first gate after SEO basics. Next in the stack:

  1. Sitemap with lastmod — discovery for all crawlers.
  2. llms.txt — a compact map for agents (not a sitemap replacement).
  3. JSON-LD Article / FAQPage — extraction hints.
  4. Related posts and tag hubs — internal cluster linking.

That cycle moved AI Readiness from 52 to 84 on stuzhuklab.ru — see the AISO audit write-up. robots.txt was one checklist item, not the whole story.

Common mistakes

  1. Staging robots copied to production — entire site Disallow: /; search consoles show excluded or low-value URLs.
  2. Allow in robots but SPA without SSR — bot gets an empty shell.
  3. Sitemap on the wrong domain — robots points at another origin than the live host.
  4. Blocking all AI bots “for security” — opts out of GEO, does not protect PII.
  5. Skipping post-deploy verification — VPS env differs from local .env.

15-minute post-deploy checklist

  1. curl -s https://<your-domain>/robots.txt — Allow for target AI agents and a Sitemap line.
  2. Confirm Yandex is not globally disallowed on the RU site (if Yandex is a goal).
  3. curl two or three blog URLs — full article text in HTML.
  4. Check canonical and hreflang on one article (no /ru/blog vs /blog duplicates on the RU domain).
  5. In Search Console / Yandex Webmaster — robots analysis and recrawl of key URLs.
  6. Cross-check the pillar guide for remaining AISO items.

FAQ

Is Allow for GPTBot enough to appear in ChatGPT?

No. Allow grants fetch permission. Citations depend on quality, novelty, and whether the page enters the provider’s index/corpus. Monitor separately via GEO Share of Voice.

Should we allow Google-Extended separately?

If Googlebot is already open on the international site, Gemini policy may overlap. On RU-only builds Googlebot may be intentionally blocked; then Google-Extended is a separate product decision.

Does robots.txt control YandexGPT?

Yandex uses its own agents (YandexBot, etc.). On stuzhuklab.ru they are covered via User-agent: * Allow, not via GPTBot lines. OpenAI and Anthropic agents need their own entries.

Can we disallow only the blog but keep the homepage?

Yes: Disallow: /blog/ for a given agent — but then AEO/GEO for articles will not apply. That is a conscious trade-off.

How is robots different from llms.txt?

Robots sets crawl rules. llms.txt is a structure hint for LLM agents without enforcement. Both help; neither guarantees citation.

How often should we review policy?

On domain changes, new admin/API routes, new documented AI agents, or SSR → SPA migrations. Minimum: a quick curl check after every production deploy.

Takeaway

AI crawler rules in robots.txt are product policy, not a one-off SEO tweak. Encode Allow for the agents you want, close operational zones, verify production after deploy, and do not expect GEO from an open door alone. For a full-stack audit see SEO / AEO / GEO or the 52→84 AISO case study.

Related: llms.txt · AEO structure · Information Gain audit