robots.txt in 2026: directives, mistakes, and AI crawlers

A badly written robots.txt can quietly slow down how Google indexes your site: one misplaced disallow line is enough. In 2026 Google clarified which directives it reads and which ones it ignores, and AI crawlers changed the game. This guide shows you how to write yours without mistakes.

Which directives Google reads in your robots.txt (and which it ignores)

Before touching anything, remember the real purpose of the file. According to Google's official documentation, robots.txt exists to control crawling and avoid server overload, and it is not a mechanism for keeping a web page out of search results. If you need the basics first, read what the robots.txt file is and why it matters for SEO. Here we go straight to the rules that actually matter in 2026.

Supported directives

Google only recognizes a handful of fields. These are the ones you can use safely:

user-agent: required. It defines which crawler the following group of rules applies to, and it can appear more than once.
disallow and allow: block or allow paths. When several rules match, the most specific one wins (more on that below).
sitemap: not tied to any user-agent, and you can declare several sitemaps with no limit. The field name is case-insensitive, but the value is case-sensitive. Crawlers ignore this line when evaluating blocking rules.

Directives Google does NOT support

Google ignores several fields that still appear in old guides: crawl-delay (in February 2026 Google officially clarified it does not support it), host, request-rate, visit-time, and above all noindex and nofollow, which only work in meta robots tags or HTTP headers.

Review your current file and remove those dead lines. They do not cause technical harm, but they confuse anyone reading the file and give a false sense of control.

How Google decides whether a URL is blocked

The rule is called "most-specific match" and it was standardized in RFC 9309 in 2022. If a URL matches several rules, the longest match wins; allow and disallow have the same priority, so what decides is which rule matches better, not the order they appear in.

There are also two wildcards: * matches any sequence of characters and $ marks the end of the URL. And one detail people forget: robots.txt paths are case-sensitive.

User-agent: *
Disallow: /admin
Allow: /admin/public

# /admin/public/* stays allowed because the Allow rule matches better

Common robots.txt mistakes that cost indexing

Most problems do not come from an exotic directive but from basic misunderstandings. These are the ones that cost the most indexing:

Blocking CSS and JavaScript: it prevents Google from rendering the page. The screenshot in URL Inspection will show you a broken version. Resources are not blocked with robots.txt but with noindex or headers.
Putting noindex inside robots.txt: it does not work; Google ignores it. To keep a page out of the index, use noindex in a meta robots tag or an HTTP header, as explained in Google's guide to blocking indexing.
Using robots.txt to "hide" private pages: besides not working, it consumes crawl budget. For sensitive content, use server-side authentication.
Blocking sections you actually want indexed: a misplaced Disallow: / or an overly broad rule with * can stop the whole site.
Mixing up user-agents: wanting to block Google-Extended (AI) and accidentally blocking Googlebot (search). They are different user-agents, as you will see in the AI crawlers section.
Forgetting the Sitemap: line: pointing to your XML sitemap complements submitting it in Search Console. If you need a refresher, check the guide on how to set up Google XML Sitemaps.

How to spot them

The typical signal shows up in the Search Console coverage report: "Blocked by robots.txt". If you see that status on URLs that should be indexed, the file is suspect number one.

robots.txt vs noindex: when to use each

The difference comes down to two words: crawling and indexing. robots.txt controls crawling, meaning what consumes resources on your server. noindex controls indexing, meaning what appears in Google.

Practical rule: if a page should not appear in Google, use noindex (the URL can be crawled and discarded). If a section is huge and adds no value, like ecommerce filters, a disallow makes sense there.

One important warning: a page blocked by robots.txt can still show up in results if it has external links, because Google can index it without crawling its content. That is why you cannot "hide" pages with robots.txt; for that you need authentication.

AI crawlers in 2026: what to block and what to allow

AI providers increasingly separate training crawlers from search crawlers. According to tracking from May 2026, around 30% of top sites block training and allow search, to stay eligible for citations in AI answers without giving content away for model training.

Training bots (block if you don't want your content used)

GPTBot (OpenAI)
Google-Extended (training for Gemini and Vertex; it does not affect your Google Search rankings)
ClaudeBot and anthropic-ai (Anthropic)
CCBot (Common Crawl)
Bytespider (ByteDance)
PerplexityBot (depending on each site's policy)
Applebot-Extended (Apple)

Search and retrieval bots (usually worth allowing)

OAI-SearchBot (ChatGPT search)
Claude-SearchBot (Claude search)
PerplexityBot (Perplexity)

Note: Google-Extended is not a search bot. Allowing retrieval bots makes you eligible to be cited in AI answers without handing content to training, a defensible policy according to the 2026 AI user-agent landscape.

The 2026 nuance: not everyone respects robots.txt

RFC 9309 is a voluntary compliance standard. Agents like Google-Agent and some automated fetchers ignore robots.txt. If you really need to restrict them, do it server-side: authentication, rate limiting, or IP/ASN blocking.

Minimal template to block training without touching search

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Bytespider
Disallow: /

Notice what is not there: no OAI-SearchBot, no PerplexityBot, no Claude-SearchBot. Leave them alone and you stay eligible to appear in AI answers.

How to test your robots.txt in 2026

The Search Console robots.txt Tester was retired in December 2023. Today you test URL by URL with the URL Inspection tool, which shows whether crawling is blocked, which rule blocks it, and a rendered screenshot of the page.

A quick manual check is to fetch the file directly: curl -s https://yourdomain.com/robots.txt (or open it in a browser) to confirm it responds with 200 and that the visible file is the correct one.

Testing checklist:

Does the file respond with HTTP 200?
Does the User-agent: * group avoid blocking public sections?
Are CSS and JavaScript accessible to Googlebot?
Does the Sitemap: line point to the correct XML sitemap URL?
Are the AI user-agents separated from Googlebot?

Template: a sample robots.txt for 2026

This file allows Googlebot and AI search bots, blocks training from the main providers, and keeps real private sections out. It is a generic template: every site adjusts the paths according to its structure.

User-agent: *
Disallow: /admin
Disallow: /api/private

User-agent: Googlebot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://yourdomain.com/sitemap.xml

Keep Googlebot separate from the training bots. If you ever decide to allow AI training, just delete those groups.

FAQ about robots.txt

Does robots.txt affect rankings?

Indirectly. Written badly, it can prevent pages you want in Google from being indexed; written well, it only manages crawling and does not improve or hurt rankings by itself.

Should I block Google-Extended?

It depends on whether you want your content used to train Gemini and Vertex. Blocking it does not affect your Google Search position, because it is a training user-agent, not a search one.

Where should the file live?

At the root of the domain: https://yourdomain.com/robots.txt. There is one file per domain, subdomain and protocol, so each variant needs its own.

If you want to dig deeper into why crawling matters, check the guide on crawl budget and how to optimize it.