---
title: "Robots.txt in Google: what works and what doesn’t (AI, emoji)"
description: "Google explains what robots.txt supports: four rules, encoding characters in paths, AI crawler tokens and the limits of the file. What it means for SEO."
date: 2026-10-19
category: SEO
tags: ["robots.txt", "Google", "AI crawlers", "Technical SEO", "crawling"]
url: https://uper.pl/en/blog/google-robots-txt-what-you-can-put-in-it/
---

# What can you put in robots.txt? — rules for AI crawlers, emoji and URLs with non-ASCII characters

**A robots.txt file understands only four fields: `user-agent`, `allow`, `disallow` and `sitemap`. Google's parser silently skips every other line.** In the second episode of the "Non-Canonical Cabinet" series, Gary Illyes and Cherry Prommawin of Google went through what actually works in this file and what only looks like it does. For me, three things from the conversation matter most: emoji and non-ASCII characters in paths are best written percent-encoded, you block an AI crawler with its real token, and robots.txt has no say in what a company does with content it has already fetched.

Barry Schwartz covered the episode on Search Engine Roundtable (October 21, 2026). Below I set what was said in the conversation against Google's documentation and the RFC 9309 standard.

## What Google said about robots.txt

Illyes described robots.txt as a plain text file in the site's root directory, where the owner tells automated clients where they may go. The rules are read by search engine crawlers, bots collecting data to train AI models and any other robots that follow the rules at all. Googlebot (and every other "well-behaved" robot) checks this file before fetching anything else.

The basis is the **RFC 9309** standard, which Google says it follows. The parser reads the file line by line and looks for four fields: `user-agent`, `allow`, `disallow`, `sitemap`. A line it doesn't understand is discarded, and it moves on to the next one.

Schwartz summed the episode up in three sentences: Googlebot and other compliant crawlers use robots.txt to limit crawling, the file should follow RFC 9309, and emoji and non-ASCII characters in rule paths are worth percent-encoding so that every crawler understands them.

## Four rules and silently discarded lines

If you write something like "don't crawl the secret page" in the file, the parser won't report an error. It will treat the line as invalid and skip it, and the page will remain accessible to the robot. As Illyes put it, a rule that makes sense to a human doesn't necessarily work for a machine.

Google's documentation adds a few details I find useful in audits:

- field names are case-insensitive, paths are not,
- an `allow` or `disallow` rule without a path is ignored,
- fields other than `allow`, `disallow` and `user-agent` (e.g. `sitemap`) don't end a group of rules, so a `sitemap` line placed between two `user-agent` lines doesn't separate them,
- Google doesn't support the `crawl-delay` field,
- the file must be UTF-8 text, and Google processes up to 500 KiB of it and ignores anything beyond that.

Comments are allowed: the crawler ignores everything from the `#` character to the end of the line, so ASCII art in comments won't break anything (although Illyes called it a questionable idea). Prommawin mentioned a robots.txt file that was 97% decoration and only 3% directives. The parser will cope with that, but the robot downloads unnecessary bytes, and decorations placed next to important directives are easy to damage while editing, so it's better to keep them away from each other.

I cover the syntax from scratch in my [complete guide to robots.txt](/en/blog/robots-txt/).

## Emoji and non-ASCII characters: tokens versus paths

The emoji thread started as a joke (someone asked whether you can write a `user-agent` with a chicken in it), but it leads to a concrete rule, separate for tokens and for paths.

### Crawler token: simple characters only

Emoji and letters outside the basic set won't work in the `user-agent` field. According to Illyes, a token can consist only of letters and hyphens. RFC 9309 is slightly broader here and allows the letters a-z and A-Z, the underscore and the hyphen. The conclusion doesn't change: an emoji is a multi-byte UTF-8 character that doesn't fit that set, so a rule using one won't work.

### Paths: use percent-encoding

In `allow` and `disallow` paths, non-ASCII characters are needed, because real URLs contain them: accented letters, German umlauts, Cyrillic, Chinese characters, emoji.

Google handles raw UTF-8. According to the documentation, it matches rules against percent-encoded URLs and converts raw characters in a rule to that form itself. Prommawin pointed out, however, that other or older crawlers may trip over this: if they can't process a raw character, they will skip the whole `disallow` line and go exactly where you wanted to stop them. That's why it's best to write the rule the way the server sees it, in encoded form.

An example for a `/münchen/` directory:

```txt
User-agent: *
# Instead of: Disallow: /münchen/
Disallow: /m%C3%BCnchen/
```

You would write the 🐔 emoji in a path the same way, as `%F0%9F%90%94`. RFC 9309 requires non-ASCII characters to be percent-encoded before matching anyway.

Accented characters in slugs are not that common, but they do show up in older online stores and classifieds sites. If you block such sections, check how the rule is written and test it against real URLs, because rules for URLs with special characters are usually something nobody has ever tested.

## AI crawlers: a token, not a decorative rule

Illyes spent the most time on generative AI. He stressed that under RFC 9309, robots.txt is an **access preference protocol**: it tells a bot whether it may come in, and that is where its role ends. What a system does with the data it has fetched is not something the file governs.

To block a specific AI crawler, you need to put its recognised name in `user-agent`. A rule like "don't use my content for training" is garbage to the parser and gets dropped during parsing. Tokens change and new ones keep appearing, so check the current list in the documentation of the operator of a given bot.

A good example from Google's official documentation is the `Google-Extended` token. Publishers use it to control whether their content is used to train future Gemini models and for grounding (feeding the model content from the search index at prompt time). According to Google, this token doesn't have a separate user agent in HTTP requests and works only in robots.txt, while crawling itself still happens under Google's existing names. The documentation also notes that it doesn't affect a page's inclusion in Search or its ranking.

It's also worth separating crawl control from content-use policy. Robots.txt can block access to a page, but it won't stop a company from training models on content it already has. For that you need licence terms, contracts or regulation, which are tools outside this file.

I discuss the split between search engine bots and AI bots in more detail in [AI crawlers vs search crawlers](/en/blog/ai-crawlers-vs-search-crawlers/), and which bots you'd better not block if you care about citations in ChatGPT or Perplexity in my article on [making your website visible to LLMs](/en/blog/make-website-visible-to-llms/).

## Robots.txt is not security

Schwartz closed by reminding readers that robots.txt is a request to crawlers willing to follow it, not a security mechanism. It won't stop an unauthorised client or a bot that ignores the file, and the standard itself states plainly that these rules are not a form of authorisation. Listing the path to a "secret" directory actually works the other way round, because it reveals its existence to anyone who opens `/robots.txt`.

Content is protected with authentication or with rules on the server or in a WAF. A page is kept out of the index with `noindex` or a response code, and `disallow` alone won't do it: Google says in its documentation that a blocked URL can still appear in results (without a description), because it can be indexed without its content being crawled. I write more about these differences in the [robots.txt guide](/en/blog/robots-txt/).

## What to do with this

The episode leaves me with a short checklist:

1. **Validate the syntax.** A readable rule doesn't necessarily work, so test it against specific URLs, for example in the robots.txt report in [Google Search Console](/en/blog/google-search-console/).
2. **Remove lines the parser doesn't recognise.** Home-made "directives", unsupported fields and notes do nothing.
3. **Percent-encode paths with non-ASCII characters and emoji** if the rules are meant to work outside Google too.
4. **Use the real tokens of AI crawlers** and check them in the operator's documentation.
5. **Don't treat robots.txt as protection** or as a way to withdraw data a company has already fetched.
6. **Watch the file's size, encoding and availability:** UTF-8, under 500 KiB and a 200 response. I explained why a robots.txt that returns 503 stops crawling in my article on [taking a site offline to patch PHP](/en/blog/google-503-take-site-offline-php-patch/).

## Limitations and unknowns

The episode covers existing rules, and Google didn't announce any change to how it handles robots.txt. One caveat runs through the whole conversation: what Google supports, Bing, DuckDuckGo or the AI crawlers of various companies don't necessarily support. Each operator can read the file its own way, and some bots don't respect it at all. The conversation also didn't include a list of AI crawler tokens, so you have to check those yourself.

## Summary

Robots.txt has a simple syntax, which makes it easy to make a mistake nobody notices: it understands four fields, discards the rest without warning, accepts only simple characters in tokens, and paths are safest when percent-encoded. You stop an AI crawler with the right token, keeping in mind that the file governs access, not later use of the content. If your file contains something "for decoration" or "just in case", remove it and check which rules actually work.

<FaqBlog
  questions={[
    {
      question: 'Which rules does robots.txt support in Google?',
      answer: 'Google supports four fields: <strong>user-agent</strong>, <strong>allow</strong>, <strong>disallow</strong> and <strong>sitemap</strong>. It ignores other lines and doesn’t support the crawl-delay field. Field names are case-insensitive, paths are not.'
    },
    {
      question: 'Can I use emoji and non-ASCII characters in robots.txt?',
      answer: 'In rule paths, yes, but it’s safest to write them percent-encoded (e.g. <code>/münchen/</code> as <code>/m%C3%BCnchen/</code>). Google handles raw UTF-8, but other or older crawlers may skip such a line. In the user-agent token, emoji and characters outside the basic set won’t work at all.'
    },
    {
      question: 'Will robots.txt stop AI models from being trained on my content?',
      answer: 'Only partly. The file tells a crawler whether it may access a page, but it doesn’t decide what a company does with data it fetched earlier. To block a specific AI bot, put its correct token in the user-agent field and check it in the operator’s documentation.'
    },
    {
      question: 'Does robots.txt protect private pages from access?',
      answer: 'No. The file is a request to crawlers that follow it, and any path listed in it is publicly visible. Content is protected with authentication or server-side rules.'
    }
  ]}
  heading="Frequently Asked Questions"
  id="faq"
/>

## Sources

1. **Google Explains robots.txt Rules For AI Crawlers, Emojis, International URLs & More** (Search Engine Roundtable)
[https://www.seroundtable.com/google-video-robots-txt-rules-42355.html](https://www.seroundtable.com/google-video-robots-txt-rules-42355.html)
2. **Robots.txt Specifications** (Google Search Central)
[https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)
3. **RFC 9309: Robots Exclusion Protocol**
[https://www.rfc-editor.org/rfc/rfc9309](https://www.rfc-editor.org/rfc/rfc9309)
4. **Google's common crawlers** (Google Search Central)
[https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)