--- title: "4xx errors and the Google index: temporary URL protection" description: "Gary Illyes: when an important URL returns a 4xx error, Google may keep it in the index for a while, minus snippet and favicon. What it means for technical SEO." date: 2026-11-07 updated: 2026-11-10 category: SEO tags: ["Google", "Indexing", "HTTP status codes", "4xx errors", "Technical SEO"] url: https://uper.pl/en/blog/google-protects-important-urls-4xx/ --- # Google can briefly protect an important URL from deindexing What Gary Illyes said about 4xx errors after the VLC favicon went missing When an important URL starts returning a **4xx error**, Google doesn't have to drop it from the index straight away. Gary Illyes wrote that Search can "temporarily protect the URL from falling out of the index right away", so the page keeps showing up in results, just without a snippet or a favicon. A URL that keeps returning 4xx will still leave the index eventually, because that rule hasn't changed. I compared what Illyes said with Google's documentation and tested the VLC server that started the whole thing myself. ## Where this came from: VLC without a favicon In early November 2026 the VideoLAN team, who develop the VLC media player, asked for help on LinkedIn: their favicon had disappeared from Google's results, and they said they hadn't changed anything. The VLC home page was still ranking, second, below the download page. Commenters blamed things like a favicon file that was too small. Illyes explained that the VLC server had been returning status code **418** ("I'm a teapot"), the joke code from the 1998 April Fools' specification (RFC 2324) that a real website should never send. Google treats it, in his words, like "most other 4xx errors: basically 'not found'". The page stayed in the index anyway, because VLC is, according to Illyes, "arguably one of the most important pages on the internet", so Search temporarily held off on removing it. Barry Schwartz of Search Engine Roundtable, who reported the exchange on 5 November, offers another explanation: the page may also have stayed because people search for it by name. Google hasn't confirmed that. ## What happens to a protected URL Google receives an error instead of content, so, as Illyes put it, "we don't know what's on the page anymore". With nothing fresh to show, the result keeps **only the link and the last good title** Google had, and the snippet and the favicon disappear. 1. The URL is indexed, with content, a title and a favicon. 2. The server starts answering the crawler with a 4xx code. 3. Google doesn't drop the important URL right away, but it has no fresh content. The result shows the link and the last known title. 4. Once the server returns 200 with content again, the data refreshes on the next crawls. If the error persists, the URL is removed, as the documentation describes. VLC stopped at step three. Illyes said the site was returning 200 OK with content again, so the result "should sort itself out soon". Schwartz added that his own site's favicon vanished from the results for a few days at the end of October and only came back after he changed his Cloudflare settings. His gut feeling is that it was related to the PHP exploit, which I covered in the article on [taking a site offline while patching PHP](/en/blog/google-503-take-site-offline-php-patch/), but he doesn't claim to know. ## Why 418: I tested the VLC server Illyes didn't say why the VLC server answered as a teapot. On 10 November 2026 I requested the videolan.org home page with different User-Agent headers: | User-Agent | Status code | |---|---| | Browser (Chrome) | 200 | | Googlebot | 200 | | GPTBot (OpenAI) | 418 | | PerplexityBot | 418 | The server still sends 418, but now only to selected **AI bots**, while Googlebot and a regular browser get 200. SaaS Link Building tested more bots earlier and got a similar result. My requests came from a non-Google IP address, so I don't know exactly what the real Googlebot received during the incident. My guess is that a rule meant to keep AI bots out caught Google for a while. Neither VideoLAN nor Google has confirmed this, but it is how AI bots are often blocked today: in a WAF, a CDN or the server configuration, not only in robots.txt. A robots.txt entry only asks a crawler to stay away, and at worst the crawler ignores it. A WAF rule answers with an error code, so one bad match is enough to serve Googlebot a 4xx. I described the difference between the two approaches in the article on [AI crawlers and search crawlers](/en/blog/ai-crawlers-vs-search-crawlers/). ## What Google's documentation says about 4xx codes Google's page on how HTTP status codes affect its crawlers makes no exceptions: - Google doesn't use content from URLs that return 4xx. If a URL was used before and now returns 4xx, Google stops using it "over time". The documentation gives no time frame. - In Google Search, URLs that return 4xx aren't indexed, and ones that are already indexed are removed. - All 4xx errors except 429 are treated the same: the crawler tells the next system that the content doesn't exist. Newly found 404 pages aren't processed, and crawl frequency gradually drops. - Google treats **429** (too many requests) as a server error, a sign of overload. Don't use 401 and 403 to limit crawl rate, because they have no effect on it. The page covers the 20 or so most common codes and explicitly leaves out "more exotic status codes, such as 418", so Illyes's comment fills that gap: 418 works like any other 4xx. Protection for important URLs isn't mentioned anywhere on it. For 5xx codes Google does document something similar: indexed URLs stay in the index for a while and are dropped if the error persists. For 4xx, the only source is that one comment from Illyes. ## Takeaways for technical SEO ### Don't count on the protection It covers URLs Google considers important, and nobody knows the criteria for that. An online shop, a blog or a local business is unlikely to qualify. VLC is one of the most popular sites on the web, so its case tells you little about what would happen to your URL. ### There is a window to fix it, but nobody knows how long An important page can start returning 4xx by accident after a WAF rule change or a deployment, and then speed matters. Some URLs may stay in the results for a while in a weaker form, but Google gave no numbers, so I don't assume any "safe" period. ### Rankings can hold while traffic drops A protected URL keeps its position, so a rank tracker won't show the problem. A result without a snippet or favicon is less inviting to click, so for important pages it is worth tracking CTR and clicks in Search Console alongside rankings. ### How to find out where the favicon went If an important page shows only a link and a title in the results, check which status code the crawler receives: 1. The [URL Inspection tool in Search Console](/en/blog/google-search-console/) and its live test show what Googlebot sees. 2. Server and CDN logs reveal codes you won't see in a browser, such as 418 or 403 served to only part of the traffic. 3. Review User-Agent matches in your WAF and bot protection rules. If you block AI bots, make sure the rule doesn't cover Googlebot or Googlebot-Image, which fetches the favicon. ## What we don't know Illyes wrote "temporarily" and "right away", but didn't say how long the protection lasts. It is also unclear how Google decides a URL is important, and whether the protection applies to every 4xx code, since the example involves 418 and Illyes spoke about 4xx in general. The mechanism may not even be new, just never described. Illyes was explaining one case and made no announcement. ## Summary Google can briefly keep an important URL in the index after it starts returning a 4xx error, so people can still find it. The price is a weaker result: without content from the server Google can't show a snippet or a favicon, only the link and the last known title. URLs with a persistent 4xx error are still removed from the index. The VLC case also shows where such an error can come from: a server that now answers AI bots with 418 was answering Google with it for a while too. If you block AI bots in a WAF or on the server, check that the rule doesn't catch Googlebot, and keep an eye on the status codes of your important pages. Search Console. Also check your server and CDN logs and WAF rules, because a wrong status code often comes from configuration rather than the application itself.' } ]} heading="Frequently Asked Questions" id="faq" /> ## Sources