Robots.txt and pop-ups: hiding a dialog from Googlebot

On November 3, 2026, Barry Schwartz wrote on Search Engine Roundtable about John Mueller’s answer to a question on Reddit: how do you stop Google from rendering a pop-up that a script inserts into the page? Mueller gave two options, and both rely on robots.txt. You can block the external domain the pop-up comes from, or the JavaScript file that loads it. The exchange is a good reminder that robots.txt also decides which files Googlebot can use to build the rendered version of a page.

The question and Mueller’s answer

The person asking runs a site that has to show an age verification pop-up for legal reasons. The pop-up lives on a separate domain and is inserted into the page through an API. They wanted to know whether a Disallow in that second domain’s robots.txt would prevent Google from rendering the pop-up.

Mueller replied: “That would generally work.” He added that if a specific JS file loads the pop-up, you could also disallow that file in robots.txt. And he gave one caveat: “if you only have one JS file that does a bunch of things, I wouldn’t do that.”

Why it works

According to Google’s documentation, Google processes JavaScript pages in three phases: crawling, rendering and indexing. Pages that return a 200 status code go into the rendering queue. Once Google’s resources allow, a headless Chromium executes the JavaScript, and Google uses the rendered HTML to index the page. How server-side and client-side rendering differ here is covered in web technologies and SEO.

Before Googlebot fetches a URL from the queue, it checks robots.txt. If the URL is disallowed, Googlebot skips the HTTP request entirely. The documentation also says plainly that Google won’t render JavaScript from blocked files or on blocked pages. Both of Mueller’s options follow from that rule.

Option 1: block it in the robots.txt of the domain serving the pop-up

Say you embed content from another domain, through an iframe or by calling its script, and that domain disallows all crawling in its robots.txt. Crawlers that follow robots.txt, such as Googlebot, will not fetch that content and will not use it in rendering. Mueller pointed out that this is how many ad services work.

He also explained what happens on Googlebot’s side. It doesn’t crawl those URLs as separate pages, but it does try to fetch them while rendering. If robots.txt blocks them, Googlebot doesn’t fetch the content and can’t use it.

The catch: you need control over that other domain’s robots.txt. If the verification service is your own, it’s a single line. A third-party service is usually not something you can reconfigure.

Option 2: block the script in your own robots.txt

If the pop-up is loaded by a separate file, for example load-verification-popup.js (Mueller’s example), you can block it in your own site’s robots.txt. The script then won’t load during rendering. Mueller said that in his experience this is less common. But if you can’t influence the robots.txt on the pop-up’s domain and you want the pop-up kept out of rendering, this is the option you have left.

The rule could look like this (the path is only an example):

User-agent: *
Disallow: /assets/js/load-verification-popup.js

User-agent: * applies to all crawlers. To block the file for Googlebot only, use User-agent: Googlebot.

What Google says about age gates

Google’s documentation on interstitials and dialogs has a separate section on dialogs a site is legally required to show, which it calls mandatory interstitials. Its example is a casino that has to show an age gate. Those interstitials are exempt from the guidance on intrusive interstitials, but Google still recommends that you:

overlay the content with the interstitial instead of replacing it, so Google can still index at least some of the content,
don’t redirect incoming requests to a separate page for consent. If every URL redirects to one page, all but that page drop out of search results, because Googlebot can only fetch that one page.

The same page has a recommendation for adult content behind an age gate: let Googlebot crawl the content without triggering the gate, by verifying that the request really comes from Googlebot and serving the content without the age gate. That is a third route to the same goal, and the only one of the three that Google documents.

Before you add a Disallow

Mueller set one condition: your pages still need to load when those URLs aren’t fetched. He added that for something like a verification pop-up this is usually not a problem. Check it on your own site anyway before you ship the block.

Google lets you block resource files such as unimportant images, scripts or style files in robots.txt if the page won’t be significantly affected without them. If their absence makes the page harder for Google’s crawler to understand, don’t block them, or Google won’t do a good job of analyzing pages that depend on them. So:

Run a live test in the URL Inspection tool in Search Console. It shows a screenshot of the page as Google’s tester saw it and the list of loaded resources. The rest of the tool is covered in the Google Search Console guide.
Make sure the main content, the menu and the links don’t depend on the script you are blocking.
Block that one URL in your browser (Chrome DevTools has request blocking in the Network panel) and see what is left of the page.

I cover general setup in the robots.txt guide, and how Google reads its rules in What can you put in robots.txt.

What this method doesn’t solve

The index stays as it is. Google states that robots.txt is not a mechanism for keeping a page out of Google. A blocked URL can still show up in results if other pages link to it, just without a description. To keep a page out of results, use noindex in a meta tag or HTTP response header, password protection, or remove the page. Mueller’s answer covers something narrower: what Google sees once it renders a given page.

Only well-behaved crawlers listen. Googlebot obeys robots.txt, but the file can’t enforce anything and other crawlers may ignore it. Users still see the pop-up.

Whether the pop-up hurts you is a separate question. Mueller talked about keeping Google from rendering the pop-up. He didn’t comment on whether a pop-up hurts a site in search. For legally required dialogs, that topic is covered by the interstitials documentation mentioned above.

The pop-up needs a URL of its own. Robots.txt blocks URLs. If the pop-up’s code is part of the page’s HTML or sits in the same JS file as everything else, there is no separate URL you could block without side effects.

Summary

When you audit robots.txt, look at resources as well as pages, because this file decides what Googlebot can use when it renders. A pop-up loaded from another domain or from its own JS file can be hidden from Google this way, as long as the page works without it. For age gates, Google’s documentation describes one more route: verify Googlebot and serve it the content without the gate. The same rule cuts the other way, too: an accidentally blocked script or stylesheet that the content depends on makes the page harder for Google to understand. So before you add a Disallow, check the rendered page in Search Console.

Frequently Asked Questions

Can robots.txt hide a pop-up from Google?

Yes, if the pop-up loads from another domain or from a separate JavaScript file. John Mueller described two options: disallow crawling in the robots.txt of the domain the pop-up comes from, or disallow the script file in your own site’s robots.txt. Googlebot then won’t fetch that resource during rendering.

Does blocking a script in robots.txt remove a page from the index?

No. According to Google, robots.txt is not a way to keep pages out of search results. Use noindex (a meta tag or a response header), password protection, or remove the page. Blocking a script only changes what Google sees when it renders the page.

When shouldn’t you block the pop-up script?

When one JavaScript file handles many things. Mueller said he wouldn’t block it in that case. Blocking such a file can break the rendering of the main content, the menu or the links.

What does Google recommend for age gates?

Legally required interstitials, such as age gates, are exempt from Google’s guidance on intrusive interstitials. Google recommends overlaying the content rather than redirecting to a separate page. For adult content behind an age gate, it recommends letting Googlebot crawl the content without triggering the gate, after verifying that the request comes from Googlebot.

How can you check whether Google sees the pop-up?

Run a live test in the URL Inspection tool in Google Search Console. It shows a screenshot of the page as Google saw it and the list of loaded resources. For general setup, see the robots.txt guide.