You put your site behind a CDN to take load off the server, and the logs show the opposite: Googlebot sends more requests than before, and most of them still land on your origin server. Google described this on the Search Central blog on 11 November 2026, in a post called “Moving to a CDN without overwhelming your origin (or blocking Googlebot)”. Gary Illyes and Cherry Prommawin also covered it in episode 5 of “The Non-Canonical Cabinet”, the video that closes the series’ first season.

The post is short and tongue-in-cheek (a rubber chicken plays the edge server), but it describes three real problems. A faster server tells Google it can crawl faster. An empty cache doesn’t protect the origin yet. And bot protection switched on in a panic can knock Googlebot out of the results. Below I go through each one and the plan Google suggests for migration day.

Why crawling speeds up after a CDN launch

A CDN (Content Delivery Network) keeps copies of your assets on edge servers close to users. It takes load off the origin server, the one that holds the original content, and cuts response times. Google admits it keeps hearing the same story from site owners, though: right after the move to a CDN, crawler traffic to their server spikes, which is exactly what the migration was meant to prevent.

The reason is how Google calculates the crawl capacity limit, also called hostload: how much crawling a given host can take. Two rules apply:

  • slower responses lower the crawl rate. Google keeps measuring how the server is doing and backs off when response times grow, so it doesn’t overload it,
  • fast, stable responses raise it. When the server answers quickly and consistently and Google has a reason to crawl more (so-called crawl demand), the limit goes up step by step.

Once DNS points to the CDN’s edge network, connection and response times drop overnight. To Google’s systems it looks as if the server just got a massive hardware upgrade, so the crawlers ramp up their requests.

Google’s crawl budget documentation adds two details that matter during a migration. The limit is calculated per hostname (www.example.com and shop.example.com have separate budgets). It is also shared by all Google crawlers, so high demand from one of them leaves less for the rest.

Cold cache: why the CDN doesn’t protect you yet

A higher crawl rate wouldn’t be a problem if the CDN handled those requests on its own. A new CDN starts with an empty, cold cache, though.

On the first request for a URL at a given edge location there is no copy of the asset yet (HTML, CSS, JavaScript, images). That’s a cache miss: the edge server has to fetch the response from the origin, store it and only then send it to the client. Only later requests for the same URL are served from cache (a cache hit) without touching the origin.

People and crawlers use the cache in completely different ways:

UsersGooglebot
Traffic patterna small set of popular URLs (homepage, most-read articles, active promotions)a systematic walk through the whole URL inventory
Cache warm-upfast, the same pages get visited again and againslow, most URLs are visited for the first time
Effect on the originsmall after a short whileevery new URL is a cache miss that goes straight to the server

Put together, it’s an awkward combination. Fast edge responses encourage Google to crawl harder, and every URL that isn’t cached yet passes through the CDN straight to the origin. Google notes that crawlers don’t ramp up instantly and that many CDNs have built-in connection pacing. Still, until the cache warms up, the origin remains the bottleneck.

The more URLs a site has, the longer this lasts. In a shop with thousands of product pages, filters and variants, the crawler keeps hitting URLs the edge has never seen. If the origin handles sudden load badly (slow database queries, no application cache), the migration plan has to account for it. That’s my conclusion from the mechanism Google describes, as Google gives neither the size of the spike nor any thresholds.

Bot protection that blocks Googlebot

A request graph in the origin logs right after the migration can be scary, especially when it’s not clear where the traffic comes from. The temptation is obvious: log into the CDN dashboard and turn on blanket bot protection. Google warns that this is a mistake.

Rules without exemptions treat all automated traffic the same, including the crawlers you care about. When a CDN or a web application firewall (WAF) intercepts a request and serves a challenge such as a CAPTCHA, Google’s crawler fails it. It gets stuck on an error or challenge page that is useless both for Search and for other Google products that rely on crawled data.

To Google, the site then looks deliberately restricted or broken. If that lasts, the crawlers back off and pages, shopping listings and the rest of the content start dropping out of search results.

This kind of block is easy to miss because the owner never sees it. Everything works in the browser while the crawler gets a 403 or a challenge screen. That’s why after every change to CDN or WAF rules I check what Googlebot actually receives: in the crawl stats report in Google Search Console and in the server logs.

A CDN migration plan: what to do before the switch

Google suggests three concrete steps.

1. Let verified bots through

If you have to block bots, add an exemption for verified crawlers in your CDN and firewall. According to Google, most major CDNs and WAFs offer a built-in toggle or managed rule along the lines of “allow known / verified bots”. You can recognise Google’s crawlers by the IP ranges Google publishes.

The verification itself is described in “Verify requests from Google crawlers and fetchers”. You run a reverse DNS lookup on the IP address from your logs and check that the domain is googlebot.com, google.com or googleusercontent.com. Then a forward DNS lookup has to return the same IP. For automated matching there are JSON files with the IP ranges.

Better still, Google says, point robots.txt or your rules at the specific bots you don’t want instead of blocking every automated client. I explain how to tell search engine crawlers from AI crawlers in AI crawlers vs search crawlers, and the file’s syntax in what you can put in robots.txt.

2. Warm the cache before you switch traffic

Before you route all traffic through the new CDN, fetch through it, at a controlled pace:

  • your highest-traffic pages,
  • pages that are about to get a traffic peak (campaigns, seasonal launches),
  • the URLs Googlebot visits most often (you’ll find them in the server logs).

That way the edge has copies before the wave of requests arrives. Google jokes that you can also ask your friends to open two URLs each. In practice a simple script that requests a list of URLs from your sitemap or logs does the job.

3. Use 429 or 503 as a temporary brake

If crawling still puts heavy load on the origin despite the warm-up, set rate limits on the server or the CDN that briefly return 429 (Too Many Requests) or 503 (Service Unavailable). Google talks about a few hours, 1 to 2 days at most, and points to its guide “Reduce the Google crawl rate”.

That guide has caveats worth knowing before you pull the brake:

  • the slowdown covers the whole host, not just the URLs returning errors. Google lowers the crawl rate for the entire hostname and raises it again on its own once the errors drop,
  • with 503 and 429 you can add a Retry-After header (a number of seconds or a UTC date) that tells the crawler when to come back,
  • if Googlebot sees these codes on the same URL for several days, the URL may be dropped from the index,
  • limiting crawling has broad effects: Google discovers fewer new pages, refreshes existing ones less often (price and stock changes show up in results later), and removed pages stay in the index longer.

I wrote about how Google treats 5xx codes on a site that’s been taken offline in the post on taking a site down to patch PHP.

How to check that the migration is going well

Google’s post gives no thresholds or metrics, so this list is my own practice, based on the mechanism it describes. During and after the switch to a CDN I watch four things:

  1. Origin logs. How many Googlebot requests reach the server compared with the period before the migration, and what share of them are cache misses. If the CDN reports a cache hit ratio, it should go up day by day.
  2. Crawl stats in Search Console. Whether the number of requests goes up while average response time drops, and whether server errors or 403 responses appear.
  3. Responses the crawler sees. Whether requests from Google’s IP ranges get CAPTCHA challenges or error pages in the WAF logs.
  4. Site sections. Which parts of the site stay “cold” the longest, meaning where cache misses persist.

A crawl spike after a migration is usually good news: Google considers the infrastructure healthy and wants to fetch more. The trouble starts when the origin can’t keep up or overly strict protection shuts the crawler out.

If traffic grows not on regular pages but on URLs with filters and sorting, the cause may be faceted navigation rather than the CDN. Google’s guide to reducing crawl rate names it as the most common source of sudden spikes, next to calendars with thousands of date URLs.

What Google doesn’t say

This is an educational post, not an announcement of an algorithm change, and its scope is narrow:

  • Google doesn’t say how much or how fast the crawl rate grows after a migration. It only says crawlers don’t speed up right away.
  • There’s no word on how long the cache warm-up takes or at what site size the problem becomes noticeable. The closest hint is the crawl budget documentation, which names sites with more than a million unique pages, or more than 10,000 pages that change daily, as its rough audience. Google stresses these are estimates, not thresholds. For a site with a few hundred URLs I’d consider the risk small.
  • There are no ready settings for specific CDN providers. Option names differ between services, so check your own provider’s documentation.

Google already covered CDNs and crawling in its “Crawling December” series in 2024. The new post focuses on the transition itself: the cold cache and the side effects of blanket bot protection. Barry Schwartz also covered the episode on Search Engine Roundtable.

Summary

A CDN speeds up responses, and Googlebot reads that as permission to crawl faster. While the cache is cold, the crawler reaches many URLs for the first time and every such request lands on the origin. Then there’s the second risk: blanket bot protection that serves the crawler a CAPTCHA and makes pages drop out of the results.

Google’s advice fits in three points: let verified bots through, warm the cache before the DNS change, and if the origin needs a breather, return 429 or 503 for 1 to 2 days at most. Before a migration I also settle who reads the logs and when, so problems get caught in the first hours rather than a week later.

Frequently Asked Questions

Why does Googlebot crawl more after a CDN launch?

A CDN cuts response times, and Google treats fast, stable responses as a sign the server can take more. If there is crawl demand, the crawl capacity limit goes up step by step and the crawler sends more requests.

Does a CDN take load off the server right away?

No. A new CDN has an empty cache, so the first request for each URL at a given location goes to the origin (a cache miss). Googlebot visits many different URLs, so a lot of those requests reach the server before the cache warms up.

Can I turn on bot protection in my CDN?

Yes, but with an exemption for verified crawlers. Blanket rules with a CAPTCHA challenge also stop Googlebot, which cannot solve it. Google then sees the site as blocked or broken, and its URLs may drop out of the results.

How do I warm up a CDN cache before a migration?

Fetch through the new CDN, at a controlled pace, your highest-traffic pages, pages with an upcoming traffic peak (campaigns, seasonal launches) and the URLs Googlebot visits most often. You will find the last group in your server logs.

How can I quickly reduce crawling when the origin is overloaded?

Return a 429 or 503 status code for a short time, from a few hours to 1 to 2 days at most, optionally with a Retry-After header. Google slows crawling for the whole host, and keeping these codes for longer can get URLs dropped from the index.

Sources

  1. Moving to a CDN without overwhelming your origin (or blocking Googlebot) Google Search Central Blog · developers.google.com ·
  2. Optimize your crawl budget Google Crawling Infrastructure · developers.google.com
  3. Reduce the Google crawl rate Google Crawling Infrastructure · developers.google.com
  4. Verify requests from Google crawlers and fetchers Google Crawling Infrastructure · developers.google.com
  5. Crawling December: CDNs and crawling Google Search Central Blog · developers.google.com ·
  6. Google: How Moving to a CDN Can Accidentally Overwhelm Your Website Search Engine Roundtable · seroundtable.com ·