Move Your Website to a CDN Without Overwhelming the Origin Server

A newly activated CDN can speed up responses and still leave the origin server facing a spike in requests. Google explains why: the cache starts empty, and the first request for each uncached URL must go back to the origin. If Googlebot sees fast responses from the new edge, it may also gradually increase its crawl rate when there is demand.

The goal of this guide is to make a gradual migration: prepare priority URLs, warm the cache at a measured pace, and have a temporary brake ready. The examples use Python and Nginx; the specific CDN configuration will depend on the provider.

flowchart TD
  A["Elegir URL prioritarias"] --> B["Precalentar CDN con pausas"]
  B --> C["Verificar respuestas y carga"]
  C --> D["Dirigir tráfico a la CDN"]
  D --> E["Vigilar origen y rastreo"]
  E --> F["Aplicar freno temporal si hace falta"]

1. Prepare a limited list of URLs

Do not start by trying to crawl the entire site. Google recommends programmatically requesting pages with the most traffic, those that could see a spike soon, and URLs that are crawled frequently. The first step is to save those addresses to a file, one per line. Use the public domain already served by the CDN here; that way, requests will pass through the edge and can fill its cache.

https://www.example.com/
https://www.example.com/noticias/
https://www.example.com/productos/ejemplo/
https://www.example.com/ayuda/

This is a deliberate selection, not a list generated from every possible combination of parameters. Google warns that certain URL structures, such as faceted navigation or calendars with many addresses, can contribute to a sharp increase in crawling.

2. Warm the cache at a controlled pace

This standard Python script reads the list and visits each address in order, with a pause between requests. It does not create concurrency or retry in a loop: if a request fails, it logs the failure and continues. Set the pause cautiously and run it on a small list first while monitoring the origin.

#!/usr/bin/env python3
"""Solicita URL públicas para precalentar la caché de una CDN."""

import sys
import time
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen

URL_FILE = "urls.txt"
PAUSE_SECONDS = 1.0
TIMEOUT_SECONDS = 15

with open(URL_FILE, encoding="utf-8") as file:
    urls = [line.strip() for line in file if line.strip()]

for url in urls:
    request = Request(url, headers={"User-Agent": "CacheWarmer/1.0"})
    try:
        with urlopen(request, timeout=TIMEOUT_SECONDS) as response:
            print(f"{response.status} {url}")
    except HTTPError as error:
        print(f"HTTP {error.code} {url}", file=sys.stderr)
    except (URLError, TimeoutError) as error:
        print(f"ERROR {url}: {error}", file=sys.stderr)
    time.sleep(PAUSE_SECONDS)

The pause alone does not guarantee a safe load: the cost depends on what the origin does to produce each response. If you see latency rise or errors appear, stop the process and slow it down. A repeated URL will only help if the CDN can cache it and the requests reach an edge location that does not yet have that copy.

3. Set an emergency limit in Nginx

At the origin, Nginx can limit the request rate per IP address. This example is a minimal, complete configuration: it defines the limit zone in the http context and applies the rule to the server. The limit_req directive can return a rejection response when the limit is exceeded; here, the status code is set to 429.

events {
    worker_connections 1024;
}

http {
    limit_req_zone $binary_remote_addr zone=per_ip:10m rate=5r/s;

    server {
        listen 8080;
        server_name _;

        location / {
            limit_req zone=per_ip burst=10 nodelay;
            limit_req_status 429;

            return 200 "Origen disponible\n";
        }
    }
}

These values are an illustrative starting point, not a figure recommended for every website. Before deploying it to production, validate the configuration with nginx -t and adjust it to the actual traffic pattern and the origin's capacity. If the CDN shares an outbound IP address among many visitors, limiting by IP at the origin may group different users together; check how the client address is passed through before basing the rule on it.

Do not block all bots as an automatic response to a spike. Google warns that CAPTCHA challenges or indiscriminate rules can prevent verified crawlers from accessing the site. Configure an exception in the CDN or firewall when the product allows it, and selectively filter unwanted traffic.

4. Check behavior before opening up traffic

Verify that the public domain responds and that the origin does not start to degrade while you warm the cache. This command displays the HTTP status code and total request time with curl. Repeat it for priority URLs and compare the results with server metrics; a successful response from the edge does not, on its own, prove that the origin is under less load.

#!/usr/bin/env bash
set -eu

while IFS= read -r url; do
    [ -n "$url" ] || continue
    curl --silent --show-error --output /dev/null \
        --write-out '%{http_code} %{time_total}s ' "$url"
    printf '%s\n' "$url"
done < urls.txt

Run the check with only a few addresses first. Once responses are correct and the origin load remains within what your service can handle, expand the list or direct more traffic to the CDN. Google notes that its crawlers do not suddenly increase their rate and that some CDNs apply connection controls, but a cold cache still sends initial requests to the origin.

5. If the origin is struggling, apply a short-term brake

If the volume continues to put the service at risk, Google recommends temporarily returning HTTP 429 (Too Many Requests) or 503 (Service Unavailable) to reduce the crawl rate. Google's guidance says these responses should be a brief measure: a few hours or, at most, one or two days. When using 429 or 503, you can add Retry-After to indicate when to retry the request.

Do not turn that brake into a permanent block or replace an overload response with a challenge page for all automated traffic. Monitor logs and server health while the measure is in effect. Google says it reduces crawling in response to a significant number of 429, 503, or 500 responses and automatically increases it when those errors decrease.

Mistakes to avoid

A cold cache does not protect against the first request for each unique URL: that request can still reach the origin. Warm the selected pages before sending all the traffic, and do not automatically interpret the first spike as evidence of an attack.

Also avoid enabling general bot protection without exceptions for verified crawlers, and do not use the Nginx limit as if you already knew the actual traffic or the IP address the server will see behind the CDN. If Googlebot increases crawling, check the logs to identify the traffic; Google's guidance also recommends reviewing URLs and site structure when the increase is sharp.

The practical sequence is simple: a small list, gradual cache warming, an origin check, and a gradual rollout. If the load spikes, apply a temporary limit and preserve access for verified crawlers.

Sources: Search Engine Roundtable, Google for Developers, Google for Developers, Google for Developers