5xx errors and SEO: what to do when your server fails (500, 502, 503, 504 and 429)

Search Console flags "Server error (5xx)" and the page loads fine when you check it. 5xx errors are not just a page that is down: Google saw it down, slowed crawling across the whole hostname, and if the error repeats it eventually drops the URL from the index.

What 5xx errors actually tell Google (and what they don't)

A 5xx is the class of response where the server admits the failure is its own: it is aware that it has erred or that it cannot complete the request. It is not a 404, where the server simply says the resource does not exist.

The practical consequence is spelled out in Google's documentation on HTTP status codes and crawlers: on 5xx errors or a 429, Googlebot temporarily slows down. Indexed URLs stay in the index but are eventually dropped, content served with a 5xx is ignored, and once the server answers 2xx again the crawl rate climbs back gradually on its own.

The part most people miss: on a 404 Google quickly concludes the page is gone and removes it; on a 5xx it concludes nothing, but it does not wait forever either. It is an extension, not a pardon.

Why the error is gone but the cost is not

Search Console shows what happened when Google crawled, not what happens when you open the URL in a browser. The Pages report help says it plainly: server errors can be temporary and the live test can pass even though the crawl failed. That gap is exactly what lets the problem become chronic, because nobody looks again.

Every 5xx has a different culprit

Each code points somewhere else, so start by naming it correctly:

• 500 Internal Server Error - the application blew up: an uncaught exception, a fatal language error, a failed database connection. It is the only one that points straight at your code or your environment.
• 502 Bad Gateway - the proxy, load balancer or CDN got an invalid response from the server behind it, usually because the app process died or the worker pool ran out.
• 503 Service Unavailable - temporary overload or scheduled maintenance. This is the most honest 5xx: if you are going to be down on purpose, this is the code to send.
• 504 Gateway Timeout - the proxy waited and the origin never answered in time. It talks about slowness, not downtime: a heavy query, a page that takes too long to render. If that rings a bell, start with time to first byte.
• 429 Too Many Requests - not a 5xx by definition, but Google treats it as a server error and groups it under HTTP 429/5XX. If your WAF rate-limits by IP, Googlebot is the first to hit the ceiling.

503 with Retry-After: telling Google when to come back

The header accepts delay-seconds (Retry-After: 120) or an absolute UTC date. Google documents the same header for both 503 and 429. It fixes nothing, but it stops Googlebot from hammering the origin while you work.

Proprietary CDN codes in the 5xx range still count

Some proxies return non-standard codes inside the 5xx range, like 52x, when they cannot talk to the origin. To Google those are server errors all the same: do not fall for the "it is not really a 500" argument.

Finding the failing layer

Walk the layers from the outside in: DNS and TLS, CDN or WAF, web server, application, database. The code usually gives the layer away: 502 and 504 point at the proxy-to-origin hop, 500 at the application, 429 at a request limit.

Compare the public URL against the origin directly:

curl -sI https://example.com/product | head -1

If the CDN URL returns 502 while the origin answers 200, the problem sits in the middle hop. If the origin fails too, keep walking down until the first broken component appears. And if the response changes with the user-agent, you already know where to look.

The WAF that blocks Googlebot

Google's own help warns that misconfigured anti-DoS systems and firewalls can block the crawler, because Googlebot sends far more requests than a human visitor. Sometimes that comes back as 403, but under bursts many return 429 or plain 5xx. Rule of thumb: if the 5xx errors show up in Google's report and never in your application metrics, check the WAF before the code.

Where to see it in Search Console

The Pages report groups URLs under the "Server error (5xx)" status: that is where you see how many there are and a few examples, which tells you whether it is a whole template or a handful of odd cases.

Settings > Crawl stats groups Googlebot requests by response code, file type and purpose, and warns you when server errors point at an availability problem. That is where a spike shows over time and whether it hit the whole hostname. For per-URL detail, the URL Inspection API automates the check.

Validating the fix without rushing it

Ask for validation once the server has been stable for a while, not right after a restart: Google retests a sample and takes days to decide. Meanwhile, your logs and the 5xx share in Crawl stats are the only reliable signal.

The three causes behind almost every case

A deployment

Missing environment variables, stale config cache, changed permissions or a migration locking a table. It is the classic Friday afternoon 500, and you spot it by matching the spike time against your deploy history.

Exhausted resources

Without free workers the proxy answers 502; with the database at its limit you get 504; with a full disk or low memory, 500 and 503 alternate. That is the difference between being down and being drowned, and the limits of your hosting plan explain a good share of cases.

Traffic and bots

A real spike, a scraper or an over-strict WAF. The 429 is the clue here: something is limiting requests and it may be limiting Googlebot too. If the spike never shows in analytics but does in the logs, it is a bot.

Short outage, maintenance window, long outage

Short outage (hours)

Fix it and let the 5xx be a 5xx. Google slows down and recovers by itself once 2xx returns. Do not force reindexing.

Planned maintenance

503 plus Retry-After. The guide on reducing Google's crawl rate is explicit: serving 500, 503 or 429 reduces crawling urgently across the whole hostname, including URLs that answer fine. Use it for hours or a day, never as a permanent state.

When it lasts more than a day or two

If Googlebot keeps seeing those codes on the same URL for several days, the URL can be dropped from the index. A week-long 503 is not vacation mode: it is a slow deindexation request.

What not to do

Returning 200 with the error page: Google reads that as a soft 404, which is worse than an honest 5xx. Using 401 or 403 to slow crawling: the documentation confirms that 4xx codes other than 429 do not affect crawl rate. And removing URLs from the sitemap during the outage.

Monitoring: find out before Search Console does

Search Console warns days late; the server knows in seconds. Bare minimum: a synthetic check per template - home, category, product page, form - with alerts on the percentage of 5xx, not just on "site down". In the logs, split 5xx by user-agent: if only Googlebot suffers them it is a WAF issue, not code. Server log analysis answers that in minutes.

Availability is not indexability

A site that returns 200 but ships empty pages because of a template bug never appears in the 5xx report and can show up as a soft 404 instead. Monitoring has to look at content, not only at the status code.

A checklist to close the case

1. Identify the affected template in the Pages report and how many URLs it covers.
2. Reproduce the error with the same user-agent and from outside your network.
3. Read the server and WAF logs in the exact spike window.
4. Fix the cause, not the symptom: if the 502 came from exhausted workers, the fix is capacity.
5. Confirm a stable 200 for at least one full traffic cycle.
6. Only then validate and watch Crawl stats until the 5xx share is back to normal.

When a 5xx is genuinely serious

Three things matter: persistence, scope and timing. An intermittent 500 on a template with thousands of URLs weighs far more than an hour of 503 on a small blog, and a 5xx that lands during a launch costs more than the same error in a quiet week. When you recover, do not request indexing in bulk: crawl rate recovers on its own and you only burn quota. If traffic already dropped, work through diagnosing a traffic drop and check your crawl budget, because a sustained 5xx wastes crawling for nothing. And do not confuse signals: a 301 is permanent and deliberate; a 5xx is telling you something broke.