My Robots.txt Welcomed GPTBot. My Host Answered 429.

My robots.txt welcomed GPTBot, and my host answered it with a 429. On the night of September 28, two of my sites on the same shared hosting account answered any user agent containing “GPTBot” or “meta-externalagent” with an HTTP 429 Too Many Requests and an empty body. It happened on every HTML page, on the first request, from a fresh IP. Googlebot, bingbot, ClaudeBot, PerplexityBot, OAI-SearchBot and ChatGPT-User got 200 from the same pages.

A few hours later, a site on a second host, run by a different company, did the same thing.

What each crawler got from the two hosts

What each crawler got from the two hosts split by bot, not by page. I sent one request per user agent and wrote down the status code.

User agent	Host one	Host two
GPTBot	429	429
meta-externalagent	429	429
OAI-SearchBot	200	200
ChatGPT-User	200	200
ClaudeBot	200	200
PerplexityBot	200	200
Googlebot	200	200
bingbot	200	200

I tried 18 user agents on the first host. Fifteen got 200, among them Applebot, Amazonbot, CCBot and facebookexternalhit, the crawler Meta uses for link previews. Bytespider got a 403. The two that got 429 are both described by their owners as collecting for AI training. OpenAI’s crawler documentation says GPTBot crawls content that may be used to train its foundation models, while OAI-SearchBot is the one that surfaces sites in ChatGPT search. Meta says meta-externalagent crawls for uses such as training AI models.

So ChatGPT search could still read these pages. My read is that the cost sat in training: what OpenAI’s models learn from the open web, as opposed to what ChatGPT finds in a live search.

Robots.txt was no help in spotting it. On the second host the file allowed every crawler in the table, and on both hosts robots.txt itself answered GPTBot with a 200 while every page behind it answered 429. A checker that reads only robots.txt would have called both sites open. On the first host, the 429s even carried the headers my own .htaccess file adds, so they came from inside my site’s server setup, though nothing in my file sends a 429.

What the host’s support said about the 429

What the host’s support said about the 429 changed once, and then the answer was no. I opened a ticket with the first host that night. The first reply said the 429 came from server-level throttling, bot management meant to stop denial-of-service attacks, and not from my files or from ModSecurity. Only their server team could change it, so they escalated.

At 06:12 UTC on September 29, GPTBot got 200 on six of six URLs on both sites. Meta’s crawler still got 429 on most pages. Then second-level support named a different cause: a server-side ModSecurity rate-limit rule called “Rate limit exceeded for Meta-ExternalAgent bot,” most likely with a twin for GPTBot. Their suggestion was to switch ModSecurity off for my domain.

I declined. ModSecurity is the web application firewall for the whole site, and I don’t switch off security settings to let a crawler in.

My robots.txt said yes to GPTBot, and the server said 429 anyway.

The 429 was back within 11 hours

The 429 was back within 11 hours of the fix. At 17:02 UTC, GPTBot got 429 on every HTML page of both sites again, while robots.txt still got 200. I confirmed it from my own connection and from DataForSEO’s on-page crawler with a custom user agent, which gave me a second IP. Meta’s crawler got 200 from DataForSEO’s IP and 429 from mine that afternoon, so part of the limit was tied to the IP.

I asked support to exclude just the two rules for my two domains and keep the rest of ModSecurity running. The answer was a refusal: the rule can’t be disabled on their shared servers, because it’s part of their protection against denial-of-service attacks.

On September 30, one of the two sites, still on that server, gave GPTBot a 200 three times out of three and meta-externalagent a 429 three times out of three. That on-and-off pattern fits a limit that resets and trips again, which is what support said it was. How a rate limit trips on the first request from a fresh IP, I can’t see from outside. I only know that it did.

A new server answered GPTBot with 200

A new server answered GPTBot with 200 the same day. The other site moved to its own hosting account on a different server at the same company, and there GPTBot, meta-externalagent, OAI-SearchBot, ClaudeBot, PerplexityBot and Googlebot all got 200 on an inner page. So the rule wasn’t on every server that host runs, and the practical fix was getting off that one. The move brought problems of its own.

I tested the second host on one server only, so I can’t say how far its rule reaches.

How I test crawler access now

I test crawler access now with one request per user agent, spaced out, on real pages, from two IPs.

Each part of that comes from this case. Robots.txt gets its own request because both hosts served it to GPTBot while refusing every page. The spacing is there because a burst can trip a rate limit by itself. The second IP is there because a per-IP limit can make a block look fixed from one address and total from another. And the recheck is there because the first fix lasted less than 11 hours.

Google’s guide to changing your hosting warns that firewall or denial-of-service protection can block Googlebot. Here the same kind of protection blocked GPTBot, and Search Console, which only reports on Google’s crawlers, would never have shown it. If your host does this, ask for the named rule to be excluded for your domain and keep the firewall on. If they say no, as mine did, the fix is a server without the rule.

Sources

1. Overview of OpenAI Crawlers (OpenAI)
2. Meta Web Crawlers (Meta for Developers)
3. 429 Too Many Requests (MDN Web Docs)
4. Changing your hosting (Google Search Central)