Verify GPTBot against the connection's original source IP and OpenAI's published CIDR feed. A User-Agent string alone is insufficient. The script below returns MATCH, NO_MATCH or UNKNOWN, keeping bad input separate from a negative result. Behind a CDN, establish a trusted client-IP chain before making the comparison.

An access-log entry that says GPTBot is a claim made by an HTTP client. Before granting an exception in a WAF, identify the address that actually sent the request and compare it with the operator's current feed. Treat the result as one input to your access policy. Network membership does not authorize private content, disable authentication or justify unlimited request rates.
Step 1. Choose the feed for the claimed crawler
Use the operator's documentation to map a crawler name to its own address list. OpenAI documents separate purposes for GPTBot, OAI-SearchBot and ChatGPT-User: training-related crawling, search crawling and user-triggered access. Do not merge them into an unrestricted rule for every request associated with the company. Your editorial decision about training access may differ from your decision about search visibility.
The public gptbot.json snapshot used here contained 18 prefixes. That count is a measurement of one saved file. It is not a permanent limit or a reason to copy the ranges into a configuration that will never be refreshed. Retain the source URL, retrieval time and file hash. For retrospective log analysis, a historical snapshot is more useful than silently checking old requests against today's feed.

Step 2. Recover a trustworthy source address
On a direct connection, use the network peer address. With Cloudflare, an AWS load balancer or another reverse proxy, the application may instead log an intermediary address. Comparing that proxy with the crawler feed answers the wrong question. Start with a diagram of the request path and inspect what each logging layer calls the client IP.
Forwarded headers are trustworthy only when received through your configured trusted proxy chain. If an origin accepts direct internet traffic and blindly trusts X-Forwarded-For, a client can supply an address from the published feed. The nginx real IP module provides explicit trusted-source configuration. Use the actual proxy addresses for your deployment. The enterno.io IP tool can help inspect a network owner, but an ASN match does not establish crawler identity.
Log the transport peer separately from the recovered client address. That preserves evidence when you investigate a mismatch. Parse IPv6 as an address, not a string prefix. Avoid resolving a logging problem by trusting an entire cloud provider's network: other services can occupy addresses in the same provider.
Step 3. Run the CIDR check
Download the official JSON over HTTPS to gptbot.json. Save the code below as verify.py and pass one source address. The verifier uses Python's standard library, validates the feed structure and supports both address families. An empty list, malformed prefix or unreadable file produces UNKNOWN. This prevents a broken feed refresh from being reported as a confirmed negative match.
import ipaddress
import json
import sys
from pathlib import Path
def verify(path, value):
data = json.loads(Path(path).read_text())
prefixes = data['prefixes']
if not isinstance(prefixes, list) or not prefixes:
raise ValueError('empty or invalid prefixes')
networks = []
for row in prefixes:
fields = [k for k in ('ipv4Prefix', 'ipv6Prefix') if k in row]
if len(fields) != 1:
raise ValueError('invalid prefix row')
key = fields[0]
network = ipaddress.ip_network(row[key])
expected = 4 if key == 'ipv4Prefix' else 6
if network.version != expected:
raise ValueError('address family mismatch')
networks.append(network)
address = ipaddress.ip_address(value)
for network in networks:
if address.version == network.version and address in network:
return 'MATCH ' + str(network), 0
return 'NO_MATCH', 1
if __name__ == '__main__':
try:
if len(sys.argv) != 3:
raise ValueError('usage: verify.py feed.json source-ip')
result, code = verify(sys.argv[1], sys.argv[2])
except (ValueError, KeyError, TypeError, OSError):
result, code = 'UNKNOWN invalid input or feed', 2
print(result)
sys.exit(code)
The ipaddress module handles subnet membership without enumerating every host. The script does not contact the address being checked. Our positive test address was selected arithmetically from the first published network; it is not a request observed in enterno.io's production logs.
python3 verify.py gptbot.json 132.196.86.1
MATCH 132.196.86.0/24
python3 verify.py gptbot.json 192.0.2.1
NO_MATCH
python3 verify.py gptbot.json bad-ip
UNKNOWN invalid input or feed
Step 4. Interpret the three outcomes
MATCH means the supplied address belongs to the saved list. NO_MATCH means it does not belong to that snapshot. UNKNOWN means the check could not be completed. Preserve those distinctions in dashboards and automation: a failed download should not inflate a counter labeled fake bots, and it should not open an unrestricted exception either.
| Test case | Result | Exit code | Next action |
|---|---|---|---|
| Address inside the first CIDR | MATCH | 0 | Apply the policy for that crawler role |
| 192.0.2.1 outside the feed | NO_MATCH | 1 | Check snapshot age and client-IP recovery |
| Invalid address string | UNKNOWN | 2 | Correct input parsing |
| Missing feed file | UNKNOWN | 2 | Restore a validated snapshot |
A negative match can result from a stale feed, the wrong crawler feed or a proxy address in the log. Investigate those causes before describing a request as impersonation. If it remains unverified, retain your normal access rules and record the reason. A verified crawler can still impose an unacceptable load, so request limits and protected routes remain part of the policy.
Step 5. Refresh feeds outside the request path
Do not fetch a remote JSON feed for every incoming request. That adds network latency and makes every policy decision depend on a third-party service. Use a background job to retrieve the file, validate its structure, and atomically replace the local snapshot. Keep the previous valid version so a transient outage does not erase the only usable data.
Choose your refresh schedule and maximum acceptable age explicitly. This article does not assert an official OpenAI update interval. A stale snapshot may be useful for investigation, but its age must travel with the result. Alert on failed updates independently of whether the verifier still returns answers from an older file. Otherwise a healthy-looking dashboard can hide a feed that stopped updating weeks earlier.
Begin with observation mode. Compare the count of MATCH, NO_MATCH and UNKNOWN entries, and look for every request sharing the same proxy address. Change enforcement only after you understand those results. The monitoring allowlist guide helps reason about a full access path, and the launch checklist gives a place to record the baseline and rollback.
Troubleshoot a surprising result
Start with one timestamped log entry. Verify that the address is the original client, then confirm that the claimed crawler was checked against the corresponding feed. Shared infrastructure does not make all crawler roles interchangeable. A training-related exception should not silently become an exception for every user-triggered fetch.
When the downloader returns HTML instead of JSON, preserve the status and diagnostic details without credentials. DNS failures, timeouts and intermediary block pages require fixing the refresh process. The curl timing breakdown helps locate a network delay. External dependency monitoring helps expose an unavailable feed. Do not replace the file with an empty object just to make the fetch step appear successful.
For a request-time implementation, parse the validated feed once per refresh and retain the resulting networks in memory. The command-line example favors clarity for an investigation, not high-volume request handling. Test any production integration with your actual proxy chain and error policy before changing WAF decisions.
How we made this
We saved the public GPTBot JSON and ran four local Python checks: a selected address inside the first prefix, a documentation address outside the feed, an invalid string and a missing file. Raw results and code are stored with the draft. The experiment tests membership and error handling; it sends no probes to crawler addresses and contains no customer traffic. Google US and Yandex Moscow searches were collected separately. The broad Russian Wordstat query returned 94; the narrow query had no estimate. Neither search demand nor this four-case test measures how often crawler impersonation occurs.
FAQ
Is the GPTBot User-Agent sufficient?
No. The client controls that header. Compare the original source IP with the appropriate official feed and verify your trusted proxy configuration.
Can I allow an entire cloud provider?
That creates a much broader exception than a crawler feed. Other services can share that provider. Keep exceptions scoped to the documented role and intended routes.
Should UNKNOWN count as a fake bot?
No. It means verification failed. Apply your predetermined ordinary policy and alert on the broken verification process without claiming proven impersonation.