Skip to content
RU
← All articles

Verify GPTBot IP ranges behind a CDN or reverse proxy

In short:

Verify GPTBot against the connection's original source IP and OpenAI's published CIDR feed. A User-Agent string alone is insufficient. The script below returns MATCH, NO_MATCH or UNKNOWN, keeping bad input separate from a negative result. Behind a CDN, establish a trusted client-IP chain before making the comparison.

Engineer comparing access logs with address ranges
Verification starts with the original IP and operator feed.

An access-log entry that says GPTBot is a claim made by an HTTP client. Before granting an exception in a WAF, identify the address that actually sent the request and compare it with the operator's current feed. Treat the result as one input to your access policy. Network membership does not authorize private content, disable authentication or justify unlimited request rates.

Step 1. Choose the feed for the claimed crawler

Use the operator's documentation to map a crawler name to its own address list. OpenAI documents separate purposes for GPTBot, OAI-SearchBot and ChatGPT-User: training-related crawling, search crawling and user-triggered access. Do not merge them into an unrestricted rule for every request associated with the company. Your editorial decision about training access may differ from your decision about search visibility.

The public gptbot.json snapshot used here contained 18 prefixes. That count is a measurement of one saved file. It is not a permanent limit or a reason to copy the ranges into a configuration that will never be refreshed. Retain the source URL, retrieval time and file hash. For retrospective log analysis, a historical snapshot is more useful than silently checking old requests against today's feed.

Trusted proxy diagram on a laptop screen
Recover the source address before checking CIDR membership.

Step 2. Recover a trustworthy source address

On a direct connection, use the network peer address. With Cloudflare, an AWS load balancer or another reverse proxy, the application may instead log an intermediary address. Comparing that proxy with the crawler feed answers the wrong question. Start with a diagram of the request path and inspect what each logging layer calls the client IP.

Forwarded headers are trustworthy only when received through your configured trusted proxy chain. If an origin accepts direct internet traffic and blindly trusts X-Forwarded-For, a client can supply an address from the published feed. The nginx real IP module provides explicit trusted-source configuration. Use the actual proxy addresses for your deployment. The enterno.io IP tool can help inspect a network owner, but an ASN match does not establish crawler identity.

Log the transport peer separately from the recovered client address. That preserves evidence when you investigate a mismatch. Parse IPv6 as an address, not a string prefix. Avoid resolving a logging problem by trusting an entire cloud provider's network: other services can occupy addresses in the same provider.

Step 3. Run the CIDR check

Download the official JSON over HTTPS to gptbot.json. Save the code below as verify.py and pass one source address. The verifier uses Python's standard library, validates the feed structure and supports both address families. An empty list, malformed prefix or unreadable file produces UNKNOWN. This prevents a broken feed refresh from being reported as a confirmed negative match.

import ipaddress
import json
import sys
from pathlib import Path

def verify(path, value):
    data = json.loads(Path(path).read_text())
    prefixes = data['prefixes']
    if not isinstance(prefixes, list) or not prefixes:
        raise ValueError('empty or invalid prefixes')
    networks = []
    for row in prefixes:
        fields = [k for k in ('ipv4Prefix', 'ipv6Prefix') if k in row]
        if len(fields) != 1:
            raise ValueError('invalid prefix row')
        key = fields[0]
        network = ipaddress.ip_network(row[key])
        expected = 4 if key == 'ipv4Prefix' else 6
        if network.version != expected:
            raise ValueError('address family mismatch')
        networks.append(network)
    address = ipaddress.ip_address(value)
    for network in networks:
        if address.version == network.version and address in network:
            return 'MATCH ' + str(network), 0
    return 'NO_MATCH', 1

if __name__ == '__main__':
    try:
        if len(sys.argv) != 3:
            raise ValueError('usage: verify.py feed.json source-ip')
        result, code = verify(sys.argv[1], sys.argv[2])
    except (ValueError, KeyError, TypeError, OSError):
        result, code = 'UNKNOWN invalid input or feed', 2
    print(result)
    sys.exit(code)

The ipaddress module handles subnet membership without enumerating every host. The script does not contact the address being checked. Our positive test address was selected arithmetically from the first published network; it is not a request observed in enterno.io's production logs.

python3 verify.py gptbot.json 132.196.86.1
MATCH 132.196.86.0/24
python3 verify.py gptbot.json 192.0.2.1
NO_MATCH
python3 verify.py gptbot.json bad-ip
UNKNOWN invalid input or feed

Step 4. Interpret the three outcomes

MATCH means the supplied address belongs to the saved list. NO_MATCH means it does not belong to that snapshot. UNKNOWN means the check could not be completed. Preserve those distinctions in dashboards and automation: a failed download should not inflate a counter labeled fake bots, and it should not open an unrestricted exception either.

Test caseResultExit codeNext action
Address inside the first CIDRMATCH0Apply the policy for that crawler role
192.0.2.1 outside the feedNO_MATCH1Check snapshot age and client-IP recovery
Invalid address stringUNKNOWN2Correct input parsing
Missing feed fileUNKNOWN2Restore a validated snapshot

A negative match can result from a stale feed, the wrong crawler feed or a proxy address in the log. Investigate those causes before describing a request as impersonation. If it remains unverified, retain your normal access rules and record the reason. A verified crawler can still impose an unacceptable load, so request limits and protected routes remain part of the policy.

Step 5. Refresh feeds outside the request path

Do not fetch a remote JSON feed for every incoming request. That adds network latency and makes every policy decision depend on a third-party service. Use a background job to retrieve the file, validate its structure, and atomically replace the local snapshot. Keep the previous valid version so a transient outage does not erase the only usable data.

Choose your refresh schedule and maximum acceptable age explicitly. This article does not assert an official OpenAI update interval. A stale snapshot may be useful for investigation, but its age must travel with the result. Alert on failed updates independently of whether the verifier still returns answers from an older file. Otherwise a healthy-looking dashboard can hide a feed that stopped updating weeks earlier.

Begin with observation mode. Compare the count of MATCH, NO_MATCH and UNKNOWN entries, and look for every request sharing the same proxy address. Change enforcement only after you understand those results. The monitoring allowlist guide helps reason about a full access path, and the launch checklist gives a place to record the baseline and rollback.

Troubleshoot a surprising result

Start with one timestamped log entry. Verify that the address is the original client, then confirm that the claimed crawler was checked against the corresponding feed. Shared infrastructure does not make all crawler roles interchangeable. A training-related exception should not silently become an exception for every user-triggered fetch.

When the downloader returns HTML instead of JSON, preserve the status and diagnostic details without credentials. DNS failures, timeouts and intermediary block pages require fixing the refresh process. The curl timing breakdown helps locate a network delay. External dependency monitoring helps expose an unavailable feed. Do not replace the file with an empty object just to make the fetch step appear successful.

For a request-time implementation, parse the validated feed once per refresh and retain the resulting networks in memory. The command-line example favors clarity for an investigation, not high-volume request handling. Test any production integration with your actual proxy chain and error policy before changing WAF decisions.

How we made this

We saved the public GPTBot JSON and ran four local Python checks: a selected address inside the first prefix, a documentation address outside the feed, an invalid string and a missing file. Raw results and code are stored with the draft. The experiment tests membership and error handling; it sends no probes to crawler addresses and contains no customer traffic. Google US and Yandex Moscow searches were collected separately. The broad Russian Wordstat query returned 94; the narrow query had no estimate. Neither search demand nor this four-case test measures how often crawler impersonation occurs.

FAQ

Is the GPTBot User-Agent sufficient?

No. The client controls that header. Compare the original source IP with the appropriate official feed and verify your trusted proxy configuration.

Can I allow an entire cloud provider?

That creates a much broader exception than a crawler feed. Other services can share that provider. Keep exceptions scoped to the documented role and intended routes.

Should UNKNOWN count as a fake bot?

No. It means verification failed. Apply your predetermined ordinary policy and alert on the broken verification process without claiming proven impersonation.

Sources

Check your website right now

Audit your site's SEO →
More articles: SEO
SEO
robots.txt Guide: Syntax, Rules, Testing and Ready-Made Files
16.03.2026 · 1 178 views
SEO
Sitemap XML: Structure, Limits, Generation and Validation
16.03.2026 · 1 121 views
SEO
Open Graph Tags: Control Link Previews in Telegram and X
21.07.2026 · 941 views
SEO
Website Migration Checklist: Avoid SEO and Downtime Pitfalls
16.03.2026 · 834 views