Open data · CC BY 4.0
State of Web Security: the top 10,000 sites
HSTS, Content-Security-Policy, TLS 1.0, certificate lifetimes, llms.txt and the robots.txt rules for AI crawlers, measured on the most popular domains of a public ranking.
Edition of · Tranco list JZPGY
Key findings
- 58.2% send an HSTS header
- 35.1% enforce a Content-Security-Policy
- 25.5% still accept TLS 1.0
- 17.2% block at least one AI crawler
- 11.6% publish llms.txt
- 90.1% negotiate TLS 1.3
Each share has its own base — the sites where it could be measured; see the table.
All figures
| Metric | Top 1,000 | All | Base |
|---|---|---|---|
| Home page served over HTTPS base 9,498 | 67.3% | 75.1% | 9,498 |
| HSTS header base 5,501 | 69.0% | 58.2% | 5,501 |
| HSTS max-age of a year or more base 5,501 | 55.4% | 44.8% | 5,501 |
| HSTS ready for the preload list base 5,501 | 23.4% | 16.3% | 5,501 |
| Enforced Content-Security-Policy base 5,501 | 51.6% | 35.1% | 5,501 |
| CSP in Report-Only mode only base 5,501 | 4.0% | 4.1% | 5,501 |
| TLS 1.3 negotiated base 7,220 | 90.6% | 90.1% | 7,220 |
| TLS 1.0 still accepted base 7,220 | 20.7% | 25.5% | 7,220 |
| Certificate expired base 7,220 | 0.0% | 0.3% | 7,220 |
| Certificate expires within 14 days base 7,220 | 1.4% | 1.0% | 7,220 |
| robots.txt blocks at least one AI crawler base 6,594 | 20.9% | 17.2% | 6,594 |
| llms.txt published base 7,278 | 16.8% | 11.6% | 7,278 |
AI crawlers blocked in robots.txt
Share of sites whose robots.txt disallows the crawler on the home page, of 6,594 sites where robots.txt could be read (a missing file allows all).
Certificate lifetimes
Validity period of the certificate each site presents, of 7,220 completed handshakes. CA/Browser Forum ballot SC-081 caps new public certificates at 200 days from 15 March 2026, 100 days from 15 March 2027 and 47 days from 15 March 2029.
Methodology
Domains: the top 10,000 of Tranco JZPGY, created . Measured 2026-11-05 – 2026-11-05 (UTC). Not measured: prohibited content (adult, gambling, sites blocked in Russia) — 195; robots.txt disallows our crawler — 239; robots.txt answered 5xx or 429 — 68 (we sent these sites nothing more; they are counted here and not listed in the data). Measured but without figures for headers: no answer — 2,186; plain HTTP only — 183; a bot-protection page — 1,407.
What we request
Per domain, at most: /robots.txt, the home page and /llms.txt — over HTTPS, over plain HTTP only when HTTPS does not answer, up to 5 redirects — and two TLS handshakes on port 443. Nothing else is fetched; bodies are read up to 64 KB (home page), 256 KB (llms.txt) and 512 KB (robots.txt). The scan runs with a global rate limit and a single request at a time per site, from one dedicated server that runs nothing else, located in Singapore.
User-Agent:
Mozilla/5.0 (compatible; EnternoResearch/1.0; +https://enterno.io/en/state-of-web-security)Redirects are followed to ports 80 and 443 only. How a site opts out: for site owners.
Definitions
- HTTPS — the home page, after redirects, is answered over HTTPS. The certificate is read, not validated: an expired or self-signed one still counts as HTTPS.
- Header metrics (HSTS, CSP) — read from the final HTTPS response of the home page. Only a 2xx/3xx answer counts; a 403, 406, 429 or 503 is a bot-protection page whose headers are not the site's: such sites are left out of the base.
- HSTS —
Strict-Transport-Securitywith max-age above 0; «a year» is max-age ≥ 31536000; «preload-ready» addsincludeSubDomainsandpreload. - CSP — an enforced
Content-Security-Policyresponse header;-Report-Onlyis counted separately; meta tags are not read. - TLS 1.3 — the version a current client (OpenSSL 3) negotiates. TLS 1.0 — a separate handshake offering TLS 1.0 only was completed; a failed one is tried once more, and a handshake lost on the network reads as «not accepted».
- Certificate — the leaf certificate presented on port 443: lifetime is notAfter − notBefore, days left are counted from the moment of the scan.
- AI crawlers — robots.txt parsed per RFC 9309 (the group for the crawler's token,
else
*; the longest rule wins) for the path/. A missing robots.txt (4xx) allows everything; a 403/406 bot-protection answer, a 5xx, a 429 or no answer is left out of the base. - llms.txt —
/llms.txtanswers 2xx with a non-HTML body that has a Markdown H1 (# Title).
Limitations
One vantage point (Singapore): sites that filter traffic by country or by data-centre address look unreachable or show a bot-protection page, and drop out of the bases above. Only the home page is read — a header set on other pages is not seen. Tranco ranks registrable domains, not the host that serves the site after a redirect.
For site owners
Requests with EnternoResearch in the User-Agent come from the research crawler of enterno.io. For each of the most popular domains of the Tranco ranking it reads robots.txt, the response headers of the home page and llms.txt, and makes two TLS handshakes — to publish aggregate figures on how widely HSTS, CSP and modern TLS are used. It does not crawl other pages, log in, send forms or test for vulnerabilities.
How to opt out
Add to your robots.txt:
User-agent: EnternoResearch
Disallow: /robots.txt is read before anything else. A site whose robots.txt disallows EnternoResearch (or every crawler) at / gets no further request and is not listed in the CSV: it only adds one to the count of sites that opted out. A robots.txt that answers 5xx or 429 is taken as the same refusal.
Contact
To have a domain removed from a published edition, or with any question about these requests, write to support@enterno.io or use the contact form.
Data and license
One row per measured domain, measured values only (1 / 0, empty — not measured). Free to use, share and adapt under CC BY 4.0 with attribution to enterno.io.
enterno-state-of-web-security-2026-11-04-tranco-JZPGY.csv
How to cite
enterno.io, "State of Web Security", November 2026 edition, https://enterno.io/en/state-of-web-security. Domain list: Tranco JZPGY (Le Pochat et al., NDSS 2019), https://tranco-list.eu/list/JZPGY/10000. CC BY 4.0.Columns
rank domain scanned_at_utc error https http_status bot_wall hsts hsts_max_age hsts_include_subdomains hsts_preload csp csp_report_only tls_version tls_1_0_accepted cert_lifetime_days cert_days_left robots_txt robots_allows_gptbot robots_allows_oai_searchbot robots_allows_claudebot robots_allows_claude_searchbot robots_allows_perplexitybot robots_allows_google_extended robots_allows_applebot_extended llms_txt