SSL Certificate Chain: How to Verify and Fix an Incomplete One

In short. A certificate chain is the path from your site's certificate up to a root CA the client already trusts. The server must send the leaf certificate and every intermediate; the client supplies the root from its own trust store. When an intermediate is missing, desktop browsers often paper over it — curl, Java, Python and mobile apps do not.

What a certificate chain actually is

A TLS certificate on its own proves nothing. It is a file that claims "this is example.com, and here is its public key". Trust appears only when the client can trace who signed that file, who signed the signer, and so on until it reaches a certificate already present in its list of trusted anchors. That sequence is the certificate chain, or chain of trust.

Three levels: leaf, intermediate, root

Root CA — self-signed, lives in the client's trust store
  └── Intermediate CA — signed by the root
        └── Leaf (end-entity) — your certificate for example.com

• Leaf (end-entity) — the certificate for a specific domain. It carries the SAN list of names, validity dates, the server's public key and the intermediate CA's signature. It is the only certificate tied to your private key.
• Intermediate — the CA's workhorse. It signs millions of leaf certificates. A chain can contain zero, one or several intermediates.
• Root — self-signed: Issuer equals Subject. Its private key is kept offline and used very rarely, which is exactly why roots do not sign end-entity certificates directly.

A two-level chain (root signs leaf) essentially does not exist on the public web — browser root programs require CAs to keep root keys offline. So "a chain of exactly one certificate" on a public site almost always means a self-signed certificate, not a conveniently short valid chain.

Why the root is not sent over the wire

The whole point of a root is that the client already has it locally and trusts it in advance. If the server ships its own copy, the client will not believe it just because it arrived — it will compare it against its own store. Sending the root adds nothing but bytes to the handshake (a typical root is 1–2 KB) and possibly an extra round trip on a slow link.

Rule: the server sends the leaf plus every intermediate up to — but not including — the root. SSL Labs flags a shipped root as "Chain issues: Contains anchor". It is not a validation failure, but it is not correct either.

What the server sends during the handshake

In the Certificate message the server transmits a list. RFC 8446, §4.4.2 (TLS 1.3) requires the leaf to come first; the ordering of the rest was softened to a recommendation, and implementations are told to tolerate arbitrary ordering and extraneous certificates. RFC 5246, §7.4.2 (TLS 1.2) made the ordering a hard requirement. In practice: modern clients usually cope with a scrambled order, older stacks do not. Build the file correctly anyway.

Trust stores: where clients keep root certificates

A trust store is the set of root certificates a system or an application trusts unconditionally. Here is the fact that derails half of all chain debugging: every platform has its own trust store, and they do not match. A certificate that is valid in Chrome on Windows can be invalid for a Java application on the very same machine.

System trust stores

Platform	Where roots live	How to inspect
Windows	System certificate store, "Trusted Root Certification Authorities"; roots arrive through the auto-update mechanism	certlm.msc, PowerShell: Get-ChildItem Cert:\LocalMachine\Root
macOS / iOS	System Roots keychain	Keychain Access; security find-certificate -a -p /System/Library/Keychains/SystemRootCertificates.keychain
Debian / Ubuntu	ca-certificates package, merged bundle at /etc/ssl/certs/ca-certificates.crt	ls /usr/share/ca-certificates/; add yours to /usr/local/share/ca-certificates/ then run update-ca-certificates
RHEL / AlmaLinux / Rocky	Bundle at /etc/pki/tls/certs/ca-bundle.crt	add to /etc/pki/ca-trust/source/anchors/ then run update-ca-trust
Alpine	ca-certificates package; the base image may ship without it entirely	apk add ca-certificates

Private stores: Firefox, Chrome, Java, Python, Node.js

• Firefox uses its own NSS store with the Mozilla root list and ignores the system store by default. The exception is enterprise roots: on Windows and macOS Firefox can pick them up from the OS (security.enterprise_roots.enabled in about:config). This is why "works in Chrome, fails in Firefox" is almost always a trust-store story, not a server story.
• Chrome in current versions uses its own Chrome Root Store and built-in verifier instead of the platform one, while still honouring roots an administrator installed into the system store. A corporate inspection proxy therefore keeps working, but an arbitrary root dropped into the OS store is no longer a guarantee.
• Java keeps roots in the cacerts keystore inside the JDK/JRE ($JAVA_HOME/lib/security/cacerts; on Java 8, $JAVA_HOME/jre/lib/security/cacerts), managed with keytool. It updates when the JDK updates — on a server whose JDK has not been touched in years, the root list is equally ancient.
• Python with requests defaults to the certifi package: a snapshot of the Mozilla root list baked into the virtual environment. It moves with the package, not with the OS.
• Node.js ships a built-in copy of the Mozilla root list. Extra roots go in via NODE_EXTRA_CA_CERTS; recent branches added a flag to use the system store instead.
• Go reads system bundles on Linux and calls platform APIs on macOS and Windows.

Android and mobile apps

On Android the system roots ship with the firmware (newer versions update part of the set through a separate module). Hence the classic problem with old handsets: a recently added root simply never appears until the OS updates, and on abandoned devices that means never. A second wrinkle: since Android 7, apps do not trust user-installed roots by default. Installing a certificate by hand and expecting an app — not just the browser — to accept it will not work without a Network Security Config change.

Whenever you ask "does the site work", ask for which client. An unqualified "it works" is meaningless: a browser, curl and a mobile app do not share a trust store.

A separate category is roots that exist in no public store but are distributed inside one country or one company. Certificates issued by such CAs are covered separately in SSL certificates from national CAs.

Path building: a chain is a tree of paths, not a list

The word "chain" is misleading. A client does not read a list top to bottom — it performs certification path building as described in RFC 5280, §6. Take the leaf, find candidate issuers (by the Issuer field and by Authority Key Identifier), try to extend the path through each candidate until you land on a trust anchor. There can be several candidates — which means several possible paths.

Cross-signing: one key, two different certificates

Cross-signing means the same CA public key is packaged into two different certificates signed by different issuers. Same Subject, same public key, different Issuer. The reason: a new CA enters the market, its root is absent from devices that never update, so it asks an older, widely distributed CA to sign its root — creating a second path that works on legacy clients.

The textbook example is Let's Encrypt: their ISRG Root X1 was for years additionally signed by IdenTrust's DST Root CA X3, which bought compatibility with old devices. Their ECDSA root ISRG Root X2 is in turn cross-signed by ISRG Root X1, so clients that only know X1 can still build a path.

# Path 1 (short): leaf → R11 → ISRG Root X1 (self-signed, in the trust store)
# Path 2 (long):  leaf → R11 → ISRG Root X1 (cross-signed) → DST Root CA X3

# Same server, same chain on disk —
# but different clients terminate at different trust anchors.

Root expiry: why some clients break and others do not

The industry's most instructive incident was the expiry of DST Root CA X3 in 2021. Servers did not change, certificates were valid, yet some clients started failing while others kept working. There are exactly three reasons, and all three are about path building:

1. Clients with ISRG Root X1 in their store found the short path and stopped there. The expired DST Root CA X3 never entered the picture.
2. Clients on old OpenSSL (the 1.0.2 branch) found the long path through the expired root, failed, and did not try the alternative — the alternate-path search landed in later branches. A valid path existed; the library never reached it.
3. Old Android kept working precisely because its verifier does not check the validity dates of the trust anchor itself. An expired root stayed trusted.

The takeaway outlives that particular incident: "the certificate is valid" is a property of the pair certificate + client, not of the server. The very same server response can be simultaneously valid for one client and invalid for another. That is normal PKI behaviour, not a bug.

Let's Encrypt later stopped issuing the DST Root CA X3 cross-signed chain, and the cross-sign itself has since expired. The certbot option --preferred-chain "DST Root CA X3", still recommended in stale tutorials, no longer does anything. The current list of chains is always on the Let's Encrypt certificates page.

AIA and caIssuers: who can fetch the missing intermediate

Certificates carry an Authority Information Access (AIA) extension with two fields: OCSP — the revocation responder URL — and caIssuers — a direct URL to the issuer's certificate. Some clients, unable to find an intermediate in the presented chain, follow that URL and download it themselves. The behaviour is called AIA fetching or AIA chasing.

# Inspect the AIA extension
openssl x509 -in cert.pem -noout -ext authorityInfoAccess

# Older syntax, works everywhere
openssl x509 -in cert.pem -noout -text | grep -A2 "Authority Information Access"

# Download the intermediate from caIssuers (usually DER) and convert to PEM
curl -sS http://r11.i.lencr.org/ | openssl x509 -inform DER -out intermediate.pem

AIA is the single reason a broken chain can live unnoticed for years: clients that do AIA fetching repair your configuration for you, and clients that do not simply fail. Caching is the second mask — a browser that saw the intermediate on another site may reuse it, so "it opens on my machine" proves nothing at all.

Never treat AIA as a substitute for a correct chain. Fetching over AIA means an extra HTTP request in the critical path of the handshake, to a third-party host, frequently over plain HTTP, with no availability guarantee. Intermediates belong in the file on your server.

Why the browser is fine but curl, Java and Python fail

This is the most common real-world scenario: the site opens in a browser, monitoring is green, and the API integration falls over. The cause is the combination of two client properties — where it gets roots from, and whether it can fetch missing intermediates.

Client	Source of roots	AIA fetching	Typical message on a broken chain
Chrome (Android)	Android system store	No	ERR_CERT_AUTHORITY_INVALID on the very site that opens on desktop
Chrome (desktop)	Chrome Root Store plus enterprise roots from the OS	Yes	ERR_CERT_AUTHORITY_INVALID (when the fetch also fails)
Firefox	Its own Mozilla NSS store	No; uses cached and preloaded intermediates instead	SEC_ERROR_UNKNOWN_ISSUER
Safari, iOS	System keychain	Yes	"Cannot establish a secure connection"
Windows: schannel, .NET, PowerShell	System store	Yes	The remote certificate is invalid
curl with OpenSSL or GnuTLS	OS CA bundle	No	unable to get local issuer certificate, exit code 60
Java (PKIX)	cacerts inside the JDK	No by default; enabled with the com.sun.security.enableAIAcaIssuers system property	PKIX path building failed: unable to find valid certification path to requested target
Python requests	certifi bundle	No	[SSL: CERTIFICATE_VERIFY_FAILED] unable to get local issuer certificate
Go	Platform system store	No	x509: certificate signed by unknown authority
Node.js	Built-in Mozilla list	No	UNABLE_TO_VERIFY_LEAF_SIGNATURE or UNABLE_TO_GET_ISSUER_CERT_LOCALLY
Mobile app (Android/iOS SDK)	Device system store	Usually yes, but depends on the networking layer	Connection drops during setup, often with no readable message

Chrome on Android deserves a separate note: unlike the desktop build, it does not fetch missing intermediates over AIA. "Opens on my laptop, fails on my phone" in the same browser is not a phone quirk — it is an incomplete chain.

The practical conclusion: test with the client that actually talks to your server. If the consumers are a mobile app and a Java payment gateway, a green padlock in Chrome is not evidence of anything.

How to verify a certificate chain

What follows is the working toolkit. Every command is read-only and runs from any machine with network access to the server.

openssl s_client: what the server really sends

openssl s_client -showcerts -connect example.com:443 -servername example.com </dev/null

Argument by argument:

• -showcerts — dump the PEM of every certificate the server sent, not just the leaf.
• -servername example.com — send SNI. Without it a multi-site server hands you its default certificate and you end up debugging somebody else's chain. This is diagnostic mistake number one.
• </dev/null — close stdin, otherwise the command hangs in interactive mode.

The fast version is simply to count how many certificates arrived:

openssl s_client -showcerts -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | grep -c "BEGIN CERTIFICATE"

For a typical public site the healthy answer is 2 (leaf plus one intermediate), sometimes 3. An answer of 1 means an incomplete chain in almost every case.

Reading the Certificate chain block

The s_client output starts with numbered s: (subject) and i: (issuer) lines:

Certificate chain
 0 s:CN=example.com
   i:C=US, O=Let's Encrypt, CN=R11
 1 s:C=US, O=Let's Encrypt, CN=R11
   i:C=US, O=Internet Security Research Group, CN=ISRG Root X1

There is one rule to check: the issuer of certificate N must equal the subject of certificate N+1. Above, i: of entry 0 matches s: of entry 1 — the chain is linked. The last i: (ISRG Root X1 here) is the root the server does not send and the client must already have.

Red flags:

• Only entry 0 exists — the server sends a single certificate, no intermediates.
• An i: matches no s: below it — the file was assembled from parts of different issuances.
• A certificate whose s: equals its i: — that is a self-signed root and it should not be sent.
• More than three certificates with obvious leftovers — "Extra download" in SSL Labs terms.

Verify return code, decoded

At the very end s_client prints OpenSSL's own verdict. It is the single most useful line in the output.

What the tool printed	What it means
Verify return code: 0 (ok)	A path was built and validated up to a trusted root. The chain is fine — for this machine and this CA bundle.
21 (unable to verify the first certificate)	The server sent only the leaf; there is nothing to build on. The classic incomplete chain.
20 (unable to get local issuer certificate)	The chain is present but its root is missing from the client's store. Either the CA is not public, or the bundle is stale.
2 (unable to get issuer certificate)	No issuer found for an intermediate — a link is missing in the middle of the file.
10 (certificate has expired)	Something on the path expired. Not necessarily the leaf — intermediates have dates too.
18 (self signed certificate)	The leaf is self-signed, which is a different story — see self-signed certificates.
19 (self signed certificate in certificate chain)	A root was placed in the chain and this machine does not trust it. Remove it from the file and/or add it to the client's store.
62 (hostname mismatch)	The chain is valid but the name does not match the request. Nothing to do with the chain itself.

Remember that Verify return code: 0 means "OK on this machine". If you test from a laptop running a current macOS while the failure is on an old Android, the result does not transfer. Verify from an environment that resembles the broken one.

openssl verify: check the chain locally

If the certificates already exist as files, the chain can be validated without touching the network:

# Validate the leaf against a known intermediate plus system roots
openssl verify -untrusted chain.pem cert.pem

# Validate against one specific root, ignoring the system store
openssl verify -CAfile root.pem -untrusted chain.pem cert.pem

# Print the path that was actually built
openssl verify -show_chain -untrusted chain.pem cert.pem

The distinction between the flags matters. -CAfile supplies trust anchors (roots); -untrusted supplies intermediates that may be used while building a path but are not trusted in themselves. A common mistake is putting the intermediate into -CAfile: verification passes, and the result proves nothing, because you declared the intermediate a trust anchor.

Taking a PEM bundle apart

A PEM bundle is just several BEGIN CERTIFICATE blocks in a row. To see what is inside and in which order:

# The classic approach, works on any OpenSSL version
openssl crl2pkcs7 -nocrl -certfile fullchain.pem \
  | openssl pkcs7 -print_certs -noout

# Modern alternative
openssl storeutl -noout -text -certs fullchain.pem | grep -E "Subject:|Issuer:"

# Names and dates of a single file
openssl x509 -in cert.pem -noout -subject -issuer -dates

The first command prints something like:

subject=CN=example.com
issuer=C=US, O=Let's Encrypt, CN=R11

subject=C=US, O=Let's Encrypt, CN=R11
issuer=C=US, O=Internet Security Research Group, CN=ISRG Root X1

Same linking rule: the issuer of the first equals the subject of the second. If the issuer of the second does not equal the subject of the third, a link is missing.

You can match leaf and intermediate more precisely through key identifiers. The leaf's Authority Key Identifier must equal the intermediate's Subject Key Identifier:

openssl x509 -in cert.pem   -noout -ext authorityKeyIdentifier
openssl x509 -in chain.pem  -noout -ext subjectKeyIdentifier

curl, keytool and a clean client in Docker

# curl prints the built chain and the verdict
curl -vI https://example.com 2>&1 | grep -Ei "subject:|issuer:|SSL certificate|verify"

# Point at a bundle explicitly — is it the chain or the store?
curl --cacert /etc/ssl/certs/ca-certificates.crt -sS -o /dev/null https://example.com

# What Java sees: prints the chain the server sent
keytool -printcert -sslserver example.com:443

# A clean client with no desktop caches and no corporate roots
docker run --rm alpine sh -c "apk add --no-cache ca-certificates curl >/dev/null && curl -sSI https://example.com | head -1"

The fresh-container test is the most honest one: no cached intermediates, no roots somebody added by hand years ago. If it works in the container but not in production, the problem is on the client side, not the server.

Certificate order in the file: cert.pem vs chain.pem vs fullchain.pem

Certbot writes four files into the live directory, and half of all chain incidents start with the wrong one ending up in the config:

File	Contents	Where it belongs
privkey.pem	Private key	ssl_certificate_key in nginx, SSLCertificateKeyFile in Apache
cert.pem	Leaf only	Apache before 2.4.8 (SSLCertificateFile) paired with SSLCertificateChainFile. In nginx it is nearly always a mistake
chain.pem	Intermediates only, no leaf and no root	SSLCertificateChainFile; in nginx, ssl_trusted_certificate for OCSP stapling
fullchain.pem	Leaf plus intermediates in the correct order	ssl_certificate in nginx, SSLCertificateFile in Apache 2.4.8 and newer

The ordering rule inside the file is bottom-up along the trust tree:

# fullchain.pem
-----BEGIN CERTIFICATE-----   ← 1. leaf (example.com)
-----BEGIN CERTIFICATE-----   ← 2. intermediate that signed the leaf
-----BEGIN CERTIFICATE-----   ← 3. higher intermediate, if any
#                               do NOT append the root

Assembling by hand:

cat cert.crt intermediate.crt > fullchain.crt

# Check the result BEFORE reloading the service
openssl crl2pkcs7 -nocrl -certfile fullchain.crt | openssl pkcs7 -print_certs -noout
openssl verify -untrusted fullchain.crt cert.crt

A frequent trap: commercial CAs deliver archives with names like example_com.crt, SectigoRSADomainValidationSecureServerCA.crt, USERTrustRSAAAACA.crt, AAACertificateServices.crt. Alphabetical order is not chain order. Assemble by reading the issuer and subject fields, not by filename.

What happens with a reversed order: some clients — mostly modern ones, in the spirit of RFC 8446 — will cope, others will not. GnuTLS, older OpenSSL builds and several Java stacks return an error. The worst part is that the failure is intermittent across the audience, so it takes a long time before anyone accepts it as a real problem.

Incomplete chain: symptoms and the fix

An incomplete chain is a specific but by far the most common failure: the server sends only the leaf, with no intermediates.

Symptoms

• Desktop Chrome works, Safari on iPhone does not.
• curl fails with unable to get local issuer certificate.
• Java application: PKIX path building failed.
• Python requests: [SSL: CERTIFICATE_VERIFY_FAILED].
• A webhook receiver rejects your endpoint even though the certificate is valid.
• A payment gateway or API partner reports an untrusted certificate while you "changed nothing".
• SSL Labs caps the grade at B with "Chain issues: Incomplete".
• The problem appeared right after a renewal — typical for hand-rolled deploy scripts.

Assembling a correct fullchain

With Let's Encrypt or any ACME client there is nothing to assemble — fullchain.pem is ready:

# Correct
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;

# Wrong: leaf only, no intermediate
ssl_certificate /etc/letsencrypt/live/example.com/cert.pem;

For a commercial CA, build the file by the ordering rule above and validate it with openssl verify before reloading anything.

Web servers and proxies

# nginx
ssl_certificate           /etc/nginx/certs/fullchain.pem;
ssl_certificate_key       /etc/nginx/certs/privkey.pem;
ssl_trusted_certificate   /etc/nginx/certs/chain.pem;   # for OCSP stapling
nginx -t && systemctl reload nginx

# Apache 2.4.8 and newer — everything in one file
SSLCertificateFile      /etc/apache2/certs/fullchain.pem
SSLCertificateKeyFile   /etc/apache2/certs/privkey.pem

# Apache before 2.4.8 — chain supplied separately
SSLCertificateFile      /etc/apache2/certs/cert.pem
SSLCertificateChainFile /etc/apache2/certs/chain.pem

apachectl configtest && systemctl reload apache2

Other common TLS termination points:

• HAProxy — a single PEM file containing the private key, the leaf and the intermediates one after another.
• Traefik — in the file provider, the certificate field must hold the full chain, not just the leaf.
• IIS — intermediates are imported into the machine's "Intermediate Certification Authorities" store, not attached to the site.
• Tomcat and other Java servers — the chain must live inside the keystore in full; importing only the leaf into an existing key pair leaves a one-element chain.
• Postfix, Dovecot and other mail servers — mail clients and MTAs almost never do AIA fetching, so an incomplete chain breaks them immediately and hard. Test mail ports separately.
• Cloud load balancers and CDNs — there is usually a dedicated "certificate chain" field; leaving it empty produces exactly the same failure.

When TLS terminates at a load balancer or CDN, the certificate configuration on the backend has no effect on what users see. Test the public endpoint, not the origin.

Where to get a lost intermediate

1. AIA inside your own certificate — the most reliable route, because it yields exactly the intermediate that signed your leaf:

   openssl x509 -in cert.pem -noout -ext authorityInfoAccess
   curl -sS http://<caIssuers-url> | openssl x509 -inform DER -out intermediate.pem

2. The CA's own site — for Let's Encrypt that is letsencrypt.org/certificates; commercial CAs publish equivalent repositories.
3. Certificate Transparency logs — look your certificate up on crt.sh and follow the issuer link.
4. Reissue the certificate — if the CA rotated its intermediate, the old one may no longer fit. A reissue through an ACME client takes a minute; see issuing a Let's Encrypt certificate.

Error triage: symptom, cause, check, fix

Symptom	Cause	How to check	What to do
unable to get local issuer certificate, curl exit code 60	The server did not send the intermediate and the client bundle holds roots only	openssl s_client -showcerts … | grep -c "BEGIN CERT" returns 1	Build a fullchain and reload the service
PKIX path building failed in Java	The same problem; Java does not chase AIA	keytool -printcert -sslserver host:443	Fix the chain on the server. Importing a root into cacerts is a workaround, not a fix
SEC_ERROR_UNKNOWN_ISSUER in Firefox only	Firefox does not chase AIA and has never seen your intermediate	Open the same URL in a private window of another browser	Add the intermediate to the server chain
x509: certificate signed by unknown authority in a Go service	Incomplete chain, or a non-public root	openssl verify -CAfile with the CA root	If the root is private, distribute it to clients explicitly instead of disabling verification
Broke right after renewal	The deploy script copies cert.pem instead of fullchain.pem, or the CA rotated its intermediate	Diff the s_client output before and after; look at the intermediate's CN	Fix the path in the deploy script and add a post-renewal check
Fails only on old devices	The CA root is absent from the firmware and no cross-signed path is available	Check which roots the device knows; review the CA's alternate chains	Pick a CA or chain with broader compatibility; for a critical audience, a separate certificate
SSL Labs: "Chain issues: Incorrect order"	Certificates in the file are out of order	openssl crl2pkcs7 … | openssl pkcs7 -print_certs -noout	Rebuild the file: leaf first, then upwards
SSL Labs: "Chain issues: Contains anchor"	The root was included in the chain	Same output: the last certificate is self-signed	Drop the root from the file — harmless but wasteful
Verify return code: 10 while the leaf is fresh	An intermediate expired	openssl x509 -in chain.pem -noout -dates	Fetch the current intermediate from the CA or reissue
Browser warns about an untrusted authority	A separate topic: root missing from the store, or traffic interception	Compare the issuer shown in the browser with s_client	See ERR_CERT_AUTHORITY_INVALID
"Cannot verify server certificate" in a mail or FTP client	Usually the same incomplete chain on a non-HTTPS port	openssl s_client -connect host:993 -servername host	See cannot verify server certificate

What a correct chain does NOT fix

Knowing the boundary saves time. The chain only governs path building. It will not help when:

• The leaf expired. The path builds and the date check still fails — see expired certificate.
• The name does not match. A certificate for example.com will not serve api.example.com unless that name is in the SAN. Coverage differences are covered in types of SSL certificates.
• The certificate was revoked. Revocation is checked separately, through OCSP or CRL.
• The certificate is self-signed. There is no chain at all — see self-signed certificates.
• The client has no root for that CA. No amount of correct file assembly makes a client trust an unknown anchor.
• TLS versions or cipher suites do not overlap. The connection never reaches the certificate exchange — see the TLS handshake explained.
• The client's clock is wrong. A device set to the wrong date will call a valid certificate expired or not yet valid.
• SNI is not configured. The server hands out its default certificate and you end up fixing the chain of the wrong site.

Checking a chain with enterno.io

When you would rather not assemble openssl output by hand, or need something to show a colleague:

• SSL checker — inspects the certificate from an external environment, free of your local caches and corporate roots, and renders the chain as a tree: what signed what, and where the break is. It is the fastest answer to "what does my server actually send".
• SSL error reference — decoding of codes and messages when you have an error and cannot tell which layer it belongs to.
• Security scanner — looks at HTTPS and headers as a whole rather than the certificate alone, useful when "something is wrong with the site" but nobody knows what.
• Monitoring — recurring certificate and chain checks with alerts. Which parameters are worth watching is covered in SSL certificate monitoring.
• To dissect the certificate itself — fields, SAN, dates, issuer — see how to check a site's SSL certificate.

Automation: stop the chain from breaking again

Chains rarely break at setup time. They break months later — at renewal, during a migration, or when the CA rotates an intermediate. A one-off fix without a recurring check is worthless.

• A post-renewal hook check. ACME clients can run a command after a successful renewal. Put a certificate count and an openssl verify in there — a failing hook is far more visible than a quietly broken site.
• A CI check. Run openssl verify -untrusted chain.pem cert.pem before shipping a config. It costs a second and catches an entire class of incidents.
• A smoke test from a clean container. After deploy, hit the site from a fresh image without -k and without added roots.
• External monitoring. Checking from inside your own network lies: corporate roots and inspection proxies live there. You need an outside view.
• An inventory of termination points. Teams remember the main domain and forget mail, the API subdomain, the webhook endpoint and the legacy balancer. The forgotten one is what breaks.

#!/bin/sh
# Minimal post-renewal check. A non-zero exit is the signal.
HOST="example.com"
N=$(openssl s_client -showcerts -connect "$HOST:443" -servername "$HOST" </dev/null 2>/dev/null \
    | grep -c "BEGIN CERTIFICATE")
if [ "$N" -lt 2 ]; then
  echo "chain too short for $HOST: $N cert(s)" >&2
  exit 1
fi
openssl s_client -connect "$HOST:443" -servername "$HOST" </dev/null 2>/dev/null \
  | grep -q "Verify return code: 0" || exit 1

Do not turn verification off and call it a fix. -k in curl, verify=False in Python, rejectUnauthorized in Node and "just import the root into cacerts" all silence the symptom and remove your protection against traffic interception exactly where you need it.

Frequently asked questions

Should the root certificate be included in fullchain.pem?

No. The client already holds the root in its store and will not trust a shipped copy on its own merit. An extra root only inflates the handshake. The one exception is a closed environment where you distribute the root yourself and know exactly why.

How many intermediates should a chain contain?

Usually one. Sometimes two, when the CA uses a second-level intermediate or a cross-sign. Zero on a public site means a self-signed certificate. More than three is a reason to check whether leftovers from a previous issuance ended up in the file.

Why does Chrome open the site while Python requests fails?

Chrome can fetch the intermediate over AIA and caches certificates it has seen before. Python requests uses the certifi bundle, which contains roots only, and never goes looking. The browser masks the broken chain; the script does not. Fix the server, not the script.

Can a chain break on its own, with no server changes?

Yes, and it is not rare. An intermediate expires, the CA introduces a new intermediate, a root that a cross-signed path relied on expires, or a client updates its CA bundle and drops an old root. The server was never touched. That is exactly why external monitoring beats "we checked it once during setup".

What does "unable to get local issuer certificate" actually mean?

Literally: the client could not find an issuer certificate for the certificate it stopped at. In practice there are two distinct cases. If the server sent one certificate, the intermediate is missing on the server. If it sent two or more, the root is missing from the client's store. Counting the certificates the server sent tells them apart.

Does the certificate order in the file really matter?

Yes. The leaf must come first — that is a protocol requirement in every version of TLS. Modern clients usually untangle the rest, but older stacks and several libraries do not. Assembling in the right order is cheaper than later hunting down which partner "sometimes" fails.

What is the fastest way to check the chain from a mobile device?

Checking from the device itself is awkward and unreliable, because the browser may use its cache and AIA. An external service that looks at the server from a clean environment is faster and more trustworthy. The enterno.io SSL checker shows exactly what the server sends, without your local roots.

Checklist

• The server sends at least two certificates, leaf plus intermediate — confirmed with grep -c "BEGIN CERTIFICATE".
• The leaf is the first certificate in the file.
• Each certificate's issuer equals the next certificate's subject.
• The root is not included in the file.
• Verify return code: 0 (ok) from a clean machine, not only from your work laptop.
• The config points at fullchain.pem, not cert.pem.
• Every check was made with -servername, that is, with correct SNI.
• All TLS termination points were checked: apex domain, subdomains, API, mail, webhooks, balancer and CDN.
• Verification was done with the class of client that actually connects: curl, Java, the mobile app.
• Validity dates were checked for every element of the chain, not just the leaf.
• A chain check runs in the renewal hook.
• External monitoring watches both the chain and the expiry date.
• No -k, verify=False or other verification bypasses remain in production.

Certification path building and validation — RFC 5280, §6. The TLS 1.3 Certificate message format — RFC 8446, §4.4.2. Current Let's Encrypt chains — letsencrypt.org/certificates.