HTTP headers explained: request, response, caching and proxy headers

In short. HTTP headers are name-value pairs that a client and a server exchange before the message body. They fall into four groups: request headers, response headers, general headers and representation headers. Headers control caching (Cache-Control, ETag), content type and charset (Content-Type), compression (Content-Encoding, Vary), redirects (Location) and security. You can inspect them with curl -sI, in the DevTools Network tab, or with an online HTTP header check.

How an HTTP message is built and where headers live

Every HTTP message has three parts: a start line, a header block and a body. A request start line carries the method, the path and the protocol version; a response start line carries the version, a numeric status code and a reason phrase. Headers follow, one per line, then an empty line, then the body. In HTTP/1.1 the line separator is a CR+LF pair, not a bare newline — a server that emits \n alone breaks strict clients and proxies.

GET /api/data HTTP/1.1
Host: example.com
Accept: application/json
Accept-Encoding: gzip, br
Authorization: Bearer eyJhbGci...
User-Agent: Mozilla/5.0
If-None-Match: "9f2c-6413ab21"

The server replies with its own header set. The blank line after them marks where the body begins:

HTTP/1.1 200 OK
Date: Mon, 03 Feb 2026 09:14:22 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 3842
Content-Encoding: gzip
Vary: Accept-Encoding
Cache-Control: private, max-age=0, no-cache
ETag: "9f2c-6413ab21"
X-Request-Id: 4f1a9c2e-7b30-4d55-9a11-2c8f0e6d1b47

{"items":[...]}

Header names in HTTP/1.1 are case-insensitive: content-type, Content-Type and CONTENT-TYPE are the same field. Order does not matter. If a header appears more than once, the recipient may join the values with commas — with one practical exception, Set-Cookie, which is always treated as a list of separate lines.

What changes in HTTP/2 and HTTP/3

There are no textual headers on the wire. HTTP/2 compresses them with HPACK, HTTP/3 with QPACK; what travels is binary table indexes, not strings. Names must be lowercase — a library will downcase Content-Type for you, but a backend that compares field names byte for byte will break. The start line is replaced by pseudo-headers: :method, :scheme, :authority, :path in requests and :status in responses. The familiar Host becomes :authority, so an application that reads only Host may see an empty string behind a proxy. Transfer-Encoding is forbidden in HTTP/2 and HTTP/3: framing is the protocol's job.

Headers are not free. By default nginx allocates roughly 8 KB for the whole header block (client_header_buffer_size plus large_client_header_buffers). An overgrown cookie jar or a huge Authorization value blows past that limit, and instead of reaching the application the client gets 400 Bad Request or 431 Request Header Fields Too Large — with nothing at all in the application log, because the request never got there.

Four groups: request, response, general and representation headers

The RFC 9110 classification is more useful than the usual request/response split, because some headers travel in both directions.

• Request headers describe the client and what it wants back: Host, User-Agent, Accept, Accept-Language, Accept-Encoding, Authorization, Cookie, Referer, Origin, Range, and the conditional If-None-Match and If-Modified-Since.
• Response headers describe the server and how to handle the answer: Server, Set-Cookie, Location, ETag, Vary, Retry-After, Accept-Ranges, Age, WWW-Authenticate.
• Representation headers describe the body itself, regardless of who sent it: Content-Type, Content-Length, Content-Encoding, Content-Language, Content-Location. In a POST they describe the request body, in a response the response body. These are exactly the fields a cache must store alongside the resource.
• General headers apply to the message as such: Date, Cache-Control, Connection, Via, Trailer, Transfer-Encoding. Cache-Control is the one that meaningfully travels both ways: in a response it sets storage rules, in a request the client uses it to bypass caches.

A second axis is end-to-end versus hop-by-hop. Most headers are end-to-end: a proxy must pass them along unchanged. But Connection, Transfer-Encoding, TE, Upgrade and Proxy-Authorization apply to a single link only and must not cross a proxy. Hence a classic confusion: Content-Encoding: gzip is a property of the resource and reaches the browser, while Transfer-Encoding: chunked exists only between two adjacent nodes and may disappear on the next hop.

Common request headers

Header	Purpose	Example
Host	Virtual host name. Mandatory in HTTP/1.1; in HTTP/2 and HTTP/3 the :authority pseudo-header takes its place	Host: example.com
Accept	Acceptable media types with q weights	Accept: application/json;q=1.0, text/html;q=0.8
Accept-Language	Preferred response languages	Accept-Language: en-GB,en;q=0.9,de;q=0.6
Accept-Encoding	Supported compression algorithms	Accept-Encoding: gzip, deflate, br, zstd
Authorization	Credentials. By default it makes the response non-cacheable in shared caches	Authorization: Bearer eyJhbGci...
Cookie	Every cookie matching the domain and path, on one line	Cookie: sid=abc; theme=dark
User-Agent	Client and platform. Not a security control: one curl flag forges it	User-Agent: Mozilla/5.0 (X11; Linux x86_64)
Referer	Where the request came from. How much is sent is governed by the response Referrer-Policy	Referer: https://example.com/catalog
Origin	Origin of a cross-site request, the basis of CORS. Scheme, host and port only — no path	Origin: https://app.example.com
Range	Requests a slice of the resource; the answer is 206	Range: bytes=0-1048575
If-None-Match	Conditional request by ETag; on a match the server returns 304	If-None-Match: "9f2c-6413ab21"
If-Modified-Since	Conditional request by date. Ignored when If-None-Match is also present	If-Modified-Since: Mon, 03 Feb 2026 09:00:00 GMT

Common server response headers: a reference table

This table is built for one practical job: you look at a server response and immediately see what is missing and what that costs you.

Header	Purpose	Typical value	What its absence costs
Content-Type	Media type and charset of the body	text/html; charset=utf-8	The browser guesses the type: garbled text, downloads instead of rendering, XSS risk via uploaded files
Content-Length	Body size in bytes after compression	Content-Length: 3842	No download progress, no byte ranges; the response falls back to chunked
Content-Encoding	Which algorithm compressed the body	Content-Encoding: br	Bandwidth and response time multiply on HTML, CSS, JS, JSON and SVG
Cache-Control	Storage rules for the response	public, max-age=31536000, immutable	Caches fall back to heuristics: static assets are refetched for nothing while HTML sticks for hours
ETag	Fingerprint of a resource version	ETag: "9f2c-6413ab21"	No cheap validation: instead of a 304 the server ships the whole body again
Last-Modified	Time the resource last changed	Mon, 03 Feb 2026 09:00:00 GMT	No fallback validator when ETag is not emitted
Vary	Which request headers change the response	Vary: Accept-Encoding	A CDN hands a compressed body to a client that never asked for compression — broken pages for part of your audience
Date	When the response was generated	Mon, 03 Feb 2026 09:14:22 GMT	Caches have no baseline for computing response age
Age	Seconds the response has spent in a shared cache	Age: 118	You cannot tell a fresh origin response from a stale CDN copy
Location	Redirect target for 3xx, or new resource for 201	Location: https://example.com/new	A redirect without a target is a dead end; the browser shows a blank page
Set-Cookie	Sets a cookie on the client	sid=abc; Path=/; Secure; HttpOnly; SameSite=Lax	No sessions; without the flags, cookie theft via XSS and leakage over plain HTTP
Accept-Ranges	Byte range support	Accept-Ranges: bytes	Video will not seek, downloads will not resume after a drop
Retry-After	When to retry after a 429 or 503	Retry-After: 120	Clients and crawlers hammer the server with retries and deepen the outage
Link	Related resources: preload, canonical, alternate	</app.css>; rel=preload; as=style	You lose preloading and canonicalisation for non-HTML files
Strict-Transport-Security	Forces HTTPS for the domain	max-age=31536000; includeSubDomains	The first plain-HTTP visit stays open to interception and tampering
X-Content-Type-Options	Disables MIME sniffing	nosniff	A user-uploaded file can be executed as a script
Content-Security-Policy	Where resources may be loaded from	default-src 'self'	Any XSS immediately becomes third-party code execution
Server	Web server name and version	Server: nginx	Its absence costs nothing; it is the exact version being present that hurts

The rule is simple: if a header affects how the client interprets the body, state it explicitly. Anything you leave unsaid will be guessed for you by the browser, the proxy and the CDN — and all three will guess differently.

Cache-Control: every directive that matters

Cache-Control is the primary caching header, defined in RFC 9111. It takes a comma-separated list of directives and addresses two different kinds of cache: the private cache (one user's browser) and shared caches — CDNs, reverse proxies, corporate caches. Half of all caching bugs come from a directive aimed at the wrong one.

max-age and s-maxage

max-age=N sets the freshness lifetime in seconds — how long a response may be served from cache without contacting the server. The countdown does not start when the client receives the response; it starts from Date, adjusted by Age. A response that sat in a CDN for 3000 seconds with max-age=3600 has 600 seconds left in the browser, not 3600.

s-maxage=N is the same thing for shared caches only, and it overrides max-age. That pairing is the workhorse for HTML: forbid browser caching, let the CDN hold a copy for a minute and absorb the load.

Cache-Control: public, max-age=0, s-maxage=60, stale-while-revalidate=600

no-cache versus no-store — the most common mix-up

These two sound alike and mean opposite things.

• no-cache — storing is allowed, but before every reuse the cache must ask the server whether the copy is still valid. In practice: "cache it, but always revalidate." The response physically sits on the user's disk.
• no-store — storing is forbidden entirely: not on disk, not in memory, not in an intermediary. This is the directive for pages with personal data, bank statements and API responses containing someone's profile.

Hence the classic failure: an account page is marked no-cache on the assumption that "there is no cache", yet it stays in the browser's disk cache and comes back with the Back button after the user logs out on a shared computer. For private content, only no-store will do.

The opposite extreme is the cargo cult of no-cache, no-store, must-revalidate. A single no-store is enough: if nothing may be stored, there is nothing to revalidate. The extra directives break nothing, but they advertise that no strategy was thought through.

private and public

private means the response is meant for a single user, so shared caches must not store it while the browser may. It is not a security control — private does not stop the response from being written to disk. public solves the opposite problem: it allows a shared cache to store a response it would otherwise refuse, for example when the request carried an Authorization header.

must-revalidate, immutable, stale-while-revalidate

must-revalidate is often described as "revalidate once max-age expires", which is inaccurate — a cache must do that anyway. The real meaning is narrower: must-revalidate forbids serving a stale copy when the server is unreachable. Without it a cache may hand out stale content during an outage; with it, the cache must return 504 instead. Use it where stale data is worse than an error: stock levels, balances, prices.

immutable states that the body will not change during the freshness lifetime. On its own the directive does nothing — it only works paired with a long max-age. Its effect is narrow but valuable: the browser will skip the conditional request even on a normal page reload. Apply it only to files whose version is baked into the filename.

stale-while-revalidate=N (RFC 5861) allows a cache to serve the stale copy instantly for N seconds past expiry while refreshing it in the background. It is the best compromise for HTML and feeds behind a CDN: this visitor does not wait for the backend, and the next one gets fresh content. Its sibling stale-if-error=N permits serving a stale copy when the origin returns 5xx — cheap insurance against a short outage.

Less common but worth knowing: proxy-revalidate (like must-revalidate, shared caches only), no-transform (forbids proxies from recompressing images or minifying text — still relevant on mobile networks), plus the request-side only-if-cached and max-age=0.

Why Expires is legacy

Expires comes from HTTP/1.0 and carries an absolute GMT expiry date. It has two built-in flaws. First, it depends on clocks: if the server or the client clock drifts, the lifetime drifts with it. Second, it cannot express anything beyond "good until" — no s-maxage, no stale-while-revalidate, no immutable. When both headers are present, Cache-Control: max-age wins and Expires is ignored. Emitting it separately only makes sense for very old clients; in nginx the expires directive sets both headers anyway, so there is nothing extra to configure.

The "flush the cache" trick using Expires: 0 or a past date does work, but bluntly: it marks the response stale, it does not forbid storage. For confidential data it is no substitute for no-store.

ETag, Last-Modified and conditional requests: where 304 comes from

Freshness answers the question "may I serve this from cache without asking?". Validation answers a different one: "is my copy still current?". Validation is cheap — only the header block crosses the network.

ETag: strong and weak

An ETag is an opaque quoted string, a fingerprint of one specific version. A strong validator such as "9f2c-6413ab21" means byte-for-byte identity. A weak one, W/"9f2c", means semantic equivalence: the body may differ in trivia but counts as the same. The distinction is not cosmetic — byte ranges via If-Range only work with a strong validator, otherwise a client risks stitching together chunks of two different versions.

Two practical gotchas. First, nginx turns a strong ETag into a weak one when it compresses on the fly, prefixing it with W/, because the compressed body is literally different bytes. That is correct behaviour and only trips up hand-rolled clients doing literal string comparison. Second, Apache historically built ETag from the file inode, size and mtime. Across a cluster the inode of the same file differs per server, so the cache misses every time the load balancer switches backends. The fix is FileETag MTime Size.

Last-Modified and one-second granularity

Last-Modified is an IMF-fixdate value, always in GMT: Mon, 03 Feb 2026 09:00:00 GMT. Its granularity is one second, so it is useless as a validator for resources that change more often than that. Treat it as a fallback: if the server emits both ETag and Last-Modified and the client sends both conditions, If-None-Match takes precedence and If-Modified-Since is simply ignored.

What a conditional request and a 304 look like

# plain request — grab the ETag
curl -sD - -o /dev/null https://example.com/assets/app.css | grep -i -E 'etag|last-modified|cache-control'
# ETag: "9f2c-6413ab21"
# Last-Modified: Mon, 03 Feb 2026 09:00:00 GMT
# Cache-Control: public, max-age=31536000, immutable

# conditional request by ETag — expect 304 and an empty body
curl -sD - -o /dev/null -w 'body=%{size_download}\n' \
     -H 'If-None-Match: "9f2c-6413ab21"' \
     https://example.com/assets/app.css
# HTTP/2 304
# body=0

# conditional request by date
curl -sI -H 'If-Modified-Since: Mon, 03 Feb 2026 09:00:00 GMT' \
     https://example.com/assets/app.css | head -1
# HTTP/2 304

A 304 Not Modified response is deliberately truncated: there is no body, and representation headers such as Content-Type and Content-Length are normally absent too. This is not a broken configuration. The server only has to send what could have changed: Date, ETag, Cache-Control, Expires, Vary. If you are auditing a header set and half of it seems to have vanished, check the status line first.

A side effect: auditing security headers against a 304 shows false negatives. Force a 200 — for example by adding -H 'Cache-Control: no-cache' to the request. Validation is covered in depth in the article on cache-control headers, and site-level planning in the guide to web caching strategies.

Caching recipes by resource type

There is no universal Cache-Control. The breakdown below covers nearly any site.

Resource type	Recommended Cache-Control	Why
Hashed static assets (app.4f1a9c.js)	public, max-age=31536000, immutable	The filename changes with the content, so expiry is unnecessary; immutable removes pointless conditional requests on reload
Unhashed static assets (logo.png, style.css)	public, max-age=86400, must-revalidate plus ETag	The file can be overwritten in place, so it needs a short lifetime and mandatory validation
Fonts (.woff2)	public, max-age=31536000, immutable	Fonts are effectively immutable and render-blocking. They must also carry a CORS header, since fonts are fetched as cross-origin resources
HTML of a public page	public, max-age=0, s-maxage=60, stale-while-revalidate=600	The browser always revalidates, the CDN holds a copy and absorbs load, and nobody waits for the backend during a background refresh
HTML of an authenticated area	private, no-store	Personal data must reach neither a shared cache nor the browser disk cache on a shared computer
Public JSON API (reference data, rates)	public, max-age=60, stale-if-error=86400	The data changes slowly; during an origin outage the client gets yesterday's data instead of an error
Private JSON API (profile, orders)	private, no-store	The response is unique per user; a caching mistake means showing someone else's data
Login and token refresh responses	no-store plus Pragma: no-cache	Tokens and session identifiers must not persist anywhere
User uploads (avatars, documents)	private, max-age=3600 plus Content-Disposition	The file belongs to one user; a shared cache must not serve it from a guessable URL
robots.txt, sitemap.xml, llms.txt	public, max-age=3600	Crawlers reread them often; a long cache delays your changes by a day or more
404 and 410 responses	public, max-age=300	A short cache damps hammering on missing URLs without blocking a quick restore
5xx responses	no-store	A cached error outlives the fix and turns a five-minute incident into an hour-long one

Here is the same breakdown as nginx configuration. Note that HTML and static assets live in separate location blocks, and every header carries always so it also lands on error responses.

server {
    listen 443 ssl;
    http2 on;
    server_name example.com;
    root /var/www/example/public;

    gzip on;
    gzip_vary on;
    gzip_types text/css text/javascript application/javascript application/json image/svg+xml;
    gzip_min_length 1024;

    etag on;

    # 1. Hashed assets — cache forever
    location ~* "^/assets/.+\.[0-9a-f]{8,}\.(css|js|woff2|png|jpg|webp|svg)$" {
        add_header Cache-Control "public, max-age=31536000, immutable" always;
        add_header X-Content-Type-Options "nosniff" always;
        access_log off;
        try_files $uri =404;
    }

    # 2. Fonts — cache forever plus CORS, otherwise the browser rejects them
    location ~* "\.(woff2|woff|ttf)$" {
        add_header Cache-Control "public, max-age=31536000, immutable" always;
        add_header Access-Control-Allow-Origin "*" always;
        add_header X-Content-Type-Options "nosniff" always;
    }

    # 3. Other unhashed static — one day plus mandatory validation
    location ~* "\.(css|js|png|jpg|jpeg|gif|webp|ico|svg)$" {
        add_header Cache-Control "public, max-age=86400, must-revalidate" always;
        add_header X-Content-Type-Options "nosniff" always;
    }

    # 4. HTML — no browser cache, one minute at the CDN
    location / {
        add_header Cache-Control "public, max-age=0, s-maxage=60, stale-while-revalidate=600" always;
        add_header X-Content-Type-Options "nosniff" always;
        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
        try_files $uri $uri/ /index.php?$query_string;
    }

    # 5. Account area and private API — store nowhere
    location ^~ /account/ {
        add_header Cache-Control "private, no-store" always;
        add_header X-Content-Type-Options "nosniff" always;
        try_files $uri /index.php?$query_string;
    }
}

Always validate before reloading: nginx -t && systemctl reload nginx. Syntax, contexts and location matching order are covered in the separate nginx configuration guide.

Content negotiation: Accept, Accept-Encoding, Content-Encoding and Vary

One URL can serve several representations: compressed and uncompressed, English and German, JSON and HTML. The client states preferences through the Accept* headers, the server picks a variant — and must declare what it picked on.

q weights and how they are read

Accept, Accept-Language and Accept-Encoding take weighted lists: Accept-Language: en-GB,en;q=0.9,de;q=0.6 means "British English ideally, then any English, German only if you must". The default weight is 1.0 and q=0 is an explicit refusal. In practice many servers ignore the weights and take the first acceptable match — if language variants matter to you, verify that behaviour separately.

Compression: Accept-Encoding and Content-Encoding

The browser sends Accept-Encoding: gzip, deflate, br, zstd, the server answers with Content-Encoding: br and a compressed body. The practical hierarchy: brotli is 15–20% smaller than gzip on text, zstd matches brotli on size while compressing faster, and gzip remains the universal fallback. Browsers only offer brotli and zstd over HTTPS. Compress text types — HTML, CSS, JS, JSON, XML, SVG. Recompressing JPEG, PNG, WebP, video and archives is pointless: no gain, wasted CPU.

Do not confuse Content-Encoding with Transfer-Encoding: the first describes the resource and reaches the client, the second describes transfer between two adjacent nodes.

Vary — the forgotten header that costs the most

Vary lists the request headers the response depends on. For a cache it is part of the key: with Vary: Accept-Encoding, compressed and uncompressed variants are stored separately.

A missing Vary: Accept-Encoding behind a CDN produces a classic outage. The first visitor arrives with Accept-Encoding: gzip, the CDN stores the compressed body and serves it to the next client — an old mobile browser, a curl script, a payment gateway — that never asked for compression and will not decompress it. That client receives binary garbage. The symptom is nasty: it works for the majority, breaks for a minority, and is nearly impossible to reproduce in a browser.

The opposite mistake is a Vary that is too broad. Vary: User-Agent means a separate copy per User-Agent string, and there are tens of thousands in the wild: the hit ratio collapses to near zero and the CDN becomes an expensive proxy. Vary: Cookie on a site where analytics sets a cookie makes every resource unique per visitor. Vary: * means "never cacheable".

Keep in Vary exactly the headers the response really differs on. Typically that is Accept-Encoding, sometimes plus Accept-Language or Origin — the last one is mandatory if you reflect the request Origin into Access-Control-Allow-Origin. Without Vary: Origin a CDN will hand another site a header issued for yours. The mechanics are covered in the article on CORS.

Content-Type, charset and body framing

What happens without Content-Type

Content-Type consists of a MIME type plus parameters, chiefly charset. The full form for HTML is text/html; charset=utf-8. The HTTP header outranks the meta charset tag inside the document: if the server says charset=iso-8859-1 while the HTML claims utf-8, the server wins and non-ASCII text turns into mojibake. The reverse happens too: the server says nothing, the browser guesses from the first bytes and gets it wrong on a short page.

With no Content-Type at all, MIME sniffing kicks in: the browser inspects the content and decides for itself. That is convenient right up to the moment a user uploads an "image" that actually contains HTML with a script — the browser dutifully recognises HTML and runs someone else's code on your origin. This is exactly why X-Content-Type-Options: nosniff is considered mandatory.

nosniff has a flip side that people discover during a deploy: if the server serves a JS file as text/plain, a browser with nosniff refuses to execute it and the page simply stops working. The symptom is a console error about a MIME type mismatch. The fix is a correct types map in nginx, not removing the header.

Two details worth remembering: JSON is always UTF-8 and the charset parameter is not defined for application/json, so it is redundant; and forcing a download is the job of Content-Disposition: attachment; filename="report.pdf", not of mislabelling the type as application/octet-stream.

Content-Length versus Transfer-Encoding: chunked

The recipient must know where the body ends. HTTP/1.1 offers exactly two ways: declare the size up front with Content-Length, or stream the body in pieces with Transfer-Encoding: chunked, terminated by a zero-length chunk.

Content-Length is the size after compression — the number of bytes actually on the wire. It gives the client a progress bar and enables byte-range requests. But it cannot be set until the response is fully built, so streamed responses — CSV generated on the fly, server-sent events, long reports — go out chunked. The side effect is that a chunked response has no Content-Length, hence no progress bar and no seeking.

Having both headers in one message is not merely untidy, it is a known attack vector: a front end and a back end may disagree on where the request ends, letting an attacker smuggle a second, hidden request through — HTTP request smuggling. The specification therefore requires that Transfer-Encoding overrides Content-Length and that intermediaries reject such messages. HTTP/2 and HTTP/3 remove the problem structurally: Transfer-Encoding is forbidden there, framing is done by protocol frames, and Content-Length is merely informational.

Security headers: the short map

Security headers are a large topic of their own, and there is no point duplicating it here. Keep the minimum in mind and follow the links for detail.

• Strict-Transport-Security — enforces HTTPS for the max-age period. Covered in the HSTS guide.
• Content-Security-Policy — an allowlist of sources for scripts, styles, images and frames. A working policy is easiest to assemble with the CSP builder.
• X-Content-Type-Options: nosniff — disables MIME type guessing.
• X-Frame-Options, or the CSP frame-ancestors directive — clickjacking protection. CSP is the modern way.
• Referrer-Policy — how much of the URL leaks in Referer to third parties. A sane default is strict-origin-when-cross-origin.
• Permissions-Policy — which browser APIs the page and its frames may use.
• Cross-Origin-Opener-Policy and Cross-Origin-Resource-Policy — window and resource isolation from third-party documents.

The full breakdown with values, rollout order and common mistakes is in the article on security headers; a quick assessment of your current state comes from the website security check.

X- headers: the prefix is deprecated, X-Forwarded-For is not

The X- prefix once meant "experimental extension". RFC 6648 declared the practice harmful back in 2012: experiments have a habit of sticking, and renaming an established header breaks everyone who depends on it. The recommendation since then is to name a header from day one the way it will always be named, without a prefix.

In practice several X- headers became de facto standards, and nobody intends to drop them.

• X-Forwarded-For — the chain of client and proxy addresses: X-Forwarded-For: 203.0.113.7, 198.51.100.10. The leftmost address is the original client; each proxy appends on the right.
• X-Real-IP — a non-standard simplification: a single address instead of a chain, set manually in nginx.
• X-Forwarded-Proto — the scheme of the original request, http or https. Without it an application behind a TLS-terminating proxy believes the connection is insecure and sends the user into an infinite redirect loop.
• X-Forwarded-Host — the original Host value before the proxy rewrote it.
• X-Request-Id — an end-to-end request identifier, generated at the edge, propagated to every service and written to every log. Without it, incident analysis in a distributed system degenerates into guessing by timestamp.
• X-Robots-Tag — indexing directives at the HTTP level, the only way to keep a PDF, an image or an export file out of search results.

The standard replacement for the first group is the Forwarded header from RFC 7239, with the syntax Forwarded: for=203.0.113.7; proto=https; host=example.com. It is more expressive and can obfuscate internal addresses, but software support is still patchy, so real-world configurations usually carry both.

Why X-Forwarded-For cannot be trusted as-is

It is an ordinary request header: any client can send it with any content. An application that takes the leftmost value and calls it the client IP is forged with a single line — and with it you forge past rate limits, geo logic, ban lists and audit trails.

The rule: trust only the value appended by your own proxy, and only when the request arrived from an address you know. In nginx that is a pair of directive groups.

# trust only our own proxies and CDN subnets
set_real_ip_from 10.0.0.0/8;
set_real_ip_from 172.16.0.0/12;
real_ip_header   X-Forwarded-For;
real_ip_recursive on;

# and rewrite headers explicitly when proxying to the app
location /api/ {
    proxy_pass http://127.0.0.1:8080;
    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Request-Id      $request_id;
}

The key part: proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for appends the peer address to the existing chain, while set_real_ip_from makes nginx ignore values that did not come from a trusted hop. Spoofing and defences are detailed in the article on X-Forwarded-For.

Server and X-Powered-By: a free hint for an attacker

An exact version in Server: nginx/1.24.0 or X-Powered-By: PHP/8.1.2 is not a vulnerability by itself, but it saves a scanner time: no need to fingerprint the stack, just consult the list of known issues for that version. Removing it is cheap: server_tokens off; hides the nginx version (dropping the name entirely requires the headers_more module), expose_php = Off in php.ini removes X-Powered-By, and ServerTokens Prod with ServerSignature Off does the same for Apache. Do not forget framework headers such as X-AspNet-Version, X-Generator or X-Drupal-Cache — those usually need separate handling.

Headers people forget: Link, Retry-After, Location, Set-Cookie, Range

Link: preload and canonical in a header

Link, defined in RFC 8288, moves resource relationships to the HTTP layer. Two workhorse uses. First, preloading critical resources: the browser learns about a font or a stylesheet before it parses the HTML, which measurably improves LCP. Second, canonicalising non-HTML: a PDF, an image or an XML file has nowhere to put a link rel="canonical" tag, so the header is the only option.

# nginx: canonical and no-index for export files
location ^~ /files/ {
    add_header X-Robots-Tag "noindex, nofollow" always;
    add_header Cache-Control "public, max-age=3600" always;
}

location = /docs/price.pdf {
    add_header Link '<https://example.com/price>; rel="canonical"' always;
    add_header X-Robots-Tag "noarchive" always;
}

# preload the main font and stylesheet for HTML pages
location / {
    add_header Link '</assets/inter.woff2>; rel=preload; as=font; type="font/woff2"; crossorigin' always;
    add_header Cache-Control "public, max-age=0, s-maxage=60" always;
}

Verifying that X-Robots-Tag actually arrived takes one command: curl -sI https://example.com/files/report.xlsx | grep -i robots.

Retry-After on 429 and 503

Retry-After tells the client when retrying makes sense. The value is either a number of seconds (Retry-After: 120) or an HTTP date. It belongs on 429 Too Many Requests, on 503 Service Unavailable during planned maintenance, and on 301 when a move is temporary. Without it, clients and crawlers retry on their own schedule — usually an aggressive one — and finish off a server that was already struggling. Search crawlers read 503 plus Retry-After as "come back later" rather than "this page is dead", which protects indexing during maintenance.

Location and redirects

Location is mandatory on every 3xx and on 201 Created. Common mistakes: non-ASCII targets without percent-encoding, query parameters dropped during the redirect, chains of three or more hops, and — the nastiest — a redirect to plain HTTP inside an HTTPS site, which defeats HSTS for that hop. Loops are their own category: example.com points to www.example.com and back. Inspect the whole chain with curl -sIL or the redirect checker; what each code means is covered in the article on HTTP status codes.

Set-Cookie

This deserves its own article; here is the minimum. Each cookie needs its own Set-Cookie header — they must not be joined with commas. Mandatory attributes for session cookies: Secure (HTTPS only), HttpOnly (invisible to JavaScript, which kills theft via XSS), SameSite=Lax or Strict (CSRF protection), plus an explicit Path and lifetime. A practical nuance: the presence of a cookie in a request almost always makes the response uncacheable at a CDN, which is why static assets are served from a host or path that cookies never reach. To see what a site actually sets and with which flags, use the cookie checker.

Range and Accept-Ranges

Accept-Ranges: bytes in a response means the server can serve slices of a file. The client asks with Range: bytes=0-1048575 and gets 206 Partial Content along with Content-Range: bytes 0-1048575/52428800. Video seeking, resuming after a dropped connection and parallel downloads of large files all rely on this.

curl -sD - -o /dev/null -r 0-1023 https://example.com/video/demo.mp4
# HTTP/2 206
# accept-ranges: bytes
# content-range: bytes 0-1023/52428800
# content-length: 1024

A typical breakage: nginx compressing a response on the fly disables range support, because the body length is not known in advance. Compression is useless for video and large archives anyway, so just keep those types out of gzip_types. The second source of trouble is Accept-Ranges: none emitted by an intermediary proxy or a hand-written download handler: the file opens, but the player will not seek.

Headers behind a CDN and a reverse proxy

As soon as a CDN or a reverse proxy sits in front of your site, the response the browser sees is no longer the response your application produced. Some headers were added, some rewritten, some removed.

What an intermediary adds

• Age — seconds spent in cache. Age: 0 means the response came straight from the origin. A non-zero Age alongside a "the site shows old prices" complaint explains itself.
• Via — the chain of proxies the response passed through.
• X-Cache, X-Cache-Status, CF-Cache-Status and friends — cache hit or miss. The names are vendor-specific, the meaning is the same: HIT, MISS, EXPIRED, BYPASS, STALE.
• Server is often replaced with the CDN's own name, so it no longer identifies the real web server.

Who wins: origin or CDN

A shared cache must obey the origin's Cache-Control, but the directives have their own precedence: s-maxage beats max-age, private forbids shared storage, no-store forbids storage entirely. That said, almost every CDN lets you override the rules from its dashboard — and then users see behaviour that appears in no server config at all. When investigating, bypass the CDN and ask the origin directly:

# go straight to the origin by IP, bypassing the CDN
curl -sI --resolve example.com:443:203.0.113.25 https://example.com/

# and compare with what the CDN serves
curl -sI https://example.com/ | grep -i -E 'cache-control|age|x-cache|vary|server'

Duplicate headers and the nginx add_header trap

The most expensive mistake in this area is not a header at all, it is nginx behaviour. add_header directives are inherited from the level above only if the current level defines none of them. Add a single add_header inside a location and the entire set from server and http disappears for that location. Security headers carefully configured at the server level vanish silently on your most important page, and the site keeps working: no error, no warning in the log.

# WRONG: /api/ ends up with only Cache-Control,
# HSTS and nosniff from the server context are gone
server {
    add_header Strict-Transport-Security "max-age=31536000" always;
    add_header X-Content-Type-Options "nosniff" always;

    location /api/ {
        add_header Cache-Control "no-store" always;   # wipes the whole set above
    }
}

# RIGHT: keep the shared set in a snippet and include it in every location
# /etc/nginx/snippets/sec-headers.conf:
#   add_header Strict-Transport-Security "max-age=31536000" always;
#   add_header X-Content-Type-Options "nosniff" always;
server {
    include snippets/sec-headers.conf;

    location /api/ {
        include snippets/sec-headers.conf;
        add_header Cache-Control "no-store" always;
    }
}

A second detail about the same directive: without the always parameter the header is only attached to a limited set of status codes — 200, 201, 204, 206 and some 3xx. Error pages 403, 404 and 500 end up with no security headers at all. Write always by default.

Check separately whether your application sets the same header. If both nginx and PHP emit a CORS header, the browser receives Access-Control-Allow-Origin twice and aborts the request with a "multiple values" error. Headers that must be singular — Access-Control-Allow-Origin, Content-Type, Content-Length, Location — must be owned by exactly one layer.

How to check a site's headers: curl, DevTools and an online check

curl: the working minimum

# 1. Quick look — a HEAD request
curl -sI https://example.com/

# 2. An honest GET: headers to stdout, body to /dev/null
curl -sD - -o /dev/null https://example.com/

# 3. The whole redirect chain with headers from every hop
curl -sIL https://example.com | grep -i -E '^HTTP/|^location'

# 4. Check compression: what comes back when brotli is offered
curl -sD - -o /dev/null -H 'Accept-Encoding: br, gzip' https://example.com/ \
  | grep -i -E 'content-encoding|vary|content-length'

# 5. Compare compressed and uncompressed — catches a missing Vary
curl -sI -H 'Accept-Encoding: gzip' https://example.com/ | grep -i content-encoding
curl -sI -H 'Accept-Encoding: identity' https://example.com/ | grep -i content-encoding

# 6. Force a 200 instead of a 304 when auditing headers
curl -sD - -o /dev/null -H 'Cache-Control: no-cache' https://example.com/

# 7. Status code, body size and timings in one line
curl -s -o /dev/null -w 'code=%{http_code} size=%{size_download} ttfb=%{time_starttransfer}\n' \
     https://example.com/

# 8. See what a specific backend returns, bypassing the load balancer
curl -sI --resolve example.com:443:203.0.113.25 https://example.com/

One thing to remember about curl -I: it issues a HEAD request, not a GET. Some applications handle HEAD in a separate code path, some CDNs do not cache it the way they cache GET, and Content-Length may be missing from a HEAD response. If a result looks odd, re-check with curl -sD - -o /dev/null — that is a real GET with the body discarded.

DevTools

Network tab, click the request, open the Headers panel. Useful details: the "view source" toggle shows the raw header block rather than the browser's tidied version; the "Disable cache" checkbox forces a full 200; filtering by resource type quickly finds an asset served without compression. The note "Provisional headers are shown" means the request never reached the network — it was served from cache, cancelled or blocked by an extension — and the headers displayed at that moment cannot be trusted.

Online check

When curl is not at hand, or you need to see the response from outside your own network, an online tool is faster. The HTTP header checker shows the full server response and flags questionable values, and the guide to analysing server response headers explains how to read the result.

Common failures: symptom, cause, check, fix

Symptom	Cause	Check	Fix
Some users see binary garbage instead of a page	Missing Vary: Accept-Encoding; the CDN serves a compressed body to a client without compression	curl -sI -H 'Accept-Encoding: identity' ... | grep -i content-encoding	gzip_vary on; in nginx, then purge the CDN cache
Non-ASCII text renders as mojibake	The charset in Content-Type is wrong or absent	curl -sI ... | grep -i content-type	An explicit charset=utf-8, and one consistent encoding across files and database
Scripts do not run, console complains about a MIME type	nosniff plus a wrong Content-Type for .js or .css	DevTools → Network → resource type	Fix the types map instead of removing nosniff
HTML edits stay invisible for hours	HTML is served with a long max-age	curl -sI ... | grep -i -E 'cache-control|age'	max-age=0 for HTML; long caching only for hashed assets
Static assets are refetched on every page	No Cache-Control and no ETag	DevTools: transferred size versus "disk cache"	Hash in the filename plus max-age=31536000, immutable
After logout the page comes back with the Back button	no-cache used where no-store was needed	curl -sI /account/ | grep -i cache-control	Cache-Control: private, no-store
Security headers present on the home page but not on /api/	An add_header inside location wiped the server set	Compare curl -sI / with curl -sI /api/	Move the set into a snippet and include it in every location
No headers on 404 and 500 pages	add_header written without always	curl -sI https://example.com/no-such-page	Add always to every add_header
Infinite HTTPS redirect loop behind a proxy	The application never receives X-Forwarded-Proto	curl -sIL ... | grep -i -E '^HTTP/|location'	proxy_set_header X-Forwarded-Proto $scheme; plus trusted-proxy config in the app
Every log line shows the same client IP	real_ip not configured, you see the proxy address	Compare $remote_addr with X-Forwarded-For in the log	set_real_ip_from plus real_ip_header X-Forwarded-For
Browser errors on multiple Access-Control-Allow-Origin values	Both the web server and the application emit the header	curl -sD - -o /dev/null -H 'Origin: https://app.example.com' ...	Let exactly one layer own CORS
Video will not seek	No Accept-Ranges: bytes, or on-the-fly compression is enabled	curl -sD - -o /dev/null -r 0-1023 ...	Exclude video from gzip_types and serve it as static
PDFs and exports appear in search results	No X-Robots-Tag: a file has nowhere to put meta robots	curl -sI /files/report.pdf | grep -i robots	add_header X-Robots-Tag "noindex" always; on the file directory
Half the headers "disappeared" during a check	The server answered 304, not 200	Read the status line first	Repeat the request with -H 'Cache-Control: no-cache'

How to check your own site

• HTTP header check — the full server response: the caching group, Content-Type, Vary, compression, and flags on questionable values.
• Website security check — HSTS, CSP, nosniff, Referrer-Policy and the rest of the defensive set, with a score.
• Speed check — shows where a long Cache-Control and compression actually save load time, and where assets are refetched for nothing.
• Redirect checker — the whole Location chain with status codes: finds loops, needless hops and drops to plain HTTP.
• Cookie checker — which cookies the site sets via Set-Cookie and whether they carry Secure, HttpOnly and SameSite.
• CORS check — what the server answers to a preflight and whether the Access-Control-* values match what the frontend expects.

Frequently asked questions

What is the difference between no-cache and no-store?

no-cache permits storing the response but requires revalidation with the server before every reuse — the copy physically sits in the browser cache. no-store forbids storing it anywhere. For pages with personal data only no-store works: with no-cache an account page stays on disk and comes back with the Back button after logout.

Why did half the headers disappear on a 304 response?

By design. 304 Not Modified has no body, so headers that describe a body — Content-Type, Content-Length, Content-Encoding — are not included. The server sends only what could have changed: Date, ETag, Cache-Control, Vary. If you are auditing security headers, force a 200 by adding -H 'Cache-Control: no-cache' to the request.

Do I still need Expires if I already set Cache-Control?

No. When both are present, caches use Cache-Control: max-age and ignore Expires. The latter depends on synchronised clocks and cannot express s-maxage, immutable or stale-while-revalidate. There is nothing to configure separately either: the nginx expires directive emits both headers anyway.

I added one add_header and the other headers vanished. Why?

That is standard nginx behaviour: add_header is inherited from the level above only when the current level defines none. A single add_header in a location cancels the whole server set. The fix is to keep shared headers in a separate file and include it in every location that adds its own.

Can I trust X-Forwarded-For?

Only the value appended by your own proxy. It is an ordinary request header that a client sends itself with arbitrary content, so reading the client IP from it naively opens a bypass around rate limits and ban lists. In nginx you need a list of trusted addresses via set_real_ip_from together with real_ip_header X-Forwarded-For.

How do I view a site's headers without curl?

Open DevTools, go to the Network tab, reload the page, click the first request and read the Headers panel — it shows both request and response. If you need a view from outside your network, or a check without a browser, use the online HTTP header checker.

Why does the browser download a file instead of displaying it?

Two usual causes. Either the server sends Content-Type: application/octet-stream instead of the real type, which happens when an extension is missing from the types map. Or the response carries Content-Disposition: attachment, which explicitly demands a download. One curl -sI tells you which; the fix is either the correct type or switching attachment to inline.

Header checklist

• Every response carries an explicit Content-Type, with charset on text types.
• HTML is served with max-age=0; hashed assets with max-age=31536000, immutable.
• Personal pages and private API responses use no-store, not no-cache.
• An ETag or Last-Modified is present, and a conditional request really returns 304.
• Compression is paired with Vary: Accept-Encoding; Vary contains no User-Agent or Cookie.
• Only text types are compressed; video and images are excluded from gzip_types.
• The security header set is identical on the home page, inner sections and error pages.
• Every add_header carries always, and the shared set lives in an included snippet.
• Exact versions in Server and X-Powered-By are hidden.
• Behind a proxy the app receives X-Forwarded-Proto, and X-Forwarded-For is accepted only from trusted addresses.
• Export files and internal directories are closed off with X-Robots-Tag.
• 429 and 503 responses include Retry-After.
• No singular-only header is emitted twice by two infrastructure layers.
• Checks were run both directly against the origin and through the CDN, and the results compared.