upstream request timeout (Envoy, Istio): Why the Proxy Gave Up and Returned 504

When the route's timer fires, Envoy's router aborts the upstream request with the response flag UT (UpstreamRequestTimeout) and, when the timeout response code is 504 Gateway Timeout (the normal case), sends the body upstream request timeout (router.cc). The response therefore looks like this:

HTTP/1.1 504 Gateway Timeout
server: envoy

upstream request timeout

The timer is the route timeout. Its documentation says: "If not specified, the default is 15s. This spans between the point at which the entire downstream request (i.e. end-of-stream) has been processed and when the upstream response has been completely processed", and "This timeout includes all retries" (route_components.proto). So a backend that streams a large response for more than 15 seconds can be cut off even though it started answering quickly.

This is not a connect problem: the connection worked. Failures to connect show up as upstream connect error with a 503.

The per-try note matters for diagnosis: with a 5xx retry policy and no per-try timeout, a request that times out is not retried, because the total budget is already spent (RetryPolicy.per_try_timeout). In Istio, the default request timeout is disabled (Istio request timeouts), so look for a timeout: line in the VirtualService first.

• A genuinely slow endpoint. Reports, exports, search over unindexed columns, image processing in the request path.
• A slow dependency. The backend waits on a database lock, a third-party API or DNS, then answers after the proxy has given up.
• Long responses counted in full. Downloads and server-sent events count against the route timeout until the last byte; streaming routes need the timeout disabled or raised.
• Retries eating the budget. Three attempts at five seconds each will not fit into a ten-second route timeout.
• Overload. Requests queue in the backend's worker pool; latency grows with load, and timeouts appear only at peak.

A timeout in Envoy does not stop the work in your application. The backend may still finish the job after the client got a 504, so retried POSTs can run twice unless the endpoint is idempotent.

1. Time the backend directly, bypassing the proxy:

   kubectl exec deploy/productpage -c istio-proxy -- curl -s -o /dev/null -w 'ttfb=%{time_starttransfer} total=%{time_total}\n' http://reviews:9080/report

   A total above the route timeout confirms the cause.
2. Find the timeout in force. Dump the route: istioctl proxy-config route deploy/productpage -o json | grep -n '"timeout"', or read your Envoy route config. No value means 15 s in plain Envoy.
3. Fix the slow part: add the missing index, move the export to a background job and return 202, cache the expensive call.
4. Set timeouts deliberately. Raise the route timeout only for the routes that need it, and set per_try_timeout lower than the route timeout if you rely on retries.
5. For streaming (downloads, SSE, long-polling) set the route timeout to 0 and rely on idle_timeout instead, which bounds silence rather than total duration.

curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' https://example.com/report
# 200 3.412s

The status must be 200 and the total time comfortably below the route timeout, not just under it: a response at 14.8 s against a 15 s limit will fail again under load. The access log should stop showing UT for the route. The HTTP status checker shows the status code and headers of your public URL from outside your network.

• upstream connect error or disconnect/reset before headers — the connection itself failed
• no healthy upstream — no backend to send to
• 504 Gateway Timeout fixes — nginx, Cloudflare and cloud load balancers
• 504 Gateway Timeout — the status code itself
• ETIMEDOUT — a timeout seen by the client application