no healthy upstream (Envoy, Istio): Why the Cluster Has No Endpoints

no healthy upstream is a local reply generated by Envoy's router filter, not by your application. When the load balancer cannot pick a host for the request, the router calls sendNoHealthyUpstreamResponse(), which sends the literal body no healthy upstream with status 503 by default and marks the request with the response flag UH (NoHealthyUpstream) in the access log (router.cc, access log response flags). You see it behind anything built on Envoy: Istio sidecars and gateways, Envoy Gateway, Contour, Emissary, Gloo and many API gateways.

The key point is that no connection to a backend was attempted. Envoy decided before connecting that the cluster had nothing to send to. That separates it from upstream connect error or disconnect/reset before headers (flag UF: a host was chosen, but the TCP or TLS connection to it failed) and from NR (no route matched, usually a 404). It is also not a timeout: that is upstream request timeout with a 504.

"Healthy" is narrower than it sounds. With default settings Envoy does not stop routing when every host fails its health checks: below the panic threshold (50% available hosts by default) it ignores health status and balances across all hosts (panic threshold). So on a default cluster the message usually means the cluster has zero hosts, not that its hosts are sick.

Outlier detection rarely empties a whole cluster by itself: by default it ejects at most 10% of hosts (max_ejection_percent), and an ejection lasts base_ejection_time, 30 seconds by default, multiplied by the number of previous ejections (outlier_detection.proto). If you raised that percentage to 100, a burst of 5xx can eject every host and produce this error until the ejections expire.

In a mesh, the cluster's endpoints come from the control plane, which builds them from Kubernetes EndpointSlices. A pod that is not Ready is not an endpoint, so a failing readiness probe removes the pod from Envoy's view before any Envoy health check runs. Start from Kubernetes, then look at what the proxy actually received:

kubectl get pods -l app=reviews -o wide
kubectl get endpointslices -l kubernetes.io/service-name=reviews -o wide
istioctl proxy-config endpoint deploy/productpage --cluster "outbound|9080|v2|reviews.default.svc.cluster.local"

An empty endpoint list for the exact cluster name in the error path means the subset or Service has no members. If the list has entries marked UNHEALTHY, look at health checks and outlier detection. In a sidecar you can also read Envoy's own view:

kubectl exec deploy/productpage -c istio-proxy -- pilot-agent request GET clusters | grep reviews

Each host line ends with its health status: healthy, or one or more of /failed_active_hc, /failed_eds_health and /failed_outlier_check (Envoy admin /clusters). No lines at all for the cluster is the zero-hosts case.

1. Confirm the 503 comes from Envoy. curl -sv https://example.com/api/ 2>&1 | grep -iE '^< (HTTP|server)|no healthy'. A server: envoy header (unless your config rewrites it) and the exact body mean the proxy answered.
2. Find the cluster. The access log line with flag UH carries the upstream cluster name if your format includes %UPSTREAM_CLUSTER%.
3. Count the endpoints with the commands above. Zero endpoints: fix the selector, the subset labels or the crashing pods.
4. Check readiness. kubectl describe pod POD | grep -A5 Readiness. A probe pointing at the wrong port or path keeps every pod out of rotation.
5. Check Envoy health checks if you configured them: the path must answer 200, on the port and protocol the check uses. Run the same request from inside the cluster: kubectl exec deploy/productpage -c istio-proxy -- curl -s -o /dev/null -w '%{http_code}\n' http://reviews:9080/health.
6. Look for ejections in the cluster stats: pilot-agent request GET stats | grep -E 'reviews.*outlier_detection.ejections_active'.

istioctl proxy-config endpoint deploy/productpage | grep reviews
# 10.0.1.17:9080   HEALTHY   OK   outbound|9080|v2|reviews.default.svc.cluster.local
curl -s -o /dev/null -w '%{http_code}\n' https://example.com/api/
# 200

The endpoint list must show at least one HEALTHY host for the cluster the route uses, and repeated requests must stop returning the 19-byte no healthy upstream body. To check the public URL from outside your network, run it through the HTTP status checker: it shows the status code and what the server sends back.

• upstream connect error or disconnect/reset before headers — a host was picked, the connection failed
• upstream request timeout — the backend answered too slowly
• no live upstreams — the nginx equivalent
• 503 Service Unavailable — what the status code means
• Service mesh — where the sidecar proxy sits