Skip to content

Amazon ECR — alert on a pull-failure climb

ECR pulls start failing consistently (IRSA expired, network ACL, repo policy mismatch) — pods in k8s cannot start, ImagePullBackOff. But the kubelet event pages nobody.

Recipe

bash
#!/usr/bin/env bash
# /etc/cron.d/ecr-pull
# */5 * * * * root /opt/ecr-pull.sh

CONTEXT=${KUBE_CONTEXT:-prod}

# Count ImagePullBackOff pods cluster-wide
COUNT=$(kubectl --context "$CONTEXT" get pods -A -o json \
  | jq '[.items[] |
         select(.status.containerStatuses // []
                | map(.state.waiting.reason // "")
                | any(. == "ImagePullBackOff" or . == "ErrImagePull"))]
        | length')

if [ "${COUNT:-0}" -gt 0 ]; then
  EXAMPLES=$(kubectl --context "$CONTEXT" get pods -A \
    --field-selector status.phase=Pending --no-headers \
    | awk '/ImagePullBackOff|ErrImagePull/ {printf "%s/%s,", $1, $2}' | head -c 250)
  curl -fsS "$HEARTBEAT_URL" --data-urlencode "imgpull_fail=$COUNT,examples=$EXAMPLES"
  exit 2
fi
echo "OK (no ImagePullBackOff)"

Same thing in Enterno.io

Wrap in an Enterno heartbeat — "pull-fail for image-X" usually means a protected policy / expired credential; catch and fix before 30 % of deploys break.

Set up API monitor → ← All recipes

Related recipes