Prometheus — alert when a scrape target is unreachable
Prometheus itself is alive, but one of its targets has up==0 — data stops flowing, graphs go blank, and alertmanager rules built on that target don't fire (no data = no alert).
Recipe
#!/usr/bin/env bash
# /etc/cron.d/prom-target
# */1 * * * * root /opt/prom-target.sh
PROM=${PROM_URL:-http://localhost:9090}
DOWN=$(curl -fsS "$PROM/api/v1/query" --data-urlencode 'query=up==0' \
| jq -r '.data.result | length')
if [ "${DOWN:-0}" -gt 0 ]; then
TARGETS=$(curl -fsS "$PROM/api/v1/query" --data-urlencode 'query=up==0' \
| jq -r '.data.result | map(.metric.instance) | join(",")')
curl -fsS "$HEARTBEAT_URL" --data "down_targets=$DOWN,instances=$TARGETS"
exit 2
fi
echo "OK (no down targets)"Same thing in Enterno.io
Stand up an Enterno heartbeat on 1-minute intervals — a meta-monitor "monitoring the monitoring" that alerts when Prometheus loses sight of a target before any user notices.
Related recipes
An alertmanager alert sits in state=pending past its for-window — it should be active but is not firing (group_wait too big? notifier broken? misconfigured route?). Nobody gets paged.
Prometheus + Alertmanager only ship alerts to email or PagerDuty. The team lives in Telegram and you need a bridge without spinning up another service.
Filebeat / Logstash silently died on one edge node. Elasticsearch ingest rate fell 40 % but no one watches dashboards. Sentry without logs is blindness.