OTP codes go through Twilio — overnight a carrier started blocking the message type, delivery dropped to 30%. You only learn from support tickets.
Monitoring cookbook
Hand-written recipes for the monitoring problems we see most often. Each recipe shows a minimal DIY script and the one-click Enterno.io monitor that covers the same concern without extra infrastructure.
Filebeat / Logstash silently died on one edge node. Elasticsearch ingest rate fell 40 % but no one watches dashboards. Sentry without logs is blindness.
A CloudFront distribution started serving 5xx 4 % of the time — far-region clients see broken pages. CloudWatch graph exists; dashboard goes unwatched.
One of several systemd units (cron, redis, postfix) crashed at 3 AM and never came back. Learned about it from complaints. Want an endpoint showing high whenever any unit is failed.
A service talks to a third-party API via JWT, token lives 24h and someone occasionally renews it manually. If they forget — the pipeline breaks at 3am.
A webhook receiver accepts GitHub events. If signature validation starts failing (someone rotated the secret in one place) — deploys silently break.
After a release Sentry only fires on big incidents. I want to catch a slow error-rate climb over a 15-minute window before it becomes a real incident.
Postfix starts bouncing chunks of mail (the sending domain just lost reputation), but you only learn from a support ticket: 'no email arrived'.
The clock drifts (timesync hung, the NTP provider went down) — TOTP starts failing, JWTs with iat in the future get rejected, logs go out of order.
Auto-issuance for 200 subdomains — eventually you hit the Let's Encrypt rate limit (50 certs/week per registrable domain) and sit without HTTPS.
A MySQL replica stops applying binlog (Slave_SQL_Running=No after a bad query) — your heartbeat-only monitor doesn't catch it because lag isn't growing linearly, it's just frozen.
Sudden ban spike in fail2ban — credential stuffing or enumeration campaign. I want to know within the first 5 minutes, not the morning after.
A GitLab pipeline sometimes hangs in running for an hour+ (runner lost connectivity, or a job sits in curl with no timeout) — you only notice when a merge request 'isn't moving'.
OpenSearch hits 85 % disk (high watermark) — indices go read-only, the write API breaks. You want to catch this before 95 % (flood stage).
DynamoDB starts throttling (hot key or low write capacity) — your app gets ProvisionedThroughputExceededException, but AWS alarms only fire on a 5-minute aggregate.
All alerts go to Slack — but if the webhook URL is revoked / 404 / a Slack incident, you don't find out precisely because the alerts can't reach you (failure-of-failure).
Redis is up but `mem_fragmentation_ratio > 1.5` — real RAM use is much bigger than what `used_memory_human` reports. OOM will hit suddenly.
Inside a K8s cluster etcd re-elects the leader every 30 s — kube-apiserver lags, controller-manager can't keep reconciling. Only visible in etcd metrics.
Your OAuth callback (`/auth/callback`) starts returning 4xx — users can't log in, but a common health check on `/` is still 200.
Chargeback rate spikes — a fraud wave or a partner policy change. Stripe only emails late, and the dispute window is just 7 days.
Have a recipe we missed?
Tell us which stack to cover next — drop a line to support@enterno.io and we'll add the recipe (and credit you on the page).
Start monitoring — free →