Incident Response: from alert to postmortem

• Severity matrix: SEV-1 (outage → all hands), SEV-2 (degradation), SEV-3 (minor)
• Incident Commander coordinates, does not code — otherwise the team loses focus
• Single source of truth: one channel (#inc-2026-04-18-db) + one doc
• Internal vs external comms: status page + Twitter, no details until RCA
• Postmortem: what happened, why, impact, action items (owner + due date)

# Incident template (Slack)
:rotating_light: SEV-2 incident declared
**Impact**: 20% of API requests returning 500
**Started**: 14:23 UTC
**IC**: @alex
**Channel**: #inc-2026-04-18-db
**Status page**: https://status.example.com/i/abc123

# Updates every 30 min
14:23 Declared, investigating
14:38 DB connection pool exhausted
14:52 Rolled back deploy 07f2
15:02 Recovery confirmed, monitoring
15:30 Resolved, postmortem 2026-04-20

• SLO / SLI
• Observability

To run an effective incident response, follow a structured approach: identify the incident, contain it, eradicate the cause, recover systems, and conduct a post-incident review. Utilize frameworks like NIST SP 800-61 for guidance and tools like SIEM for monitoring. A typical incident response time should aim for less than 30 minutes for containment and under 24 hours for recovery.

Identifying an incident is the first critical step in your incident response plan. Utilize monitoring tools such as Splunk or ELK Stack to detect anomalies in your systems. Key indicators of compromise (IoCs) include unusual logins, unexpected outbound traffic, or changes in file integrity. Once an incident is identified, immediate containment is crucial. This can involve isolating affected systems from the network.

For example, if a suspicious process is detected, use the following command to isolate a Linux server:

sudo iptables -A INPUT -s  -j DROP

This command blocks incoming traffic from a specific IP address, helping to prevent further damage. Additionally, ensure that you log all containment actions for later analysis.

After containment, the next step is eradication. This involves removing the root cause of the incident, which may include deleting malware, patching vulnerabilities, or changing compromised credentials. For instance, if a vulnerability in a web application is exploited, apply the relevant patches immediately. Use a command like:

sudo apt-get update && sudo apt-get upgrade

to update your system packages in a Debian-based environment.

Once eradication is complete, recovery can begin. This typically involves restoring systems from clean backups and monitoring them closely for any signs of residual issues. It’s important to validate that all systems are functioning normally before reconnecting them to the network.

Finally, conduct a post-incident analysis to review the incident response process. Document the timeline of events, the effectiveness of the response, and identify areas for improvement. Utilize frameworks such as the NIST Cybersecurity Framework to guide your analysis. Conducting this review not only improves your future incident response efforts but also helps in compliance with regulations such as GDPR or CCPA.