Module 15Emergency Standard Operating ProceduresHigh Severity Incident Response

Production DBA Incident Runbooks

Prescriptive, battle-tested operational incident runbooks designed for high-pressure production outages. Each runbook includes structured triage steps, diagnostic evidence collection commands, decision gates, and post-fix verification.

Incident Response Golden Rule: Never execute destructive trial-and-error operations (e.g. kill -9, deleting files in /hana/log with rm, or blind restarts) during an active outage. Always capture active statements, trace dumps, and mount statistics first.
P1 - Critical (DB Down / Hung)

P1 Incident: /hana/log Volume 100% Full Emergency

Target MTTR: ~15 mins•Affected: hdbindexserver, Persistence Layer, All SAP ABAP Work Processes

Trigger Symptoms & Alert Signatures:

  • Database stops accepting commits; all update transactions hang.
  • alert_<SID>.trc displays: "Cannot allocate log segment; disk full on /hana/log".
  • Operating system command df -h /hana/log shows 100% utilization.
  • SAP GUI users receive DBSQL_SQL_ERROR or timeout errors.

Step-by-Step Incident Response Workflow (0/5 Steps Completed)

Detection & Triage

Verify if /hana/log is truly at 100% capacity using df -h.

BASH
df -hT /hana/log
Expected Finding: Use% shows 99% or 100%; available capacity is 0 MB.
Decision Gate: If /hana/log is 100% full, proceed immediately to Step 2. DO NOT restart the database, as it will crash during restart log replay!
Diagnostic Evidence Collection

Identify why log backups stopped clearing closed log segments.

SQL
SELECT STATE, COUNT(*) AS SEGMENT_COUNT, ROUND(SUM(TOTAL_SEGMENT_SIZE)/1024/1024, 2) AS TOTAL_MB 
FROM SYS.M_LOG_SEGMENTS 
GROUP BY STATE;
Expected Finding: Segments in state "Closed" or "Writing" dominate; 0 segments in state "Free".
Decision Gate: If log backups are failing due to a full backup destination or broken Backint pipe, resolve backup target immediately.
Remediation Action

If backup destination is full, re-route log backup target to a temporary directory with free space.

SQL
ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'SYSTEM') 
SET ('persistence', 'basepath_logbackup') = '/tmp/emergency_log_backup' 
WITH RECONFIGURE;
Decision Gate: Monitor /hana/log with df -h. As HANA writes log backups to /tmp, closed segments transition to "Backed Up" and are freed.
Escalation Trigger: If SQL connection cannot be established due to hung engine, proceed to Option B (OS-level temporary volume expansion).
Remediation Action

If LVM volume group has free extents, extend /hana/log dynamically on the Linux host.

BASH
sudo lvextend -L +20G /dev/mapper/vg_log-lv_log && sudo xfs_growfs /hana/log
Expected Finding: df -h shows 20 GB free space instantly; engine resumes writing commits automatically.
Post-Fix Verification

Ensure log segments transition to Free state and user transactions resume without errors.

SQL
SELECT STATE, COUNT(*) FROM SYS.M_LOG_SEGMENTS GROUP BY STATE;
Expected Finding: Free segments are present; active transactions resume in M_ACTIVE_STATEMENTS.
Post-Incident Prevention Guidance: Configure automated monitoring alerts in SAP HANA Cockpit at 85% and 90% filesystem thresholds. Never locate log backups on the /hana/log mount.