Troubleshooting Labs: Real-World Case Studies & Forensic Analysis
A collection of forensic, step-by-step troubleshooting case studies directly modeled on real production incidents and lab scenarios: indexserver startup crashes, OOM dumps, replication desync, and Rocky Linux lab configuration.
Advanced (Senior DBA)Troubleshooting Labs~24 min studyHANA 2.0 (All SPS)
Core Learning Objectives
- 1.Forensically dissect an indexserver immediate crash after host reboot.
- 2.Analyze real OOM trace dumps and identify memory-hogging SQL hashes.
- 3.Diagnose and repair SAP HANA System Replication SSFS key divergence.
- 4.Configure an experimental Rocky Linux 9 lab environment correctly with swap, hosts, and firewalld rules.
Prerequisites:hana-architecture-services-overviewhana-internals-persistence-savepointshsr-architecture-modes
Definition & Architectural Rationale
Technical Definition:
Troubleshooting Labs are hands-on, scenario-driven operational exercises designed to develop senior-level diagnostic muscle memory through forensic log inspection, root cause isolation, and verified remediation procedures.
Why this component exists:
Theoretical knowledge alone fails in high-pressure production outages. Mastering exact trace grep patterns, diagnostic SQL views, and OS commands separates an entry-level candidate from an experienced 3-5 year SAP HANA DBA.
Internal Working & Mechanics
Lab Case 1: The Post-Reboot Crash Loop (Nameserver UP, Indexserver DOWN):
- Scenario: After an abrupt server power loss, HDB start is executed. Nameserver comes up GREEN, but indexserver crashes within 10 seconds. daemon.trc shows indexserver repeatedly respawning and dying.
- Diagnostic Action:
1. Inspect indexserver_*.trc: Look for the last executed operation before crash.
2. Finding: "Restart replay failed at LSN 184029410. Inconsistent redo log segment /hana/log/.../logsegment_001_00000002.dat."
3. Root Cause: Corrupted write-ahead log buffer caused by hard power cut without battery-backed disk controller cache.
4. Remediation: Recover tenant database to the last verified consistent savepoint using SystemDB.
Lab Case 2: Rocky Linux 9 Lab Environment Setup (Experimental):
- Scenario: Candidate setting up a 2-node lab VM (site-secondary) on Rocky Linux 9.8 with 42 GiB RAM for HSR testing.
- Mandatory Lab Actions:
1. Label: Explicitly label as "Experimental Lab - Unsupported by SAP PAM for Production".
2. Hostname Resolution: Correct /etc/hosts so site-secondary maps to 192.168.1.15; verify getent ahostsv4 site-secondary.
3. Swap Creation: Add 4GB swap safety buffer using fallocate -l 4G /swapfile, chmod 600, mkswap, and swapon.
4. Firewall: Allow SAP HANA ports 30013, 30015, 50013, 1128 through firewalld or disable firewalld for lab testing.
Architecture Flow Diagram
Mermaid DiagramTroubleshooting Labs: Real-World Case Studies & Forensic Analysis
Architectural Flow: Forensic troubleshooting workflow demonstrates how isolating log replay failure points enables rapid decision-making: immediately rolling back to the last consistent savepoint rather than looping blindly.
Troubleshooting & Root Cause Analysis
Production Best Practices & Operational Risks
Mandatory Best Practices
- Always retain crash trace files (*.trc) by copying them to a safe folder before running HDB start, as traces can be overwritten on restart.
- In lab environments, always verify IPv4 resolution with getent ahostsv4 to avoid subtle DNS timeouts.