Instruction 06-disaster-recovery-drills: Reading

A backup is only an input to disaster recovery. Recovery also requires people who know when to invoke the plan, credentials that remain available during an outage, documented dependencies, compatible replacement capacity, validated data, and a method for deciding when service is safe to return. A successful backup job therefore does not prove that a service can be restored. A drill closes this assurance gap by selecting a scenario, declaring boundaries, injecting a controlled failure, executing the runbook, verifying the result, and recording improvements.

Two common planning measures are recovery time objective and recovery point objective. RTO is a target for how long service restoration may take. RPO is a target for how much recent data may be lost. The observed values from an exercise are recovery time actual and recovery point actual. Targets and actuals must not be confused. A five-minute restore in a tiny classroom directory does not predict the recovery time of a large production system. Production timing includes data transfer, replacement infrastructure, dependency recovery, authentication, network changes, validation, approvals, and user communication.

A rigorous drill starts with scope and safety controls. Identify the protected service, recovery point, incident controller, operator, observer, start condition, stop conditions, communications path, and escalation method. Confirm that the backup is isolated from the simulated failure. Record the exact artifact selected for recovery and validate that it can be read before creating the outage. For an archive, validation should include checking the compression stream, listing members, reviewing paths, and retaining a cryptographic digest. Restoration should occur into an isolated location when possible so that integrity and application checks can happen before cutover.

Verification must test more than file existence. Structural verification asks whether required files and directories are present. Integrity verification compares trusted hashes or application-level records. Functional verification checks whether the service can perform its intended work. Dependency verification checks identity, storage, name resolution, networking, certificates, secrets, and downstream systems. Business validation asks whether the recovered state is acceptable to the service owner. In this laboratory, a deterministic file manifest represents integrity verification, while a configuration-and-data health check represents a minimal functional gate.

Drills should produce evidence and corrective actions rather than a pass-or-fail slogan. Record when the incident was declared, when recovery began, which backup was selected, every deviation from the runbook, when technical verification completed, and who authorized return to service. Separate operator mistakes from runbook defects and environmental defects. Assign each corrective action an owner and due date, then retest material changes. Repeated exercises should vary scenarios: corrupted data, unavailable backup credentials, loss of a host, failed dependencies, compromised administrative accounts, or an unusable recent recovery point. The goal is not theatrical failure. The goal is repeatable proof that recovery assumptions remain true.