Snapshot-first: why your restore tool should never trust the live environment

Most disaster recovery runbooks contain a sentence like this one:

Recreate the external location pointing at the original storage path.

It reads fine in a review. It is useless at 02:00, because it quietly assumes you can still find out what the original storage path was — and the system you would normally ask is the one that just broke.

I want to describe the design constraint I ended up adopting to remove that assumption entirely: a restore resolves against a snapshot of the environment captured before the incident, never against a live query. It sounds like a small implementation detail. It is not. Adopting it changes what your recovery code can be tested against, what your drills cost, and — in at least one case I will come to — whether a particular class of recovery is possible at all.

My examples are Azure Databricks with Unity Catalog over ADLS Gen2, but the principle transfers to any platform where the metadata layer and the data layer can fail independently.

The failure mode this fixes

Consider a plausible incident: someone drops a schema containing a few hundred Unity Catalog managed tables. The underlying storage has soft delete enabled, so the data is still there. You are, technically, fine.

Then you go and look at the storage account, and find this:

abfss://[email protected]/3f2504e0-4f89-11d3-9a0c-0305e82c3301/
  tables/
    9b2e4c71-0a5d-4f6e-9c3a-2b1d8e7f4a05/
    c4a0f2b8-77e1-4a9d-8b3c-6f0d5e1a2b94/
    e17d3a92-5c4b-4e88-9f2a-1d0c7b6e5a43/
    ... 400 more

There is nothing here that says which directory was sales.orders and which was sales.order_lines. Unity Catalog knew the mapping. Unity Catalog is the thing that lost it. Every one of those GUID directories holds a perfectly intact Delta table that you cannot identify, and no amount of storage-layer recovery will tell you the names.

The recovery is trivially easy if you wrote the mapping down beforehand, and effectively impossible if you did not. That asymmetry is the whole argument. There is a category of recovery metadata whose only viable capture point is before the incident, and a restore tool that queries live state has, by construction, no access to it.

What a snapshot is — and what it is not

A snapshot is a configuration capture, not a data backup. It records the shape and intended state of the environment: what exists, where it points, who can read it, and how it is protected. It does not contain a single row of business data.

That distinction matters because it sets expectations correctly in both directions. A snapshot alone cannot restore anything — it always pairs with the file or Delta backup that holds the actual bytes. But the backup alone cannot restore anything usefully either, because it restores a pile of anonymous directories. You need both, and it is worth drawing an explicit coverage matrix showing which recovery scenario needs which half.

A snapshot is also not a monitoring system. It is written on a schedule, retained for a long time, and never overwritten in place. Its value is precisely that it is old.

What to capture

The rule of thumb: capture anything you would have to look up in order to rebuild, and anything whose absence you would only notice under pressure.

For the Databricks metadata layer:

For the Azure layer underneath:

The two connectivity rows deserve a note, because it is a trap I have watched people fall into. A storage account with public network access disabled is typically reachable by two independent private paths: customer-VNet private endpoints for classic compute, and an account-level network connectivity rule for serverless compute. Repairing one does not repair the other, and a successful read over the classic path proves nothing about the serverless path. They are separate rows in the snapshot for exactly that reason — capturing them together is what makes the difference visible before someone assumes it away.

Structure on disk

Keep it boring and filesystem-shaped:

output/snapshots/{env}/{YYYY-MM-DD_HHmm}/
    storage_accounts.json          # Azure artefacts at the root
    private_endpoints.json
    prod/                          # Databricks artefacts, per stage
        catalogs.json
        external_locations.json
        storage_credentials.json
        groups.json
        prod_2026-07-02_1002_table_inventory.csv
    test/
        ...

Three properties are worth being deliberate about:

1. Timestamped directory names that sort lexicographically by time. YYYY-MM-DD_HHmm gives you “the latest snapshot” as a sorted(...)[-1], with no metadata index to maintain or corrupt.
2. Immutable directories. A snapshot run creates a new directory; it never edits an old one. This is what lets you diff two points in time.
3. Plain JSON and CSV. During an incident someone will want to open these in an editor or grep them. Do not make that hard.

Retention is a real decision, not an afterthought. Ninety days of CI artefacts is a reasonable starting point, but the retention window is also your maximum blast radius for a slow corruption: if a bad change went in four months ago and every snapshot you still hold records the damaged state, the snapshot cannot tell you what “correct” looked like.

The capture side

Each artefact gets its own small extraction script. They are boring on purpose — query the SDK, normalise, write JSON:

"""Capture catalogs, schemas, and their grants."""
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.catalog import SecurableType


def extract_catalogs(client: WorkspaceClient) -> list[dict]:
    catalogs = []
    for catalog in client.catalogs.list():
        grants = client.grants.get(
            securable_type=SecurableType.CATALOG, full_name=catalog.name
        )
        catalogs.append(
            {
                "name": catalog.name,
                "owner": catalog.owner,
                "schemas": [
                    {"name": s.name, "owner": s.owner}
                    for s in client.schemas.list(catalog_name=catalog.name)
                ],
                "grants": [
                    {"principal": a.principal, "privileges": [p.value for p in a.privileges]}
                    for a in (grants.privilege_assignments or [])
                ],
            }
        )
    return catalogs

Two conventions that pay for themselves:

Wrap the payload. Write {"generated": "<iso timestamp>", "catalogs": [...]} rather than a bare list. You will want to know when a file was written without trusting the directory name, and you will eventually want to add a field beside the payload.

Record permission failures rather than omitting them. If the caller lacks the rights to read something, write a marker — {"access": "permission_denied"} — instead of an empty list or nothing at all. An empty list is indistinguishable from “there genuinely aren’t any”, and that ambiguity will cost someone an hour during an incident. This happens more than you would expect: some account-level settings are only readable by an account administrator, so a snapshot taken with workspace-level credentials legitimately cannot see them.

The read side

The restore tool never touches the snapshot files directly. It goes through one small reader class, which is where all the format tolerance lives:

class Snapshot:
    """Read access to one snapshot directory."""

    def __init__(self, path: Path, stage: str = "prod") -> None:
        if not path.is_dir():
            raise SnapshotError(f"Snapshot directory does not exist: {path}")
        self.path = path
        self.stage = stage

    def _unwrap(self, name: str, key: str) -> list[dict]:
        """Load an artefact written as {"generated": ..., "<key>": [...]}.

        A bare list is also accepted, so a snapshot-format change does not
        break a restore.
        """
        data = json.loads((self.path / self.stage / name).read_text())
        return data.get(key, []) if isinstance(data, dict) else data

    def catalogs(self) -> list[dict]:
        return self._unwrap("catalogs.json", "catalogs")

    def managed_path_map(self) -> dict[str, str]:
        """Map fully-qualified table name → physical storage path."""
        return {
            t.full_name: t.storage_location
            for t in self.table_details()
            if t.is_managed and t.storage_location
        }

That _unwrap tolerance is not fussiness. The snapshot format will change, and the snapshots you most need are the old ones. A reader that only understands this month’s format has quietly expired your entire archive.

What falls out of the constraint

This is the part that surprised me. I adopted snapshot sourcing for correctness, but most of the benefit turned up somewhere else entirely.

Resolution becomes a pure function

If a restore scenario reads only files, then “work out what we would do” is a pure function from snapshot to plan. So express it that way: every scenario exposes a plan_* function that takes a context and returns an ordered list of steps, and nothing executes while resolving.

def plan_reregister_external_table(context: RestoreContext, table: str) -> RestorePlan:
    plan = RestorePlan(scenario="3", title="Re-register external table", target=table)
    detail = context.snapshot.table_details_by_name().get(table)

    if detail is None:
        plan.notes.append(f"{table} is absent from the snapshot inventory.")
        plan.add(
            "Identify the table's storage path by hand before proceeding",
            manual_reason="Not in the snapshot — the path must not be guessed.",
        )
        return plan

    plan.add(
        "Recreate the table at its recorded location",
        command=(
            f"CREATE TABLE {quote_full_name(table)} "
            f"LOCATION {quote_location(detail.storage_location)}"
        ),
        action=lambda: context.sql(...),
    )
    return plan

Tests become fully offline

Because resolution is pure, the entire test suite runs with no cloud credentials. A pytest fixture builds a miniature snapshot under tmp_path and the tests assert on plans:

def test_missing_table_degrades_to_a_manual_step(tiny_snapshot):
    plan = plan_reregister_external_table(ctx(tiny_snapshot), "sales.not_captured")
    assert plan.steps[0].is_manual
    assert "must not be guessed" in plan.steps[0].manual_reason

And crucially you can test the things that actually go wrong in a restore — that a schema is never created before its catalog, that a grant is never replayed before the object it applies to, that validation happens before consumers are re-pointed and not after. In a restore, the order of operations is the correctness. Ordering is almost impossible to test against a live environment and trivial to test against a fixture.

“Missing data” gets a correct home

When a plan needs a field the snapshot does not contain, there is exactly one right answer: fix the extraction script, take a new snapshot, and carry on. The wrong answer — inferring it, defaulting it, reconstructing it from a naming convention — is now visibly wrong, because the code has no live client to infer it from.

Make the degradation explicit and unpleasant. Never guess a storage path, particularly a managed table’s GUID path: a plausible-looking wrong path in a recovery plan is worse than no plan, because it will be run. Emit a step with no action and a manual_reason instead.

That leads to the guardrail I would keep even if I kept nothing else: a plan containing any manual step refuses to execute.

def execute(self) -> None:
    if self.has_manual_steps:
        raise RuntimeError(
            "Plan contains manual steps and cannot be executed automatically. "
            "Run it as a dry run and perform the manual steps by hand."
        )
    for step in self.steps:
        step.run()

Dry run is the default; --execute is opt-in and refused while anything is unresolved. A step only loses its manual flag once a drill has been recorded for it — automation is earned, scenario by scenario, rather than assumed at design time.

Drills stop being expensive

A dry run exercises the whole resolution path — snapshot lookup, path mapping, dependency ordering, SQL generation — while touching nothing. You can run every scenario against last night’s snapshot on a laptop, in seconds, and see the exact commands each one would issue. That is a rehearsal cheap enough to run continuously, which means the runbook stops rotting between annual DR tests.

Diffs explain drift

Two immutable, timestamped captures of the same environment are a diff waiting to happen:

diff-snapshots output/snapshots/prod/2026-06-30_0908 \
               output/snapshots/prod/2026-07-02_1002

A newly-failing configuration assertion becomes a concrete line of output — this grant was removed, this endpoint went from Approved to Pending — instead of a debate. This is also the natural home for compliance evidence: you are not asserting that permissions were correct on a given date, you are holding the capture.

The trap: the latest snapshot is not necessarily a good one

The obvious default — restore from the most recent snapshot — is wrong, and worth guarding against explicitly.

A scheduled capture keeps running throughout an incident. In a corruption or accidental-deletion event, the newest snapshot may faithfully record the damaged state: the grant already removed, the table already dropped, the endpoint already broken. Restoring from it reproduces the incident with great fidelity.

So make “latest” a convenience and “explicit” a first-class option:

def find_snapshot_dir(root: Path, env: str, explicit_dir: Path | None = None) -> Path:
    """Locate the snapshot to restore from.

    Args:
        explicit_dir: An exact snapshot directory, bypassing the "latest" lookup.
            Use this whenever the newest snapshot is not known-clean.
    """

And say so loudly in the runbook, at the top, where someone under pressure will actually read it. The first question of any restore is not “what do I run” but “which snapshot am I trusting, and do I know it predates the incident?”

Where this does not apply

Three honest limits.

Some checks genuinely cannot be static. Whether a TCP endpoint is currently accepting connections is not a property you can capture in JSON. Keep those as live checks, mark them clearly as such in the code, and accept that they are a different kind of test.

A snapshot is not a data backup, and the failure mode of forgetting that is severe. Draw the coverage matrix, keep it current, and make sure every scenario in the runbook names both halves of what it needs.

Capture is bounded by the credentials it runs with. Some settings need account-level rights that a workspace-scoped identity does not have. Decide deliberately whether to elevate the snapshot job or to accept a documented blind spot — but write the blind spot down, in the snapshot, as a marker.

Getting started

If you want to try this on an existing platform, the smallest useful increment is not the whole framework. It is one CSV.

1. Capture the table inventory — fully-qualified name, table type, physical storage location — on a schedule, and keep it. This is the single highest-value artefact, because it is the one that is impossible to reconstruct afterwards.
2. Add grants next, then external locations and storage credentials.
3. Write one scenario as a plan_* function that reads that inventory and returns steps it refuses to execute. Print it. Show it to whoever owns the runbook.
4. Add the Azure protection settings — locks, soft delete, public network access — so the snapshot can also answer “what recovery options do we actually have?”
5. Diff two snapshots and see what has drifted since you started.

Step 1 alone converts one impossible recovery into a routine one. Everything after it is compounding interest.

The pattern described here came out of building recovery tooling for a Unity Catalog data platform on Azure. The specifics — artefact names, scenario numbering — are implementation detail; the constraint is the transferable part. During an incident, the live environment is not a source of truth about intended state. Write it down first.