The databasePart 6 of 6

What a rollback is allowed to delete

A time-travel rollback has to land a project in a coherent state across three very different stores: git source, the live Kubernetes runtime, and a shared Postgres branch. The interesting question isn't "how do we restore" — it's "what are we allowed to physically delete, and how do we stay reversible?" The answer is a single invariant: only remove what is either recoverable from git history or regenerable from the spec, and soft-archive the one relational anchor that ties them together.

What a rollback is allowed to delete

"Roll the project back to yesterday" sounds like one operation. It isn't. A project on this platform is spread across three stores with completely different durability and identity models:

  • Source — files in a per-project git repo (/data/projects/{id}), one working tree containing every app under apps/{slug}/.
  • Runtime — Kubernetes Deployments, Services, and Ingress rules in the project's namespace, generated from a spec.
  • Data — a copy-on-write Postgres branch on shared Neon storage, plus the relational rows (project_apps, plan revisions) that describe what exists.

A rollback has to make all three agree on the same past moment. The restore itself — checkout this commit, fork the Neon timeline at that LSN — is the easy half. The half that actually decides whether time travel is trustworthy is the teardown: when you roll back to a point before a feature existed, what do you physically delete, and how do you guarantee you can still go forward again?

Get it wrong in one direction and the rollback is a lie — the spec says the feature is gone but its code still sits on disk and its container still serves traffic. Get it wrong in the other direction and the rollback is destructive — you delete something you can't bring back, and "reversible time travel" quietly isn't.

This post is the rule we settled on, and why each store obeys it.

The shape of the problem

Concretely: a project is a todo app. Later it grows a second app — a blog — with its own source under apps/blog/, its own app-blog Deployment and Service, a routing rule on the shared preview Ingress, and some rows in Postgres. Then the user rolls back to a snapshot taken before the blog existed.

What "before the blog" means per store:

                      target snapshot              now
 git source     ───────────●──────────────────────●  blog files committed after ●
 k8s runtime    ───────────●──────────────────────●  app-blog Deployment running
 neon data      ───────────●──────────────────────●  blog tables written after ●
 project_apps   ───────────●──────────────────────●  blog app row created after ●

Each store needs to end up at the left dot. But they don't roll back the same way, and the naïve per-store operation is wrong for two of the three.

Source: "restore the files" is not "match the commit"

The instinct is git checkout <target> -- .. It's wrong, and subtly so.

git checkout <commit> -- <pathspec> updates the working tree for paths that exist in <commit>. A file that was added after the target — every blog source file — isn't in the target tree, so checkout simply doesn't mention it. It stays on disk. You then git add -A and commit, and the "restore" commit faithfully captures a tree that still contains the blog. The rollback ran, the diff looked plausible, and nothing got removed.

The fix is to make the working tree match the target exactly, deletions included. We capture the set of paths added between target and HEAD and remove them explicitly:

// project-git.js — restoreVersion()
await snapshotWorkingTree(dir, `restore: working-tree snapshot before reverting to ${short}`);
// Files present in HEAD but absent in the target tree (added after the target).
const addedSince = await git(dir, ["diff", "--name-only", "--diff-filter=A", target, "HEAD"]);

await git(dir, ["checkout", target, "--", "."]);   // restore modified/deleted paths
if (addedSince.length) {
  await git(dir, ["rm", "-f", "--ignore-unmatch", "--", ...addedSince]); // remove added-since
}
await git(dir, ["add", "-A"]);
await git(dir, ["commit", "-m", `restore: reverted to ${short}`]);

(git read-tree -u --reset <target> does the same thing in one move; we keep the explicit checkout-plus-delete because the failure it fixes is exactly about which files are not mentioned, and the code reads as that intent.)

One detail matters enormously here and we'll come back to it: the restore is a new commit on top of HEAD. We never move the branch pointer or rewrite history. The pre-rollback commit — the one with the blog — is still fully reachable in the object graph. That single fact is what makes deleting the files safe: they aren't gone, they're one commit away.

The same checkout also drives index maintenance. Every changed path — including the now-deleted ones — is fed to reindexFiles, which removes vanished files from the code graph and the vector index instead of leaving dangling embeddings:

// file-sync.js — reindexFiles()
if (content === null) {            // file gone after the checkout
  await fanoutSyncRemove(projectId, projectRelPath);
  scheduleVectorFlush(projectId, projectRelPath, null);
}

So the source store reaches the target by deletion that's recoverable from history. Hold that thought.

Runtime: don't restore it, regenerate it

The blog's app-blog Deployment is still running. You could try to "roll back Kubernetes" — but there's nothing to roll back to; K8s has no history. And you don't need one, because the runtime is not a store of record. Every Deployment, Service, and Ingress rule is derived — generated from the project's spec by the manifest generator on each start.

That changes the question from "how do I restore the runtime" to "how do I make the derivation stop including the blog." The spec is built from the live app list, and that query is filtered:

-- listProjectApps(): the spec only ever sees live apps
SELECT id, slug, is_primary, fs_dir_name
  FROM project_apps
 WHERE project_id = $1 AND archived_at IS NULL;

So the moment the blog's row is archived, it falls out of the spec, and the next manifest generation simply doesn't emit app-blog or its Ingress rule. The runtime converges by regeneration, not restoration.

There's still one explicit step: archiving a row removes the app from future manifests, but the Deployment that's already running keeps running until something deletes it. So the restore reconcile archives each post-target app and then re-applies the spec in one pass. The apply prunes every object the platform owns that the spec no longer names:

// db-restore-reconcile.js — reconcileAfterRestore()
const postAppIds = selectPostTargetAppIds(appRows, targetMs); // never the primary app
for (const id of postAppIds) {
  // files and data are already reverted by the git and branch restores
  await appAdmin.archiveApp(projectId, id, { deleteFiles: false, dropData: false, rebuild: false, actor: "restore" });
}
if (out.appsArchived) {
  // re-apply the spec; owned objects it no longer names are pruned:
  // the archived apps' Deployments, Services, Secrets and Ingress rules
  await reconcileLocked(projectId, { reason: "restore" });
}

Two guard rails are load-bearing. The selection never picks the primary app. The project's anchor app is never archived by a restore, so it is never pruned. And the teardown is not a list of hand-written deletes. It is the same desired-state apply every start runs, so it is safe to run twice, and it removes exactly what the spec stopped naming. The kept apps keep running. Deleting derived runtime loses nothing, because the next apply regenerates the manifests from whatever the spec names.

Data: the branch restore already did it

The blog wrote tables into Postgres. But apps share one Neon database per project, and the combined rollback restores that database by forking the branch at the snapshot's LSN. DDL and DML after that LSN — the blog's tables and their rows — are simply on the other side of the fork. There is no per-app data to delete, because point-in-time is the deletion. (This is also why a data-only restore has to check for schema drift first — reverting the branch reverts the schema too. That's its own post.)

What does not get hard-deleted is the relational anchor — the project_apps row. We soft-archive it (archived_at = now()), never DELETE. The row is the identity that ties the three stores together. It is kept, not deleted, so the app's identity and everything that references it survive the rollback. Hard-deleting it would orphan any history that referenced it.

The invariant

Step back and the three stores obey one rule. A rollback may physically delete a thing only if that thing is:

  1. Recoverable from git history — source files. The pre-rollback commit stays reachable, so deletion is a pointer away from undone.
  2. Regenerable from the spec — K8s Deployments, Services, Ingress rules, and catalog add-on services. Derived state; deleting it is free because the next apply rebuilds it. (Add-ons follow the restored spec.json: one added after the target is deprovisioned.)

Everything else is soft-archived, not destroyed — the project_apps row, the prior plan revisions. And shared data isn't deleted at all; it's reverted by the branch fork, which is itself reversible (the pre-rollback state is captured as a roll-forward snapshot before the fork).

recoverable from history

regenerable from spec

pre-fork snapshot captured

rollback to target

source: git reset worktree to target
delete added-since files

runtime: archive row → re-apply spec
prune what it no longer names

data: fork Neon branch at snapshot LSN

roll-forward

Reversibility isn't a feature bolted on top — it's the property that defines what's safe to delete in the first place. If you can't get it back, you don't delete it; you archive it or you revert-not-erase it.

The boundary of the rule

The rule only holds for state that is either in git or derived from the spec. Anything that is neither must not be blind-deleted by a rollback. Two kinds of state sit near that line, and each is handled deliberately:

  • Files that were never committed — an edit or an uploaded asset that no generation has captured yet. Before it touches the tree, restoreVersion commits the working tree into its own snapshot commit (restore: working-tree snapshot before reverting to …), and it refuses to restore if that commit can't be made. Uncommitted work becomes the roll-forward point, so the deletion that follows is recoverable from history like everything else.
  • Objects in the project's media bucket — runtime user uploads the app wrote straight to S3. The Neon branch fork reverts the project database, but it knows nothing about a bucket's contents, so a rollback leaves the bucket alone instead of guessing. A media object that a reverted row no longer references is harmless. Deleting one that is still wanted could not be undone.

The move is the same as for the relational row: capture before you remove, or leave it in place. The invariant is precise about its own boundary: recoverable or derived — and if a piece of state is neither, it is made recoverable before a rollback touches it, or the rollback doesn't touch it.

Why this shape

The deeper reason all of this works is that we keep a strict separation between stores of record and derived state, and we never let a rollback confuse the two. Source and the relational anchor are records — they get reverted with history preserved. The Kubernetes runtime is derived — it gets regenerated. Shared data is a record with its own time-travel primitive — it gets reverted by branch fork. The rollback's only job is to move each store to the same instant using that store's reversible operation, and to refuse to destroy anything it can't reconstruct.

Land that, and "go back to yesterday" stops being a hopeful one-liner and becomes what it should be: a coherent, reversible move across three stores that each remember the past in their own way.

Build on the platform these posts describe.

Describe your app in plain English — igentik writes the code, sets up the database, and ships it live.

Start building free