Expiry is two decisions, in this order:
- Which snapshots go. The spec's retention rules, as Java applies them:
every branch keeps its head, its last
min-snapshots-to-keep ancestors and
every ancestor newer than max-snapshot-age-ms (the branch's own values,
or the table's history.expire.* properties, or the caller's); every tag
keeps the snapshot it names; a branch or tag older than max-ref-age-ms
is dropped first (never main). A snapshot no ref reaches is kept only
while it is newer than the age cut. A caller can also pin snapshot ids
outright (retain), which is how a scheduler keeps the snapshots a live
run has read.
- Which files go with them. A file is deleted only when no retained
snapshot still lists it live.
The second decision is where expiry used to be quadratic: it read every
retained snapshot's manifest list and then every manifest on it, so a table
with one manifest per commit cost snapshots x manifests reads. It is now set
arithmetic on manifest paths, which is linear:
- Every manifest list is read once (they are batched on object storage).
That gives the manifests the retained snapshots reference, and the ones the
expired snapshots reference. A manifest is immutable and its path names it,
so a manifest some retained snapshot lists is needed, whole, and is never
opened to find that out.
- Only the manifests that expired snapshots list and no retained one does are
deleted, and only they are opened: what they name are the candidates.
- A candidate data file is still needed if a retained manifest lists it live.
It can only be listed there because a rewrite (a copy-on-write delete, a
partition overwrite, a manifest merge, a compaction) carried it as an
existing entry: a file is added once, by one snapshot, under a path no
other file gets. The snapshot that rewrote a manifest is the one that
dropped it from the list, and the manifest it wrote is credited to it. So
on one line of history (every snapshot an ancestor of the current one,
the retained ones the newest), only the retained manifests added by the
snapshots that dropped an unreferenced manifest, and that carry existing
or deleted entries, are opened to check the candidates against. With
other branches or tags, or snapshots off the line, every retained
manifest is opened once instead — still linear, never per snapshot.
A DELETED entry does not protect a file — that is the entry that says it is
gone. With every retained manifest opened, a file a retained snapshot marks
DELETED that no retained manifest lists live is an orphan and is deleted
too; with the shortcut it waits for the snapshot that deleted it to expire.
The new metadata is written before any file is deleted. A crash in between
leaves orphans, which cost storage; the other order leaves a snapshot pointing
at files that are not there, which costs the table. dry_run stops after the
decision and returns the list, which is the only safe way to find out what an
expiry would do.
Metadata files (*.metadata.json) are not this module's business: the
catalog's commit trims the metadata-log to write.metadata.previous-versions-max
and, with write.metadata.delete-after-commit.enabled, deletes the files that
fell off it.