iceberg · API

iceberg.maintain

Expiring snapshots, and deleting the files that nothing points at any more.

Expiry is two decisions, in this order:

  1. Which snapshots go. The spec's retention rules, as Java applies them: every branch keeps its head, its last min-snapshots-to-keep ancestors and every ancestor newer than max-snapshot-age-ms (the branch's own values, or the table's history.expire.* properties, or the caller's); every tag keeps the snapshot it names; a branch or tag older than max-ref-age-ms is dropped first (never main). A snapshot no ref reaches is kept only while it is newer than the age cut. A caller can also pin snapshot ids outright (retain), which is how a scheduler keeps the snapshots a live run has read.
  2. Which files go with them. A file is deleted only when no retained snapshot still lists it live.

The second decision is where expiry used to be quadratic: it read every retained snapshot's manifest list and then every manifest on it, so a table with one manifest per commit cost snapshots x manifests reads. It is now set arithmetic on manifest paths, which is linear:

  • Every manifest list is read once (they are batched on object storage). That gives the manifests the retained snapshots reference, and the ones the expired snapshots reference. A manifest is immutable and its path names it, so a manifest some retained snapshot lists is needed, whole, and is never opened to find that out.
  • Only the manifests that expired snapshots list and no retained one does are deleted, and only they are opened: what they name are the candidates.
  • A candidate data file is still needed if a retained manifest lists it live. It can only be listed there because a rewrite (a copy-on-write delete, a partition overwrite, a manifest merge, a compaction) carried it as an existing entry: a file is added once, by one snapshot, under a path no other file gets. The snapshot that rewrote a manifest is the one that dropped it from the list, and the manifest it wrote is credited to it. So on one line of history (every snapshot an ancestor of the current one, the retained ones the newest), only the retained manifests added by the snapshots that dropped an unreferenced manifest, and that carry existing or deleted entries, are opened to check the candidates against. With other branches or tags, or snapshots off the line, every retained manifest is opened once instead — still linear, never per snapshot.

A DELETED entry does not protect a file — that is the entry that says it is gone. With every retained manifest opened, a file a retained snapshot marks DELETED that no retained manifest lists live is an orphan and is deleted too; with the shortcut it waits for the snapshot that deleted it to expire.

The new metadata is written before any file is deleted. A crash in between leaves orphans, which cost storage; the other order leaves a snapshot pointing at files that are not there, which costs the table. dry_run stops after the decision and returns the list, which is the only safe way to find out what an expiry would do.

Metadata files (*.metadata.json) are not this module's business: the catalog's commit trims the metadata-log to write.metadata.previous-versions-max and, with write.metadata.delete-after-commit.enabled, deletes the files that fell off it.

Structs

ExpirePolicy

struct ExpirePolicy

What an expiry keeps. The defaults expire nothing but history: every snapshot that is not a ref's head and not among the last one.

Implements Copyable, Defaultable, Movable

Fields
older_than_ms: Int64: Snapshots at or after this timestamp are kept; NO_AGE_LIMIT lets age protect nothing (only keep_last and refs do).
keep_last: Int: The fewest ancestors each branch keeps, its head included.
max_ref_age_ms: Int64: Branches and tags (not main) whose snapshot is older than this are removed; -1 keeps them all.
now_ms: Int64
retain: List[Int64]: Snapshot ids never expired, whatever their age.

__init__

def __init__(out self)
def __init__(out self, older_than_ms: Int64, keep_last: Int)

The two arguments expire_snapshots always took; a negative older_than_ms is no age limit.

from_properties

def from_properties(metadata: TableMetadata) -> Self

The table's own policy: history.expire.max-snapshot-age-ms (five days), history.expire.min-snapshots-to-keep (1) and history.expire.max-ref-age-ms (forever).

ExpireResult

struct ExpireResult

What an expiry decided, and what it did about it.

Implements Copyable, Defaultable, Movable

Fields
expired: List[Int64]: Snapshot ids removed from the metadata.
retained: List[Int64]
removed_refs: List[String]: Branches and tags dropped for max-ref-age-ms.
deleted_files: List[String]: Data and delete files nothing retained points at any more.
deleted_manifests: List[String]: Manifests and manifest lists, likewise.
deleted_statistics: List[String]: Statistics files of expired snapshots.
dry_run: Bool
manifest_lists_read: Int
manifests_read: Int
reachable: Bool: Whether every retained manifest had to be opened (other branches or tags, or a history that is not one line).

__init__

def __init__(out self)

total_deleted

def total_deleted(self) -> Int

ExpiryPlan

struct ExpiryPlan

Implements Copyable, Defaultable, Movable

Fields
expired: List[Int64]
removed_refs: List[String]

__init__

def __init__(out self)

Functions

plan_expiry

def plan_expiry(metadata: TableMetadata, policy: ExpirePolicy) -> ExpiryPlan

The snapshots and refs an expiry would remove; snapshots oldest first.

choose_expired

def choose_expired(metadata: TableMetadata, older_than_ms: Int64, keep_last: Int) -> List[Int64]

The snapshot ids an expiry would remove, oldest first.

A ref's head is never a candidate, and each branch keeps at least keep_last of its ancestors. A negative older_than_ms is no age limit.

expire_snapshots

def expire_snapshots(io: FileIO, metadata: TableMetadata, older_than_ms: Int64 = Int64(-1), keep_last: Int = Int(1), dry_run: Bool = False) -> Tuple[TableMetadata, ExpireResult]

expire_with_policy with an age cut (negative: none) and a floor.

expire_with_policy

def expire_with_policy(io: FileIO, metadata: TableMetadata, policy: ExpirePolicy, dry_run: Bool = False) -> Tuple[TableMetadata, ExpireResult]

Decide the expiry and the files it frees.

Returns the pruned metadata and a report. The caller commits the metadata — and must do so before the files go (delete_expired_files), which is why nothing is deleted here.

delete_expired_files

def delete_expired_files(io: FileIO, result: ExpireResult) -> Int

Remove what an expiry decided is unreachable. Content first.

A file that is already gone is not an error: an expiry that was interrupted, or two of them racing, both end at the same place.

Aliases

comptime MAX_SNAPSHOT_AGE_MS = String("history.expire.max-snapshot-age-ms")
comptime MIN_SNAPSHOTS_TO_KEEP = String("history.expire.min-snapshots-to-keep")
comptime MAX_REF_AGE_MS = String("history.expire.max-ref-age-ms")
comptime DEFAULT_MAX_SNAPSHOT_AGE_MS = Int64(432000000)

Five days, Java's default.

comptime DEFAULT_MIN_SNAPSHOTS_TO_KEEP = 1
comptime NO_AGE_LIMIT = Int64.MAX