# iceberg.maintain

> A module of iceberg's API.

Expiring snapshots, and deleting the files that nothing points at any more.

Expiry is two decisions, in this order:

1. **Which snapshots go.** The spec's retention rules, as Java applies them:
   every branch keeps its head, its last `min-snapshots-to-keep` ancestors and
   every ancestor newer than `max-snapshot-age-ms` (the branch's own values,
   or the table's `history.expire.*` properties, or the caller's); every tag
   keeps the snapshot it names; a branch or tag older than `max-ref-age-ms`
   is dropped first (never `main`). A snapshot no ref reaches is kept only
   while it is newer than the age cut. A caller can also pin snapshot ids
   outright (`retain`), which is how a scheduler keeps the snapshots a live
   run has read.
2. **Which files go with them.** A file is deleted only when *no retained
   snapshot still lists it live*.

The second decision is where expiry used to be quadratic: it read every
retained snapshot's manifest list and then every manifest on it, so a table
with one manifest per commit cost snapshots x manifests reads. It is now set
arithmetic on manifest *paths*, which is linear:

* Every manifest list is read once (they are batched on object storage).
  That gives the manifests the retained snapshots reference, and the ones the
  expired snapshots reference. A manifest is immutable and its path names it,
  so a manifest some retained snapshot lists is needed, whole, and is never
  opened to find that out.
* Only the manifests that expired snapshots list and no retained one does are
  deleted, and only they are opened: what they name are the candidates.
* A candidate data file is still needed if a retained manifest lists it live.
  It can only be listed there because a rewrite (a copy-on-write delete, a
  partition overwrite, a manifest merge, a compaction) carried it as an
  *existing* entry: a file is added once, by one snapshot, under a path no
  other file gets. The snapshot that rewrote a manifest is the one that
  dropped it from the list, and the manifest it wrote is credited to it. So
  on one line of history (every snapshot an ancestor of the current one,
  the retained ones the newest), only the retained manifests added by the
  snapshots that dropped an unreferenced manifest, and that carry existing
  or deleted entries, are opened to check the candidates against. With
  other branches or tags, or snapshots off the line, every retained
  manifest is opened once instead — still linear, never per snapshot.

A `DELETED` entry does not protect a file — that is the entry that says it is
gone. With every retained manifest opened, a file a retained snapshot marks
`DELETED` that no retained manifest lists live is an orphan and is deleted
too; with the shortcut it waits for the snapshot that deleted it to expire.

The new metadata is written **before** any file is deleted. A crash in between
leaves orphans, which cost storage; the other order leaves a snapshot pointing
at files that are not there, which costs the table. `dry_run` stops after the
decision and returns the list, which is the only safe way to find out what an
expiry would do.

Metadata files (`*.metadata.json`) are not this module's business: the
catalog's commit trims the `metadata-log` to `write.metadata.previous-versions-max`
and, with `write.metadata.delete-after-commit.enabled`, deletes the files that
fell off it.

## Structs

### `ExpirePolicy`

```mojo
struct ExpirePolicy
```

What an expiry keeps. The defaults expire nothing but history: every snapshot that is not a ref's head and not among the last one.

Implements: `Copyable`, `Defaultable`, `Movable`

Fields:

- `older_than_ms: Int64`: Snapshots at or after this timestamp are kept; `NO_AGE_LIMIT` lets age protect nothing (only `keep_last` and refs do).
- `keep_last: Int`: The fewest ancestors each branch keeps, its head included.
- `max_ref_age_ms: Int64`: Branches and tags (not `main`) whose snapshot is older than this are removed; -1 keeps them all.
- `now_ms: Int64`
- `retain: List[Int64]`: Snapshot ids never expired, whatever their age.

#### `__init__`

```mojo
def __init__(out self)
```

```mojo
def __init__(out self, older_than_ms: Int64, keep_last: Int)
```

The two arguments `expire_snapshots` always took; a negative `older_than_ms` is no age limit.

#### `from_properties`

```mojo
def from_properties(metadata: TableMetadata) -> Self
```

The table's own policy: `history.expire.max-snapshot-age-ms` (five days), `history.expire.min-snapshots-to-keep` (1) and `history.expire.max-ref-age-ms` (forever).

### `ExpireResult`

```mojo
struct ExpireResult
```

What an expiry decided, and what it did about it.

Implements: `Copyable`, `Defaultable`, `Movable`

Fields:

- `expired: List[Int64]`: Snapshot ids removed from the metadata.
- `retained: List[Int64]`
- `removed_refs: List[String]`: Branches and tags dropped for `max-ref-age-ms`.
- `deleted_files: List[String]`: Data and delete files nothing retained points at any more.
- `deleted_manifests: List[String]`: Manifests and manifest lists, likewise.
- `deleted_statistics: List[String]`: Statistics files of expired snapshots.
- `dry_run: Bool`
- `manifest_lists_read: Int`
- `manifests_read: Int`
- `reachable: Bool`: Whether every retained manifest had to be opened (other branches or tags, or a history that is not one line).

#### `__init__`

```mojo
def __init__(out self)
```

#### `total_deleted`

```mojo
def total_deleted(self) -> Int
```

### `ExpiryPlan`

```mojo
struct ExpiryPlan
```

Implements: `Copyable`, `Defaultable`, `Movable`

Fields:

- `expired: List[Int64]`
- `removed_refs: List[String]`

#### `__init__`

```mojo
def __init__(out self)
```

## Functions

### `plan_expiry`

```mojo
def plan_expiry(metadata: TableMetadata, policy: ExpirePolicy) -> ExpiryPlan
```

The snapshots and refs an expiry would remove; snapshots oldest first.

### `choose_expired`

```mojo
def choose_expired(metadata: TableMetadata, older_than_ms: Int64, keep_last: Int) -> List[Int64]
```

The snapshot ids an expiry would remove, oldest first.

A ref's head is never a candidate, and each branch keeps at least
`keep_last` of its ancestors. A negative `older_than_ms` is no age limit.

### `expire_snapshots`

```mojo
def expire_snapshots(io: FileIO, metadata: TableMetadata, older_than_ms: Int64 = Int64(-1), keep_last: Int = Int(1), dry_run: Bool = False) -> Tuple[TableMetadata, ExpireResult]
```

`expire_with_policy` with an age cut (negative: none) and a floor.

### `expire_with_policy`

```mojo
def expire_with_policy(io: FileIO, metadata: TableMetadata, policy: ExpirePolicy, dry_run: Bool = False) -> Tuple[TableMetadata, ExpireResult]
```

Decide the expiry and the files it frees.

Returns the pruned metadata and a report. The caller commits the metadata
— and must do so *before* the files go (`delete_expired_files`), which is
why nothing is deleted here.

### `delete_expired_files`

```mojo
def delete_expired_files(io: FileIO, result: ExpireResult) -> Int
```

Remove what an expiry decided is unreachable. Content first.

A file that is already gone is not an error: an expiry that was
interrupted, or two of them racing, both end at the same place.

## Aliases

### `MAX_SNAPSHOT_AGE_MS`

```mojo
comptime MAX_SNAPSHOT_AGE_MS = String("history.expire.max-snapshot-age-ms")
```

### `MIN_SNAPSHOTS_TO_KEEP`

```mojo
comptime MIN_SNAPSHOTS_TO_KEEP = String("history.expire.min-snapshots-to-keep")
```

### `MAX_REF_AGE_MS`

```mojo
comptime MAX_REF_AGE_MS = String("history.expire.max-ref-age-ms")
```

### `DEFAULT_MAX_SNAPSHOT_AGE_MS`

```mojo
comptime DEFAULT_MAX_SNAPSHOT_AGE_MS = Int64(432000000)
```

Five days, Java's default.

### `DEFAULT_MIN_SNAPSHOTS_TO_KEEP`

```mojo
comptime DEFAULT_MIN_SNAPSHOTS_TO_KEEP = 1
```

### `NO_AGE_LIMIT`

```mojo
comptime NO_AGE_LIMIT = Int64.MAX
```
