iceberg · API

iceberg.metadata_log

How many previous metadata files a table remembers, and whether it deletes the ones it forgets.

Every commit writes a whole new *.metadata.json and adds the one it replaced to the metadata-log. Left alone, both grow by one per commit for ever: the log inside every new file, and the files themselves in metadata/. Two table properties bound them, with Java's defaults:

  • write.metadata.previous-versions-max (100) — the log keeps the newest this many entries. This applies to every commit, as in Java.
  • write.metadata.delete-after-commit.enabled (false) — after a successful commit, the files that fell off the log are deleted. Off, they stay where they are and nothing points at them.

Deleting happens only after the new metadata is in place, and a file that is already gone is not an error, so a crash in between leaves at worst a few unreferenced files.

Functions

previous_versions_max

def previous_versions_max(metadata: TableMetadata) -> Int

delete_after_commit

def delete_after_commit(metadata: TableMetadata) -> Bool

record_previous

def record_previous(mut metadata: TableMetadata, previous: String, previous_updated_ms: Int64)

Make sure the file this commit replaces is the log's last entry.

The prepare_* functions add it themselves; a commit made from metadata edited by hand (expiry, a property change) may not have.

trim_metadata_log

def trim_metadata_log(mut metadata: TableMetadata) -> List[String]

Keep the newest previous-versions-max entries; return the files of the ones dropped, oldest first.

delete_metadata_files

def delete_metadata_files(io: FileIO, files: List[String]) -> Int

Delete what trim_metadata_log dropped; returns how many went.

Aliases

comptime PREVIOUS_VERSIONS_MAX = String("write.metadata.previous-versions-max")
comptime DELETE_AFTER_COMMIT = String("write.metadata.delete-after-commit.enabled")
comptime DEFAULT_PREVIOUS_VERSIONS_MAX = 100