Table
struct TableA loaded table: its metadata, where that metadata came from, and the FileIO its locations resolve through.
Implements Copyable, Movable
- Fields
metadata: TableMetadatametadata_location: Stringio: FileIOname: String
__init__
def __init__(out self, var metadata: TableMetadata, var metadata_location: String, var io: FileIO, var name: String) load
def load(location: String, var io: FileIO) -> Self Load a filesystem table from a directory or a metadata file path.
load_local
def load_local(location: String) -> Self scan
def scan(self) -> TableScan refresh
def refresh(mut self) Reload, stepping forward from the version this table already has (or from version-hint.text, when that is further on): no listing of the metadata directory.
new_append
def new_append(self) -> AppendFiles Start a fast-append transaction against this table.
append
def append(mut self, batches: List[RecordBatch]) -> Int64 Write batches and commit them as one snapshot. Rows appended.
delete_where
def delete_where(mut self, filter_dsl: String, mode: String = "") -> Int64 DELETE FROM t WHERE <filter>. Returns the rows removed.
The strategy comes from write.delete.mode unless mode overrides
it: copy-on-write (the default) rewrites the affected data files,
merge-on-read writes deletion vectors on v3 and position delete
files on v2.
delete_by_equality
def delete_by_equality(mut self, rows: RecordBatch, equality_ids: List[Int]) -> Int64 Write one equality delete file. Returns the delete rows written.
The rows are values, not positions: every table row whose
equality_ids columns match one of them is gone from then on. v2 and
unpartitioned only — see iceberg.delete.write_equality_deletes.
overwrite
def overwrite(mut self, batches: List[RecordBatch], filter_dsl: String = "[\22true\22]") -> Int64 Delete every row the filter matches and add batches, in one overwrite snapshot. With no filter this replaces the whole table. Returns the rows added.
dynamic_partition_overwrite
def dynamic_partition_overwrite(mut self, batches: List[RecordBatch]) -> Int64 Replace exactly the partitions the new rows land in. Rows added.
expire_snapshots
def expire_snapshots(mut self, older_than_ms: Int64 = Int64(-1), keep_last: Int = Int(1), dry_run: Bool = False) -> ExpireResult Drop old snapshots and the files nothing points at any more.
older_than_ms is an age cut (negative: none) and keep_last the
fewest ancestors each branch keeps; refs keep their own settings.
expire_by_properties
def expire_by_properties(mut self, dry_run: Bool = False) -> ExpireResult Expire by the table's own history.expire.* properties (Java's defaults where unset: five days, at least one snapshot).
expire
def expire(mut self, policy: ExpirePolicy, dry_run: Bool = False) -> ExpireResult Expire by policy.
The pruned metadata is committed first and the files go after, so
an interruption leaves orphans rather than a snapshot with holes in
it. A concurrent commit makes this reload and decide again: the files
to delete are only ever decided from the metadata that was committed.
dry_run stops after the decision and returns the list.
update_schema
def update_schema(mut self, update: UpdateSchema) -> Schema Commit schema changes; returns the new current schema.
No data file is touched: one new metadata file with the new schema
appended and made current. A concurrent commit makes this reload and
apply the same changes to whatever schema is current then, so a
column another writer added in the meantime is kept, and a clash
(both adding note) raises instead of committing twice.
commit
def commit(mut self, result: AppendResult) Persist a prepared commit as a new <V>-<uuid>.metadata.json.
commit_metadata
def commit_metadata(mut self, var metadata: TableMetadata) Persist table metadata as a new <V>-<uuid>.metadata.json.
Writers racing on version V settle it before any of them writes a
metadata file: each writes a claim (<V>-<uuid>.<ms>.claim), lists
the version's files, and goes on only if it finds no other claim and
no metadata file of V; otherwise it removes its claim and raises,
having written nothing a reader could load. Of two claims, the later
one's listing sees the earlier, so at most one writer goes on; it
writes its metadata file and then removes its claim.
Readers never see a loser's metadata. They did when the metadata file was itself the claim: a reader could load it in the moment before its writer saw the clash and deleted it, and commit the next version on top of it, so a commit its writer retried landed twice, or one that won was lost (orca.mojo#164).
A claim older than a minute is from a writer that died before finishing and is ignored (and removed). This is still a check-then-act protocol: it needs a listing that sees every write before it (a local filesystem, or S3 since 2020), and is unsafe over an object store that lacks one, as every filesystem-table writer is.