iceberg · API

iceberg.write

Writing data files: Arrow in, Parquet out, a DataFile record back.

write_data_files takes Arrow RecordBatches, aligns their columns to the table's schema by name (falling back to the field id when the batch carries one), partitions each batch by the spec's transforms, writes one Parquet file per partition per batch through parquet.mojo's writer, and returns the data_file records a manifest needs — record count, file size, split offsets, and the per-column sizes, value counts, null counts and Appendix-D bounds that InclusiveMetricsEvaluator prunes on.

The statistics come from the Parquet footer the writer just produced rather than from a second pass over the values: whatever the writer decided the min and max were is what the manifest reports, which is the only way the two can agree. Bounds are truncated the way write.metadata.metrics.default's truncate(16) says — a lower bound is cut, an upper bound is cut and then incremented so it stays an upper bound, and dropped when it cannot be.

Structs

WriteOptions

struct WriteOptions

How to write data files.

Implements Copyable, Defaultable, Movable

Fields
codec: String
row_group_rows: Int: Rows per row group, which is what parquet.mojo's writer splits on.
page_size: Int: Target uncompressed bytes of values per data page.
compression_level: Int: The codec's level: 0 is the codec's own default. ZSTD takes 1 to 22 and is the only codec that reads it. A higher level writes a smaller file more slowly and changes nothing for readers.
use_dictionary: Bool: Whether column chunks may be dictionary-encoded. On, the writer still falls back to plain for any chunk where a dictionary would not pay.
truncate: Int: write.metadata.metrics.default = truncate(N); 0 disables truncation.
num_workers: Int: Threads that encode each data file's pages (parquet.mojo's WriterOptions.num_workers): 1, the default, encodes on the calling thread, 0 on every core. The files are the same either way.

__init__

def __init__(out self)

from_properties

def from_properties(properties: Dict[String, String]) -> Self

Functions

codec_value

def codec_value(name: String) -> Int32

truncate_lower

def truncate_lower(raw: List[UInt8], kind: UInt8, n: Int) -> List[UInt8]

A lower bound cut to n units — code points for a string, bytes otherwise. Cutting a lower bound can only lower it, which is safe.

truncate_upper

def truncate_upper(raw: List[UInt8], kind: UInt8, n: Int) -> Tuple[List[UInt8], Bool]

An upper bound cut to n units and then incremented.

Truncation lowers a value, so an upper bound has to be nudged back up or it stops bounding. For a string that means incrementing the last code point that is not the maximum; for binary, the last byte that is not 0xFF. When every unit is at its maximum the bound cannot be truncated at all and the second element of the result is False — the spec's answer is to omit it.

bound_bytes

def bound_bytes(v: ScalarValue, kind: UInt8, precision: Int, scale: Int) -> Tuple[List[UInt8], Bool]

human_partition_value

def human_partition_value(d: Datum, transform_kind: UInt8) -> String

The Hive-style rendering of one partition value.

escape_path

def escape_path(s: String) -> String

Percent-escape everything a Hive path may not contain.

partition_path

def partition_path(spec: PartitionSpec, values: List[Datum]) -> String

align_batch

def align_batch(batch: RecordBatch, schema: Schema) -> List[ColumnTree]

The batch's columns in the table's schema order, at the table's types.

A nested column is matched child by child by field id, so a batch whose struct has its fields in another order, under other names, or one field short still lines up with the table — the same resolution the read path does, run the other way.

Every column comes back in an arena of its own, so this copies each one once. write_data_files does not go through here: it uses _align, which leaves a column that already matches where it is.

write_parquet

def write_parquet(columns: List[ColumnTree], schema: Schema, options: WriteOptions) -> List[UInt8]

encode_parquet

def encode_parquet(arena: ArrayArena, roots: List[Int], schema: Schema, options: WriteOptions) -> List[UInt8]

One Parquet file of the arrays at roots, which are already aligned to schema. The arena may hold other arrays too; only roots and what they reach are written.

write_parquet_to

def write_parquet_to[S: ByteSink](arena: ArrayArena, roots: List[Int], schema: Schema, options: WriteOptions, mut sink: S) -> WrittenFile

encode_parquet, handing each row group to sink as soon as it is encoded, and returning the footer the writer wrote — which is what the data_file is built from.

data_file_from_parquet

def data_file_from_parquet(data: List[UInt8], path: String, schema: Schema, spec: PartitionSpec, partition: List[Datum], sort_order_id: Int, options: WriteOptions) -> DataFile

Read back the footer of a file already in memory and turn it into a data_file. Only the footer is parsed, from data where it lies.

write_data_files

def write_data_files(io: FileIO, location: String, batches: List[RecordBatch], schema: Schema, spec: PartitionSpec, sort_order_id: Int, options: WriteOptions) -> List[DataFile]

Write every batch out, one Parquet file per partition per batch.

Aliases

comptime DEFAULT_METRICS_TRUNCATE = 16
comptime HIVE_NULL = String("__HIVE_DEFAULT_PARTITION__")
comptime PROP_COMPRESSION = String("write.parquet.compression-codec")
comptime PROP_COMPRESSION_LEVEL = String("write.parquet.compression-level")

Iceberg's name for the codec level. Honoured for ZSTD (1 to 22); the other codecs have no level here and ignore it.

comptime PROP_DICT_SIZE = String("write.parquet.dict-size-bytes")

Iceberg's dictionary-page cap. parquet.mojo sizes its dictionary by value count rather than bytes, so only the one answer that means the same thing in both is honoured: 0 switches dictionary encoding off, anything else leaves it on, as the writer decides per column chunk whether a dictionary pays.

comptime PROP_PAGE_SIZE = String("write.parquet.page-size-bytes")
comptime PROP_ROW_GROUP_ROWS = String("write.parquet.row-group-size-rows")

Not an Iceberg property.

Iceberg spells the row-group target in bytes (write.parquet.row-group-size-bytes), and parquet.mojo's writer splits a batch by a row count. Converting between them means guessing a compression ratio before compressing, which would make the setting mean something different from what it says, so the byte property is ignored and this one — under a name no reader will mistake for the spec's — is honoured instead.