iceberg · API

iceberg.batch

Building an Arrow RecordBatch from Mojo values.

The write path takes Arrow, which is the right input when the data already is Arrow — a scan of another table, a Parquet file, anything speaking the C Data Interface. When it is not, this is the bridge: a ColumnBuilder per column, a Datum per cell, and batch_of to tie them together.

var ids = ColumnBuilder(String("id"), 1, P_LONG)
var region = ColumnBuilder(String("region"), 2, P_STRING)
for k in range(n):
    ids.add(Datum.long_(Int64(k)))
    region.add(Datum.string_(names[k]))
var batch = batch_of([ids^, region^])

add(Datum) is the convenient side — one tagged value per cell, which is exactly what iceberg.kernels exists to avoid on the read path. A builder fed by a parser that already holds the bytes or the integer should use add_bytes / add_int / add_float / add_bool instead, which append the value straight to the Arrow buffers, and take the array at the end rather than build a copy of it. Either way nobody lays out validity bitmaps and offset buffers by hand.

Structs

ColumnBuilder

struct ColumnBuilder

One column's values, before they become an Arrow array.

add takes a Datum per cell. add_bytes, add_int, add_float and add_bool take the value itself and append it straight to the Arrow buffers, with no tagged value at all — what a parser that already holds the bytes or the integer should use. The two may be mixed: Datums wait in values and are folded into array, in order, before the next direct append and when the column is built.

Implements Copyable, Movable, Sized

Fields
name: String
field_id: Int
kind: UInt8
precision: Int
scale: Int
length: Int
nullable: Bool
values: List[Datum]: Datums added since the last direct append, not yet in array.
array: ArrayData: Everything before values. Its last validity byte may be short until the column is built.

__init__

def __init__(out self, var name: String, field_id: Int, kind: UInt8, precision: Int = Int(0), scale: Int = Int(0), length: Int = Int(0), nullable: Bool = True)

of

def of(schema: Schema, field_id: Int) -> Self

A builder typed by one of the table's columns.

add

def add(mut self, var value: Datum)

add_null

def add_null(mut self)

add_bytes

def add_bytes(mut self, value: Span[UInt8])

A string or binary value, as its bytes (UTF-8 for a string).

add_int

def add_int(mut self, value: Int64)

An int, long, date, time or timestamp value, in the units the column stores (days, microseconds, nanoseconds).

add_float

def add_float(mut self, value: Float64)

A float or double value.

add_bool

def add_bool(mut self, value: Bool)

A boolean value.

__len__

def __len__(self) -> Int

build_tree

def build_tree(self) -> ColumnTree

The same array, ready to sit beside a nested one.

build

def build(self) -> ArrayData

The column as an Arrow array, leaving the builder as it was.

take

def take(deinit self) -> ArrayData

The column as an Arrow array, without build's copy of it.

NestedBuilder

struct NestedBuilder

A struct, list or map column, one JSON value per row.

A nested cell has no scalar to carry it, so the input here is text: the same JSON shape ScanResult.cell prints, which makes a round trip through this library one string per cell.

var addr = NestedBuilder.of(schema, 4)
addr.add('{"city":"eu","zip":10}')
addr.add_null()
var batch = batch_of_columns([ids^.build_tree(), addr^.build()])

Structs are objects keyed by field name, lists are arrays, and a map is either {"keys": [...], "values": [...]} — the spec's own shape — or a plain object when its keys are strings.

Implements Copyable, Movable, Sized

Fields
name: String
field_id: Int
ctype: ColumnType
nullable: Bool
rows: List[String]

__init__

def __init__(out self, var name: String, field_id: Int, var ctype: ColumnType, nullable: Bool = True)

of

def of(schema: Schema, field_id: Int) -> Self

A builder typed by one of the table's columns.

add

def add(mut self, var json: String)

add_null

def add_null(mut self)

__len__

def __len__(self) -> Int

build

def build(self) -> ColumnTree

Functions

array_from_datums

def array_from_datums(name: String, field_id: Int, kind: UInt8, precision: Int, scale: Int, length: Int, nullable: Bool, values: List[Datum]) -> ArrayData

One Arrow array from a column of tagged values.

is_var_width_array

def is_var_width_array(a: ArrayData) -> Bool

append_datum

def append_datum(mut a: ArrayData, d: Datum, kind: UInt8)

Append one value to a primitive array, growing every buffer it has.

finish_array

def finish_array(mut a: ArrayData)

Pad the buffers a partly filled last byte leaves short.

batch_of

def batch_of(columns: List[ColumnBuilder]) -> RecordBatch

One RecordBatch from a set of equally long column builders.

push_json

def push_json(mut arena: ArrayArena, node: Int, store: TypeStore, ti: Int, doc: Json, at: Int)

Append one JSON value to a (possibly nested) Arrow array.

finish_tree

def finish_tree(mut arena: ArrayArena, node: Int)

Pad every buffer in a freshly built tree.

batch_of_columns

def batch_of_columns(var columns: List[ColumnTree]) -> RecordBatch

One RecordBatch from column trees, nested or not.