# iceberg.batch

> A module of iceberg's API.

Building an Arrow `RecordBatch` from Mojo values.

The write path takes Arrow, which is the right input when the data already
*is* Arrow — a scan of another table, a Parquet file, anything speaking the C
Data Interface. When it is not, this is the bridge: a `ColumnBuilder` per
column, a `Datum` per cell, and `batch_of` to tie them together.

```mojo
var ids = ColumnBuilder(String("id"), 1, P_LONG)
var region = ColumnBuilder(String("region"), 2, P_STRING)
for k in range(n):
    ids.add(Datum.long_(Int64(k)))
    region.add(Datum.string_(names[k]))
var batch = batch_of([ids^, region^])
```

`add(Datum)` is the convenient side — one tagged value per cell, which is
exactly what `iceberg.kernels` exists to avoid on the read path. A builder
fed by a parser that already holds the bytes or the integer should use
`add_bytes` / `add_int` / `add_float` / `add_bool` instead, which append the
value straight to the Arrow buffers, and `take` the array at the end rather
than `build` a copy of it. Either way nobody lays out validity bitmaps and
offset buffers by hand.

## Structs

### `ColumnBuilder`

```mojo
struct ColumnBuilder
```

One column's values, before they become an Arrow array.

`add` takes a `Datum` per cell. `add_bytes`, `add_int`, `add_float` and
`add_bool` take the value itself and append it straight to the Arrow
buffers, with no tagged value at all — what a parser that already holds
the bytes or the integer should use. The two may be mixed: `Datum`s wait
in `values` and are folded into `array`, in order, before the next direct
append and when the column is built.

Implements: `Copyable`, `Movable`, `Sized`

Fields:

- `name: String`
- `field_id: Int`
- `kind: UInt8`
- `precision: Int`
- `scale: Int`
- `length: Int`
- `nullable: Bool`
- `values: List[Datum]`: `Datum`s added since the last direct append, not yet in `array`.
- `array: ArrayData`: Everything before `values`. Its last validity byte may be short until the column is built.

#### `__init__`

```mojo
def __init__(out self, var name: String, field_id: Int, kind: UInt8, precision: Int = Int(0), scale: Int = Int(0), length: Int = Int(0), nullable: Bool = True)
```

#### `of`

```mojo
def of(schema: Schema, field_id: Int) -> Self
```

A builder typed by one of the table's columns.

#### `add`

```mojo
def add(mut self, var value: Datum)
```

#### `add_null`

```mojo
def add_null(mut self)
```

#### `add_bytes`

```mojo
def add_bytes(mut self, value: Span[UInt8])
```

A `string` or `binary` value, as its bytes (UTF-8 for a string).

#### `add_int`

```mojo
def add_int(mut self, value: Int64)
```

An `int`, `long`, `date`, `time` or timestamp value, in the units the column stores (days, microseconds, nanoseconds).

#### `add_float`

```mojo
def add_float(mut self, value: Float64)
```

A `float` or `double` value.

#### `add_bool`

```mojo
def add_bool(mut self, value: Bool)
```

A `boolean` value.

#### `__len__`

```mojo
def __len__(self) -> Int
```

#### `build_tree`

```mojo
def build_tree(self) -> ColumnTree
```

The same array, ready to sit beside a nested one.

#### `build`

```mojo
def build(self) -> ArrayData
```

The column as an Arrow array, leaving the builder as it was.

#### `take`

```mojo
def take(deinit self) -> ArrayData
```

The column as an Arrow array, without `build`'s copy of it.

### `NestedBuilder`

```mojo
struct NestedBuilder
```

A struct, list or map column, one JSON value per row.

A nested cell has no scalar to carry it, so the input here is text: the
same JSON shape `ScanResult.cell` prints, which makes a round trip through
this library one string per cell.

```mojo
var addr = NestedBuilder.of(schema, 4)
addr.add('{"city":"eu","zip":10}')
addr.add_null()
var batch = batch_of_columns([ids^.build_tree(), addr^.build()])
```

Structs are objects keyed by field **name**, lists are arrays, and a map is
either `{"keys": [...], "values": [...]}` — the spec's own shape — or a
plain object when its keys are strings.

Implements: `Copyable`, `Movable`, `Sized`

Fields:

- `name: String`
- `field_id: Int`
- `ctype: ColumnType`
- `nullable: Bool`
- `rows: List[String]`

#### `__init__`

```mojo
def __init__(out self, var name: String, field_id: Int, var ctype: ColumnType, nullable: Bool = True)
```

#### `of`

```mojo
def of(schema: Schema, field_id: Int) -> Self
```

A builder typed by one of the table's columns.

#### `add`

```mojo
def add(mut self, var json: String)
```

#### `add_null`

```mojo
def add_null(mut self)
```

#### `__len__`

```mojo
def __len__(self) -> Int
```

#### `build`

```mojo
def build(self) -> ColumnTree
```

## Functions

### `array_from_datums`

```mojo
def array_from_datums(name: String, field_id: Int, kind: UInt8, precision: Int, scale: Int, length: Int, nullable: Bool, values: List[Datum]) -> ArrayData
```

One Arrow array from a column of tagged values.

### `is_var_width_array`

```mojo
def is_var_width_array(a: ArrayData) -> Bool
```

### `append_datum`

```mojo
def append_datum(mut a: ArrayData, d: Datum, kind: UInt8)
```

Append one value to a primitive array, growing every buffer it has.

### `finish_array`

```mojo
def finish_array(mut a: ArrayData)
```

Pad the buffers a partly filled last byte leaves short.

### `batch_of`

```mojo
def batch_of(columns: List[ColumnBuilder]) -> RecordBatch
```

One `RecordBatch` from a set of equally long column builders.

### `push_json`

```mojo
def push_json(mut arena: ArrayArena, node: Int, store: TypeStore, ti: Int, doc: Json, at: Int)
```

Append one JSON value to a (possibly nested) Arrow array.

### `finish_tree`

```mojo
def finish_tree(mut arena: ArrayArena, node: Int)
```

Pad every buffer in a freshly built tree.

### `batch_of_columns`

```mojo
def batch_of_columns(var columns: List[ColumnTree]) -> RecordBatch
```

One `RecordBatch` from column trees, nested or not.
