iceberg · API

iceberg.wire

The byte primitives of iceberg's own wire form for data file entries.

DataFile.encode / DataFile.decode (and Datum's and ColumnMetrics' in them) send a data file entry from one process to another without writing a manifest: a worker writes Parquet files and returns their entries, and another process commits them. That form is built out of these:

  • an integer is 8 bytes, little-endian two's complement (Int, Int64);
  • a Float64 is its 8 IEEE 754 bytes, little-endian, NaN payloads kept;
  • a UInt8 is one byte, and a Bool one byte, 0 or 1;
  • a byte string is its length as an integer, then the bytes; a String is a byte string of its UTF-8;
  • a list is its length as an integer, then each element.

The layout of a DataFile is versioned: its first byte is iceberg.manifest.DATA_FILE_WIRE_VERSION, and a reader refuses a version it doesn't know rather than misreading it, so two builds that disagree say so. Every read is bounds-checked, and a length that doesn't fit what's left raises, so a truncated or corrupt buffer is an error, never a read past the end.

The signatures follow one shape, encode(self, mut out: List[UInt8]) and @staticmethod decode(buf: Span[UInt8, _], mut pos: Int) raises -> Self, which is the shape of dista's Codec trait. iceberg doesn't depend on dista, so its types don't declare Codec; a wrapper that does is two one-line methods (orca's orca.tables.DataFiles is one).

Functions

need

def need(buf: Span[UInt8], pos: Int, n: Int)

Raises unless n bytes are left at pos.

put_u8

def put_u8(mut out: List[UInt8], v: UInt8)

get_u8

def get_u8(buf: Span[UInt8], mut pos: Int) -> UInt8

put_bool

def put_bool(mut out: List[UInt8], v: Bool)

get_bool

def get_bool(buf: Span[UInt8], mut pos: Int) -> Bool

put_i64

def put_i64(mut out: List[UInt8], v: Int64)

get_i64

def get_i64(buf: Span[UInt8], mut pos: Int) -> Int64

put_int

def put_int(mut out: List[UInt8], v: Int)

get_int

def get_int(buf: Span[UInt8], mut pos: Int) -> Int

put_f64

def put_f64(mut out: List[UInt8], v: Float64)

get_f64

def get_f64(buf: Span[UInt8], mut pos: Int) -> Float64

get_count

def get_count(buf: Span[UInt8], mut pos: Int, min_size: Int) -> Int

A list length, refused if its elements, of at least min_size bytes each, can't fit in what's left.

put_bytes

def put_bytes(mut out: List[UInt8], b: List[UInt8])

get_bytes

def get_bytes(buf: Span[UInt8], mut pos: Int) -> List[UInt8]

put_string

def put_string(mut out: List[UInt8], s: String)

get_string

def get_string(buf: Span[UInt8], mut pos: Int) -> String