The writer takes exactly what the reader produces: an ArrayArena and the
indices of the top-level arrays in it. That makes a round trip one line, and it
means anything the reader can decode, the writer can write back.
from parquet import ParquetReader, ParquetWriter, WriterOptions
var r = ParquetReader.open("in.parquet")
var t = r.read_table()
var opts = WriterOptions()
opts.codec = CompressionCodec.SNAPPY.value
var w = ParquetWriter(opts^)
for b in t.batches:
w.write_batch(b.arena, b.roots)
var bytes = w^.finish()
A file does not have to be held whole: write_batch_to hands each row group to
a ByteSink as soon as it is encoded, and finish_to writes the page indexes
and footer and returns the FileMetaData it wrote, so the caller has every
chunk's sizes and statistics without reading the file back.
var sink = FileSink("out.parquet")
var w = ParquetWriter(WriterOptions())
for b in t.batches:
w.write_batch_to(b.arena, b.roots, sink)
var written = w^.finish_to(sink) # written.metadata, written.file_size
sink.close()
WriterOptions.num_workers encodes the pages of a row group on that many
threads; the bytes are the same at any setting.
What it writes: Parquet 2.6 metadata with v1 data pages, PLAIN and
RLE_DICTIONARY values, RLE levels, three-level lists, key_value maps,
structs, field ids, per-chunk statistics with the sort order each logical type
requires, an OffsetIndex and a ColumnIndex per chunk, and key/value
metadata. Decimals go out as FIXED_LEN_BYTE_ARRAY(16), which is legal for
every precision Arrow can hold.
Structs
WriterOptions
struct WriterOptions
How to write. Every field has a sensible default.
Implements Copyable, Defaultable, Movable
- Fields
codec: Int32row_group_size: Intdata_page_size: Int: Target uncompressed bytes of values per data page.use_dictionary: Boolcompression_level: Int: The codec's compression level, where the codec has one (ZSTD: 1 to 22). 0, the default, is the codec's own default. A higher level is slower to write and smaller; reading is unaffected. Ignored by codecs without levels.write_statistics: Boolwrite_page_index: Boolcreated_by: Stringnum_workers: Int: Threads that encode a row group: 1 (the default) encodes on the calling thread, 0 means one per core.
WNode
struct WNode
One node of the tree the writer walks, mirroring the Arrow field tree.
Implements Copyable, Defaultable, Movable
- Fields
name: Stringkind: Intarrow: ArrowTypenullable: Boolfield_id: Int32children: List[Int]def_level: Intrep_level: Intelem_def_level: Intelem_rep_level: Intleaf: Int
WLeaf
struct WLeaf
A column the writer will produce chunks for.
Implements Copyable, Defaultable, Movable
- Fields
path: List[String]arrow: ArrowTypephysical: Int32type_length: Intmax_def: Intmax_rep: Intfield_id: Int32
WriteSchema
struct WriteSchema
The writer's mirror of the Arrow field tree, plus the Parquet schema.
Implements Copyable, Defaultable, Movable
- Fields
nodes: List[WNode]roots: List[Int]leaves: List[WLeaf]elements: List[SchemaElement]
LeafBuffer
struct LeafBuffer
Implements Copyable, Defaultable, Movable
- Fields
defs: List[UInt16]reps: List[UInt16]values: PhysBuffer
DictBuilder
struct DictBuilder
Distinct values in first-seen order, with an open-addressed index.
The index is a flat power-of-two table of Int32 dictionary indices with
linear probing — one array, no per-bucket allocation, no Dict and no
List of collisions.
A 4- or 8-byte fixed-width column sets key_width and is indexed by the
integer value itself: the hash is a multiply-shift of the key and the
comparison is an integer compare, so a value costs no span construction
and no byte loop at all. Everything else hashes and compares bytes.
Implements Defaultable, Movable
- Fields
values: PhysBufferslots: List[Int32]mask: UInt64used: Intkey_width: Int: 4 or 8 for the integer-keyed mode, 0 for the byte-keyed one.
__init__
def __init__(out self, kind: Int, width: Int, expect: Int = Int(0))
index_of_key
def index_of_key(mut self, key: UInt64) -> Int
The dictionary index of an integer-keyed value, adding it if new.
index_of
def index_of(mut self, v: Span[UInt8]) -> Int
ListSink
struct ListSink
Collects the output in memory. What ParquetWriter.finish uses.
Implements ByteSink, Defaultable, Movable
- Fields
bytes: List[UInt8]
write_bytes
def write_bytes(mut self, data: Span[UInt8])
FileSink
struct FileSink
Writes the output to a local file as it is produced.
Implements ByteSink, Movable
- Fields
handle: FileHandle
__init__
def __init__(out self, path: StringSpan)
write_bytes
def write_bytes(mut self, data: Span[UInt8])
WrittenFile
struct WrittenFile
What ParquetWriter.finish_to wrote: the footer and the file's size.
metadata is the FileMetaData the footer was serialised from — row
counts, every column chunk's sizes, value counts, statistics and page
offsets — so a caller that needs them (an Iceberg data_file, say) can
take them from here rather than parsing the bytes it just wrote.
Implements Movable
- Fields
metadata: FileMetaDatafile_size: Int
__init__
def __init__(out self, var metadata: FileMetaData, file_size: Int)
ParquetWriter
struct ParquetWriter[Codecs: CodecSet = DefaultCodecs]
Writes Arrow arrays out as Parquet. One write_batch per record batch.
Finished row groups wait in the writer until they are handed to a
ByteSink: flush_to hands over what is ready, write_batch_to does it
after every row group, and finish_to writes the rest and the footer.
finish is the in-memory spelling of the same thing. A writer that is
flushed as it goes holds at most one row group's encoded bytes, rather
than the whole file.
Implements Movable
- Fields
options: WriterOptionsschema: WriteSchemapending: List[List[UInt8]]: Encoded bytes not yet handed to a sink, in file order.pending_bytes: Intflushed: Int: Bytes already handed to a sink. With pending_bytes, the position.row_groups: List[RowGroup]offset_indexes: List[List[OffsetIndex]]column_indexes: List[List[ColumnIndex]]num_rows: Int64kv: List[KeyValue]
__init__
def __init__(out self, var options: WriterOptions)
position
def position(self) -> Int
Bytes of the file produced so far, flushed or not.
flush_to
def flush_to[S: ByteSink](mut self, mut sink: S)
Hand every finished byte to sink and stop holding it.
write_batch
def write_batch(mut self, arena: ArrayArena, roots: List[Int])
Write one record batch, split into row groups of row_group_size.
The encoded row groups stay in the writer until flush_to,
finish_to or finish; write_batch_to flushes as it goes.
write_batch_to
def write_batch_to[S: ByteSink](mut self, arena: ArrayArena, roots: List[Int], mut sink: S)
write_batch, handing each row group to sink as soon as it is encoded, so the writer never holds more than one of them.
finish_to
def finish_to[S: ByteSink](deinit self, mut sink: S) -> WrittenFile
Write what is left — row groups not yet flushed, the page indexes and the footer — to sink, and return the footer that was written.
finish
def finish(deinit self) -> List[UInt8]
Close the file and return its bytes — those not already flushed.