magmalake

Performance

Measured, including where it loses

I re-measured all of this on 5 September 2026, on an Apple M4 — ten cores, four of them performance — with nothing else building on the machine. Same machine, same files, and every row reproducible from its own repository. magmalake beats pyarrow on flat columnar Parquet reads, on one core and on four. It is at parity on writes. It loses on nested and mixed data, and an Iceberg scan is still about 1.3× behind pyarrow at matched thread counts. Those rows are in the table too.

Almost every number here is a p50 over real iterations, with the p90 beside it where the spread is the point — not a mean, which one slow sample can move by 30%, and not a best-of-N. The two repos whose benchmarks are not on that harness yet, iceberg and objectstore, say so in their own rows. Each benchmark is compiled and then run: JIT-executing a suite with mojo run inflated one of these figures by 1.9×, and comparing a JIT number against a compiled one produced two confident wrong diagnoses before that was found. Rows are taken per benchmark rather than from one pass over a whole suite, and the harness times a reference kernel either side of every benchmark and reads the load average, so a run taken on a busy machine says so instead of quietly reporting inflated numbers. Where a row measures a C library rather than Mojo, it says so.

“Single core” is no longer a property of this stack. ParquetReader.num_workers and ScanOptions.num_workers both exist now, so every row carries its own thread count — and so does every pyarrow reference, which is quoted either as one thread (set_cpu_count(1), set_io_thread_count(1), use_threads=False) or as threaded, pyarrow's own default. On the 1M-row file those two legs are 2.9× apart, so a comparison that does not name the leg is not a comparison.

Table and file formats

Flat columnar Parquet is where the native stack pays off. Nested and mixed data is where it does not, and an Iceberg scan is still behind pyarrow at matched thread counts — both are in the table.

Operation magmalake Reference
Parquet read, 1M rows — 1 core int64, double and two dictionary columns, uncompressed, four row groups 3.77 ms p50 / 3.83 p90 — 265M rows/s pyarrow, one thread: 8.0 ms — we are 2.1× faster
Parquet read, 1M rows — 4 workers the same file, ParquetReader.num_workers = 4 1.96 ms p50 / 2.07 p90 — 510M rows/s pyarrow, threaded across all 10 CPUs: 2.57 ms — we are 1.3× faster while using four threads to its ten
Parquet read, worker scaling the same file at 1 / 2 / 4 / 8 / 10 workers 3.77 / 2.42 / 1.96 / 1.90 / 1.90 ms it bends at four, which is how many performance cores this M4 has. Past four the p50 buys about 3% and the p90 gets worse — 2.07 ms at four workers, 2.50 ms at eight.
Parquet read, 100k rows, nested and mixed int64, double, string, bool and a list<int32>, snappy — 1 core / 4 workers 2.63 ms / 0.98 ms — 0.78 ms at eight workers pyarrow 2.24 ms one thread, 0.66 ms threaded — 1.17× and 1.18× faster than us. Still the file we lose, but by a fifth rather than by half.
Parquet write, 1M rows 32.8 ms pyarrow 31.7 ms — parity, inside the run-to-run spread
Parquet footer 1,000 columns × 50 row groups = 50,000 column chunks 56.6 ms read / 2.6 ms write —
Iceberg scan, 1M rows — 1 core / 4 workers zstd, six columns; best-of-three warm, not the percentile harness 36.3 ms / 11.3 ms pyarrow on the same four data files: 27.3 ms one thread, 8.8 ms threaded — about 1.3× ahead of us at either thread count. PyIceberg 0.11.1 does the whole scan in 8.0 ms on pyarrow's pool.
Iceberg scan, nested columns, 200k rows a struct and a list column, 1 core 10.1 ms PyIceberg 4.0 ms on pyarrow's thread pool — 2.5× faster. Nested reconstruction is the slowest path we have: on the mixed Parquet file above, Dremel assembly into Arrow buffers is 30% of the read and decompression another 25%.
Iceberg scan, 2M rows over eight files 1 / 2 / 4 / 8 workers, to_batches 71.2 / 38.1 / 21.8 / 16.5 ms — 4.3× at eight eight files give the workers more to divide than the four-file table above; PyIceberg reads the same table in 15 ms
Iceberg append, 1M rows data files, manifests and commit 192 ms PyIceberg 165 ms, writing three times the Parquet bytes
Iceberg scan planning, 500 manifests 21.2 ms for 2,000 file tasks 31.5 µs fixed per manifest, 11.6 µs of that the file read itself
Avro decode, manifest-shaped records the schema-compiled cursor, 1 core 18.1M records/s — 25.9M with field selection fastavro 1.12.2 on the same file: 1.70M — 10.7× faster
Avro inflate 1.0 GB/s —

scroll the table sideways →

Primitives

Codecs, hashes and threads — the layer everything above is only as fast as. Single core except where a row counts workers.

Operation magmalake Reference
SHA-256 pure Mojo with ARMv8 crypto intrinsics; a single timed pass, median of three 2.70 GB/s — 590 MB/s scalar fallback OpenSSL 3.17 GB/s on the same 64 MiB, in the same process
zstd / lz4 / brotli decompress FFI — these measure libzstd, liblz4 and libbrotli, not Mojo 11.1–32.9 / 13.5–20.2 / 2.4 GB/s the zstd and lz4 ranges span compression levels and compressible versus random input
snappy decompress pure Mojo, 64 MiB 3.15 GB/s compressible, 31.8 GB/s incompressible the incompressible path is essentially a memcpy and swings like one — 17.5 GB/s at p90
CRC-32 / murmur3 / XXH64 pure Mojo, scalar 1.42 / 1.61 / 1.34 GB/s —
Roaring bitmap, 10M random values add / serialize / deserialize 1.30 s / 22.2 ms / 60.3 ms the add figure was published as 5–11 s until the RNG was moved out of the timed loop
threads: spawn and join 500 sequential pthread round trips ≈11 µs warm 15–20 µs for the first threads a process spawns, so a pool pays that once rather than per task
threads: parallel_for over 100M Int64 1 / 2 / 4 / 8 / 10 workers 2.3× at four workers, 3.5× at ten a parallel memcpy on the same machine reaches 3.9× at two workers and never improves — that is the memory-bandwidth ceiling

scroll the table sideways →

Object storage

Measured against MinIO on the same machine, so the network is not the story. This repo's bench is a single timed pass rather than the percentile harness; these are the median of three runs.

Operation magmalake Reference
S3 multipart upload, 16 MB 8 MB parts, each signed with SigV4 396 MB/s the same object comes back down at 2.3 GB/s — the upload is what pays for hashing and part framing
HTTP range read, pooled connection 200 × 64 KiB, one request each 0.147 ms local / 0.495 ms signed S3 —
The same 200 ranges, coalesced adjacent spans asked for at once — the shape a Parquet scan actually has 0.022 ms local / 0.029 ms signed S3, per range 6.7× and 17× cheaper per range than asking one at a time

scroll the table sideways →

Benchmark history

The table above is hand-measured; these are not. Those numbers were taken in one sitting on the M4 named at the top, which is why they carry a date rather than a build number. Every merge also runs the same benchmarks on a GitHub runner and appends to a history each repository publishes itself:

Expect those to be slower and noisier than the table — a shared runner is not a workstation. They are worth reading as trends, not as absolute figures: each history is keyed by machine, so a run on a laptop and a run on CI stay separate series and are never averaged, and a benchmark is only flagged as regressed when it falls further than its own recent variance.

Reproducing any of this

Every tin carries its own benchmarks and runs them through the same harness, bench.mojo. Clone the repo a row belongs to and run: 

git clone https://github.com/magmalake/parquet.mojo
cd parquet.mojo

pixi run -e bench bench                    # the table, with spread
pixi run -e bench bench -- --json          # every repetition, machine-readable
pixi run -e bench bench -- --list          # what this repo measures
pixi run -e bench bench -- --only bench_read_wide

The bench environment is stable Mojo 1.0.0: a precompiled Mojo package is stamped with the compiler version that built it, and every tin builds with 1.0.0. The libraries themselves test on stable and nightly.

Two caveats on reproducing the numbers above exactly. The tins currently pin bench.mojo 0.2.0, which prints the same p50 and p90 but not the machine: steady verdict — that arrived in 0.4.0, which is what these rows were taken with, and the tins have yet to be moved onto it. And the worker ladders contend with each other if you run them together, so take them one at a time: --only bench_read_wide_w4.

A few benchmarks need a fixture that is generated rather than committed — parquet wants python tools/bench_pyarrow.py --make first, iceberg builds a warehouse, objectstore needs MinIO. Each repo's README says which, and the bench stops with instructions rather than measuring the wrong thing. Expect your absolute numbers to differ from an M4; the ratios are what travel.

Threads, for now, come from a tin

Mojo's standard library currently has no threading. parallelize was removed and there is no thread pool reachable from user code, so every number above that mentions more than one core goes through magmalake/threads, which wraps pthreads directly via external_call.

That tin is inspired by and distilled from flare — Ehsan Mokhtarian's project, MIT-licensed — which worked out how to get real OS threads out of Mojo in the first place. threads.mojo keeps the parts a data stack needs and drops the rest.

It is explicitly a stopgap. When the language ships its own threading, this tin should stop existing.

How these numbers were found

Four optimisation passes. I profiled before each one instead of guessing, which was just as well — not one of the bottlenecks was where I would have looked first.

  1. A dlopen on every single decompress call About 450 µs per call, spent opening a library that was already open.
  2. SHA-256 running 45× off what the hardware can do The scalar implementation was correct and slow; the CPU had crypto instructions sitting idle.
  3. Avro boxing roughly 60 allocations per record And JSON schema parsing turning out to be 72% of the cost of reading a manifest.
  4. 500 byte-identical manifest schemas, each parsed from scratch Caching the parsed schema and the plan is what puts scan planning at 21 ms; it was three times that before.