Data infrastructure for Mojo, version 0.x
A data lake that lives
natively in Mojo
Mojo can open an Apache Iceberg table today without reaching back through Python or the JVM. magmalake is the set of tins that makes that true — Iceberg and everything underneath it, each independently usable and each checked against the reference implementation of the format it speaks. It is early, and it is meant to grow into the data layer the language does not have yet.
from iceberg import Catalog
var table = Catalog.rest("https://polaris.example/api/catalog").load("db.events")
var rows = table.scan().filter('["=", "region", "eu"]').select(["id", "amount"]).to_table()Metadata, manifests, scan planning, Parquet decode, deletes and deletion vectors, S3 IO and the REST catalog — all of it Mojo.
- 15
- tins on mojoshelf
- v1–v3
- Iceberg formats read
- 211M
- Parquet rows/s, one core
- 0
- JVM or Python in the path
What this is
The start of a data ecosystem for Mojo
magmalake began as a question — now that Mojo has reached 1.0, can one implement a meaningfully large data library in it? — and I picked Apache Iceberg as the target. The answer was yes, and what came out is a stack rather than a demo. Parquet, Avro, Thrift, Iceberg itself, Roaring bitmaps, Snappy and the hashes are implemented in Mojo, not wrapped. A few very mature C libraries are linked rather than reimplemented — libzstd, liblz4 and libbrotli for those three codecs, libcurl for HTTP and TLS, and pthreads for the threads the language does not yet provide. What is gone is the Python and JVM layer that every other route from Mojo to Iceberg has to go through.
Where it stands today: native Parquet decode is faster than pyarrow on a single core; writes and thread-pooled scans still lose, and the language is still missing pieces a data stack assumes. I don't read the misses as limits of the approach — they are the current position on a roadmap, and the measurements say exactly where the line is.
The stack
How they layer
Iceberg sits on Parquet, Avro, Roaring and objectstore; those sit on Thrift, the compression codecs and the hashes; threads runs alongside all of it. I kept them separate, so you can take just the one you need.
The tins
Tins at a glance
15 tins, each independently usable and checked against the reference implementation of the format it speaks. What each one is and what proves it lives on the tins page.
Table format
File formats & storage
Primitives
Cross-implementation oracle
Take a tin
Each of the 16 repositories is independently usable and independently installable. Start with whichever layer you need — you do not have to take the stack.