Interop

Arrow through shared memory, between two processes

19 September 2026 · memory-region.mojo

Arrow’s C Data Interface hands a consumer pointers, so it stops at the process boundary: a pointer means nothing in another address space. A shared memory mapping allows you to use different processed on the same machine. Since Arrow’s buffers are already in their final layout, a consumer that maps them can use them directly.

Performance characteristics

Let’s use Daft to read one column of a 79,478,796-row Iceberg table. These are run on an Apple M4, warm cache, p50 of five reads.

where your consumer is	use
the same process	the C Data Interface	89 ms
another process, same machine	shared memory	112 ms
another machine	Arrow Flight	149 ms on loopback

Crossing a process on the same machine costs about 1.3×. Flight costs about 1.7×, and buys a crash boundary, a retry boundary and a network you can point somewhere else.

A shared memory mapping is only worth it when both ends are on one machine and the producer writes Arrow buffers into it. Writing Arrow IPC into a mapping — the obvious shortcut, since the file format is the memory format — measures 228 ms here, slower than streaming the same bytes over Flight.

Communication costs

Let’s look just at the communication costs for one column, already Arrow, with no Parquet in the path:


mapped, read where it lies	25 ms
read into the heap	56 ms

This is good news for the consumer. The producer has to do more work: it has to get the rows into the mapping, and a mapping’s create, size, map, unmap and unlink cycle costs 4.01 ms per 8 MB batch against 3.05 ms for the copy inside it.

To really unlock the benefits you have to use one mapping per unit of work, not one per batch.

How

Two tins. memory-region.mojo is a region — heap or mapping — and a bump allocator over it whose claim returns an offset, never an address, because the mapping lands somewhere different in every process that maps it. arrow-mlake.mojo’s export_shared_into writes a batch’s buffers into one and describes where each landed.

iceberg.mojo ships both ends of a working example:

printf '%s\n' "$tickets" \
  | ib-shm-publish <table-dir> <split-bytes> <columns> <dir> \
  | ib-shm-consume

ib-shm-publish scans a split, writes its buffers into one mapping and prints a manifest of offsets per batch as each lands. ib-shm-consume maps and folds them. Either end can be something else: a Python consumer is about twenty lines of pa.foreign_buffer and Array.from_buffers, which is what pyarrow-flight.example does to feed Daft — pixi run transports reproduces every number above.

What to know before you choose

The consumer keeps the mapping. Its arrays point into your bytes, so the producer cannot reuse or truncate that file until the consumer is done with every array built on it. There is no signal for that. Give each unit of work its own mapping and let the consumer unlink it.

Nested columns are not supported yet — primitive, utf8 and binary only. A manifest that describes a tree is a bigger thing than the one here.

The producer still copies once. Its decode wrote the buffers, and publishing moves them into the mapping. Removing that too means the decode allocating inside the mapping, which is worth about 4 ms of a 28 ms producer — less than using one mapping per split rather than per batch already bought.