Getting started

Develop locally

For changing Orca, Dista, Iceberg, Parquet or the census and running the change on a laptop, macOS on Apple silicon or Linux, before anything is released or deployed. Nothing on this page touches AWS. Every command was run on a Mac at the revisions CI pinned on 2026-11-06; timings are from that run and will differ on yours.

Lay out the checkouts

You need pixi and mojoshelf's CLI, which gives pixi shelf and shelf release. Point TMPDIR at a disk with room: builds, and the throwaway Postgres and MinIO servers the tests start, write there.

curl -fsSL https://pixi.sh/install.sh | sh
pixi global install --channel https://mojoshelf.org/channel mojoshelf
export TMPDIR=/Volumes/Big/tmp     # somewhere with tens of GB; the default is fine on a roomy disk

Clone the repositories side by side in one directory. Orca's tests put its siblings on the include path as -I ../dista.mojo/src and so on (DEP_INC in its pixi.toml), so the directory names matter. Which siblings an environment needs:

CI checks each out at a pinned commit, and the tests mean what CI means only at the same ones. The data job of orca's .github/workflows/ci.yml lists every sibling that environment needs with its commit, so read the list from the file instead of copying it from here, where it would go stale. In 16 seconds:

mkdir lake && cd lake
git clone https://github.com/magmalake/orca.mojo
sed -n '/^  data:/,/^  ci:/p' orca.mojo/.github/workflows/ci.yml \
  | grep -o 'repository: [^,]*, ref: [0-9a-f]*, path: [^}]*' \
  | while IFS=', ' read -r _ repo _ ref _ dir; do
      git clone -q --no-checkout "https://github.com/$repo" "$dir" && git -C "$dir" checkout -q "$ref"
    done
git clone https://github.com/magmalake/mojo-census

The loop calls its directory dir because zsh ties path to PATH: read ... path empties the command search path, and every later command is "not found". For the nightly environment, also clone flare_mlake and json_mlake at the commits of the nightly job's steps (grep -A3 'magmalake/flare_mlake' orca.mojo/.github/workflows/ci.yml). The census needs none of this: it takes Orca and every library as pixi packages, pinned by commit in its pixi.toml, and builds on its own (until you test a library change in it).

Some of Orca's tasks are defined in more than one environment, which makes a task that depends on one of them ambiguous: pixi run test stops with "the task 'build-shared' is ambiguous". Name the environment, as CI does: pixi run -e default test.

Two things about the size. A checkout of a tin that the environment also installs as a package is shadowed by the package: in the data environment an edit to iceberg.mojo's source is compiled in, and one to zstd.mojo's, lz4's, brotli's, sqlite's, postgres's or objectstore's is silently ignored. Test a change to those with a pin, as below. And pixi-build leaves its build trees in .pixi/bld: after the first pixi run, 5.7 GB in Orca's default environment, 9.5 GB once data exists, and 18 GB in the census, beside 14 GB in pixi's cache (~/Library/Caches/rattler on macOS). The environments themselves are about 1 GB each.

Orca on its own

The examples

The first pixi run installs the environment: a minute. Each example then builds in 10 to 15 seconds, and the orca command in 45. A pipeline runs as a plain process and prints its run, with no database or scheduler:

cd orca.mojo
pixi run toy && ./build/toy run --partitions 2024-01..2024-03
pixi run fanout && ./build/fanout run --partitions 2024-01      # steps calling Dista tasks
pixi run daily && ./build/daily run --partitions 2024-02-27..2024-03-02

With no ORCA_DATABASE_URL a run is kept in memory. Give it a SQLite file and the run is recorded and resumable, with no server:

export ORCA_DATABASE_URL=sqlite:build/runs.db
./build/toy run --partitions 2024-01..2024-03
./build/toy status
unset ORCA_DATABASE_URL

For Postgres, scripts/with-postgres.sh starts a throwaway server on a Unix socket under $TMPDIR, exports ORCA_DATABASE_URL for the command you give it, and removes the server when the command ends. The server comes from conda-forge's postgresql, so run it through pixi:

pixi run scripts/with-postgres.sh sh -c './build/toy run --partitions 2024-01..2024-03 && ./build/toy status'

The playground

To look at the UI before running anything real, the playground builds the examples, starts a throwaway Postgres, makes runs that succeed and fail, and runs two fanout runs against a stand-in EC2 that is a local Python process. It is ready in about a minute and a half. Log in to the UI's actions as demo / demo-password. Ctrl-C ends it and removes everything:

ORCA_UI_ADDR=127.0.0.1:8275 pixi run playground   # http://127.0.0.1:8275

The scheduler and the UI

The scheduler fires schedules, backfills and sensors, and orca ui serves the runs. Both need Postgres. This one lasts between sessions, in build/pg with its socket in build/:

pixi run orca
pixi run initdb -D build/pg --auth=trust --username=postgres -E UTF8 --locale=C
pixi run pg_ctl -D build/pg -l build/pg.log -o "-k $PWD/build -c listen_addresses=" -w start
pixi run createdb -h "$PWD/build" -U postgres orca
export ORCA_DATABASE_URL="postgresql:///orca?host=$PWD/build&user=postgres"
export ORCA_UI_ADDR=127.0.0.1:8275

./build/orca register ./build/toy ./build/daily
./build/orca scheduler > build/scheduler.out 2>&1 & echo $! > build/scheduler.pid
./build/orca ui > build/ui.out 2>&1 & echo $! > build/ui.pid
./build/orca backfill daily 2024-02-27..2024-03-02
./build/orca status daily          # five runs queued, done in a few seconds

The watchdog is on by default. orca watchdog status prints each rule with its value and limit (launches an hour, cost, idle CPU, failures, stuck runs, the control instance's load), and orca watchdog snooze silences one. Nothing local trips it, since there are no instances to launch. Per-pipeline placement (orca register --placement control, #152) is on main and ships in the next orca release.

orca login and --remote work against the local UI exactly as against a deployed one, and plain http is allowed only to this machine. The password is read from stdin:

printf 'a password\n' | ./build/orca user add alice
printf 'a password\n' | ./build/orca login http://127.0.0.1:8275 --user alice
./build/orca --remote http://127.0.0.1:8275 pause toy
./build/orca --remote http://127.0.0.1:8275 unpause toy
./build/orca --remote http://127.0.0.1:8275 backfill --preview daily 2024-03-03
./build/orca logout http://127.0.0.1:8275

Stop exactly what you started: the two PIDs you saved, then the server. pkill orca would also stop every other checkout's UI on the machine.

kill $(cat build/scheduler.pid) $(cat build/ui.pid)
pixi run pg_ctl -D build/pg stop

The test suites

Every test is a pixi task. Most start their own throwaway Postgres, and each picks its ports from its PID, so suites can run side by side. A task's binaries are cached under build/.mojo-build by the content of their sources, so a second run of a test builds nothing.

pixi run test-steps                 # one suite: seconds
pixi run test-cli
pixi run test-pause                 # a scheduler suite, on a throwaway Postgres
pixi run -e default test-executor   # CI's first shard: 8 minutes
pixi run -e default test-scheduler-ui   # its second: 3 minutes
pixi run -e default test            # both
pixi run -e nightly test-steps      # the nightly compiler (needs flare_mlake, json_mlake)
pixi run -e fmt format-check

The data environment adds orca.tables, which reads and commits Iceberg tables, and is the one that needs the siblings of the table above. Its tests are in three groups, as CI runs them, and individually:

pixi run -e data test-tables             # pinned reads, commits: 31 s
pixi run -e data test-data-tables        # tables, append-files, replace, chaos, lineage, expire, taxi
pixi run -e data test-data-sensors       # sensors and rebuilds; lineage and sensors on MinIO
pixi run -e data test-data-keyed         # partitions named by a column, 20,000 keys
pixi run -e data test-lineage-s3         # one suite over MinIO: 111 s including the install

The tests over object storage run against MinIO, which scripts/with-minio.sh starts on a free port with a bucket orca and the AWS_* variables pointing at it, the way with-postgres.sh does for Postgres. MinIO's open-source repository is archived and dl.min.io serves no binaries, so the data environment installs conda-forge's minio-server (macOS arm64 and linux-64 only). To use it in your own run:

pixi run -e data scripts/with-minio.sh sh -c 'echo $ORCA_S3_BUCKET $AWS_ENDPOINT_URL'

The census locally

The census runs entirely on a laptop, against a slice of the archive. It supports macOS arm64 and linux-aarch64; there is no linux-64 build in its pixi.toml. The first build installs the environment and compiles every dependency, about nine minutes; pixi run test then takes 72 seconds and uses no network.

cd ../mojo-census
pixi run build      # build/orca, the pipelines and build/console
pixi run test

The pipelines on their own

No database. GHARCHIVE_HOURS keeps the download to two hourly files of about 100 MB each, and MOJO_HISTORY_MAX_KB skips repositories over 100 MB, which includes modular/modular. On 2024-06-12, hours 15 and 17, the run below takes about 40 seconds and leaves 165 MB under build/warehouse:

export ORCA_WAREHOUSE=$PWD/build/warehouse
GHARCHIVE_HOURS=15,17 pixi run ./build/gharchive run --partitions 2024-06-12   # 6 s
pixi run ./build/mojo_repos run --partitions 2024-06-12                          # 12 s
MOJO_HISTORY_MAX_KB=100000 pixi run ./build/mojo_history run --partitions @all   # 6 s
pixi run ./build/mojo_activity run --partitions 2024-06-12
pixi run ./build/mojo_versions run --partitions @all                             # 6 s
pixi run ./build/mojo_stats run --partitions $(date +%F)
PORT=8080 pixi run ./build/console      # http://127.0.0.1:8080

The result is 476,227 events, 23 candidate repositories, 4 of them Mojo (NuMojo, Mojonet, mojo-flx and modular/modular, which is skipped for its size), 3 histories with 372 commits and 14 tags, and the console answering with its overview, repository list and JSON under /api/. mojo_stats takes today's date, not 2024's: its partitions start on 2026-11-01. The census's one-off pipelines (gharchive_index, mojo_retool) are for repairing a lake that already exists, and a fresh warehouse has no use for them.

The Orca way

This runs the pipelines as production does, with the scheduler, the sensors and the UI. Start a Postgres under build/, the same one as above. Then scripts/env.sh points ORCA_DATABASE_URL at it, puts the warehouse, logs and clones under build/ and sets the small slice of the archive. A sensor counts only snapshots made after it was registered, so start from an empty warehouse (delete the build/warehouse of the run above), register every pipeline first, then backfill one day:

pixi run initdb -D build/pg --auth=trust --username=postgres -E UTF8 --locale=C
pixi run pg_ctl -D build/pg -l build/pg.log -o "-k $PWD/build -c listen_addresses=" -w start
pixi run createdb -h "$PWD/build" -U postgres orca

pixi shell
. scripts/env.sh
export ORCA_UI_ADDR=127.0.0.1:8275      # 8270 is the default; use another if one is running
for p in gharchive mojo_repos mojo_history mojo_activity mojo_stats; do ./build/orca locate build/$p --root .; done
./build/orca register ./build/gharchive ./build/mojo_repos ./build/mojo_history ./build/mojo_activity ./build/mojo_stats
./build/orca scheduler > build/scheduler.out 2>&1 & echo $! > build/scheduler.pid
./build/orca ui > build/ui.out 2>&1 & echo $! > build/ui.pid
./build/orca backfill gharchive 2024-06-12

gharchive loads the day, which starts mojo_repos for it, whose new repositories start mojo_history and mojo_activity. On the run above all of that took 25 seconds, and ./build/orca lineage mojo.commits traced the histories back through mojo.repos and gharchive.candidates to the gharchive run. The UI shows the same, and build/logs/<pipeline>/<run>/ has the logs. The console from above reads the warehouse these runs wrote. Stop with the two PIDs and pg_ctl -D build/pg stop, as in Orca's section.

What needs the network

• GH Archive (data.gharchive.org): gharchive downloads about 100 MB an hour. Nothing else reads it; mojo_activity works from the tables gharchive wrote.
• The GitHub API: mojo_repos asks about each candidate and mojo_stats about each repository, 60 requests an hour without a token. Unauthenticated, mojo_repos checks 25 candidates a run and records the rest as unchecked for the next run; with GITHUB_TOKEN set (a token with no scopes will do) the budget is 1000 and the limit 5000 an hour. To check the rest of a day after the limit resets: ./build/orca backfill --rerun mojo_repos 2024-06-12.
• github.com: mojo_history and mojo_versions clone each repository.
• Nothing: building and testing once the environment is installed, mojo_activity, the console on a local warehouse, the scheduler, the UI and Postgres.

pixi run test-daily compares gharchive's commits of a day (one snapshot of each index table) with a build from before that change: it needs build/gharchive-main, built from the commit before 8922605, and would fail against today's main, which already commits daily. It downloads three hours and takes about 50 seconds, and is for changes to how gharchive commits.

A library change in the census

To see a change to Orca, Iceberg or Parquet work in the census before releasing it, build the census against your checkout. The census pins every tin by commit in [dependencies], and orca-mojo, which pixi-build compiles from source, pins the tins it is built from in its own [package.host-dependencies] and [package.run-dependencies]. pixi solves them as one environment, so a tin must have one source, the same URL and commit, everywhere in that closure. Two commits of one tin fail the solve:

failed to solve the environment
encountered duplicate records for iceberg-mojo-0.10.0-h60d57d3_0_source.conda

That is why a release of one tin is a release of every tin that pins it, and why the same rule applies to a trial. Three cases, easiest first.

A change in Orca alone. Point the census at the checkout, as long as the checkout pins the same commit of every other tin as the census does (a checkout of Orca's main does at the time of writing). Locking takes 2 seconds, and building takes a minute, since pixi compiles orca-mojo from the checkout:

cd mojo-census
sed -i.bak 's|^orca-mojo = .*|orca-mojo = { path = "../orca.mojo" }|' pixi.toml
pixi lock
pixi run build-orca && ./build/orca --help | grep watchdog   # a command main has and 0.8.0 lacks

A branch pushed to GitHub, for a change someone else will test or that CI builds. Pin its head commit with rev, not branch: a git dependency without a rev stays at whatever pixi.lock first resolved, and a branch moves under it.

sed -i.bak "s|^orca-mojo = .*|orca-mojo = { git = \"https://github.com/magmalake/orca.mojo.git\", rev = \"$(git -C ../orca.mojo rev-parse HEAD)\" }|" pixi.toml
pixi run build-orca       # 24 s

A change in a library Orca depends on. Every place that names the tin must agree: the census's pixi.toml, Orca's two package tables and any other tin in the closure that pins it. Find them with grep -n 'iceberg' */pixi.toml in the directory that holds the checkouts, change each to the same pin (a pushed commit, or path = "../iceberg.mojo" written relative to the file it is in), and run pixi lock. A tin left on its old commit anywhere gives the error above. A sibling checkout on the include path is easy to miss, so lock after every change. With the paths in place, a change to the checkout's source is rebuilt by the next pixi run (22 seconds for Iceberg and Orca).

To go back to the released pins, restore the files and the lock; the lock is authoritative, so nothing else changes:

git restore pixi.toml pixi.lock && rm -f pixi.toml.bak
pixi install --locked

What a release costs is a question for shelf release plan, run from the directory that holds the checkouts. It changes nothing and reads the registry, which is not on AWS. A tin's dependents must be checked out beside it, and the plan names any that are not. For Iceberg, with the checkouts above, it says that Orca moves too:

cd ..
shelf release plan iceberg-mojo
# release plan, parents first (2 repositories, 2 tins):
#  1. iceberg.mojo  release/iceberg-mojo-0.10.1    iceberg-mojo  0.10.0 -> 0.10.1  named
#  2. orca.mojo     release/orca-mojo-0.8.1        orca-mojo     0.8.0 -> 0.8.1  pins iceberg-mojo
#     pixi.toml [package.host-dependencies] iceberg-mojo  ab62e59fbf3f -> release/iceberg-mojo-0.10.1 head
#     pixi.toml [package.run-dependencies]  iceberg-mojo  ab62e59fbf3f -> release/iceberg-mojo-0.10.1 head
#     .github/workflows/ci.yml:312  ab62e59fbf3f -> release/iceberg-mojo-0.10.1 head

The census is an application, not a tin, so it follows a release with a pull request that moves its pins to the commits the registry published. The rules, and shelf release apply that cuts the whole cascade, are in the mojoshelf-publish skill, under "Releasing tins that pin each other".

Caveats

• macOS, local mode and an S3 warehouse crash. A pipeline run on its own on macOS with ORCA_WAREHOUSE=s3://... fails with "no workers left: every local worker went away": the driver used libcurl before forking its Dista workers, and macOS aborts a forked child of a process that initialised it (orca #135). It reproduced against a local MinIO, and the OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES stopgap the issue suggests did not get around it. Use a directory for the warehouse. Linux is not affected, and test-lineage-s3 passes on macOS.
• Ports. The UI listens on 127.0.0.1:8270 and the console on PORT. Another checkout's UI may hold 8270, so set ORCA_UI_ADDR and, when a page looks wrong, check whose UI you are looking at.
• Stopping everything. Save the PID of every background process you start and kill those; stop a server you started with pg_ctl -D build/pg stop. A throwaway server stops itself with its command. If a test is interrupted, look for orcapg.* and orcaminio.* directories under $TMPDIR.
• Socket paths. macOS limits a Unix socket's path to 104 bytes. A checkout deep in a long path, or a long default TMPDIR, can break Postgres's socket; keep the paths short.
• Disk. See the sizes above. .pixi/bld is build scratch and can be deleted once the environment is installed (pixi run does not need it again); build/warehouse is the lake you made, and the census's two-hour slice is 165 MB. A day of the archive is about 15 GB to download.
• A busy machine. test-fleet sorts hosts by CPU, and the control instance it lists is your machine. With builds running beside it, the test failed once with "by CPU, highest first"; run alone, the shard passed.

From local to AWS

When it works here, release what changed with shelf release (the cascade above), move the census's pins to the published commits, and deploy with orca aws: the deploy steps, the command's reference, local or AWS? and the runners in Orca's docs, and the magmalake-apps skill's AWS reference. A stack's template changes should reach a running stack by replacing its control instance from a clean boot, which orca #154 is building as orca aws replace-control, not by editing the live instance by hand. Until it lands, rehearse a stack change on a separate stack first. The release and operations routine is in Maintain and release.