3. The arena and vast.ai workers

On 10-02 we told the orchestrator to stop running our own league and use the team arena instead: tokenz-arena, built the evening before (first commit 10-01 20:35, 129 commits in all, on GitHub). From then on, every verdict on every bot came from it. An import agent loaded 209 of our old local variants as drafts on the first day.

We built it because our teammates wanted to work on the bots too, and the local league from the previous post lived on one laptop. That league’s simulator had also disagreed with the official judge, so the arena was built not to have one. Its design doc keeps two lessons from the league: a child against its parent is the cleanest signal, and local wins aren’t server wins.

The design

The first principle in tokenz-arena’s DESIGN.md is “the official engine is the only judge”. Every game is unswbc run <map> A B --sandbox --seed S at the toolkit version the competition server runs. There are no fast simulators. The rest follows from wanting agents to use it without help:

• One HTTP API. Upload a bot as a zip, wait for its verdict, read its report, download replays and sources. An agent needs only curl. The same endpoints are exposed as MCP tools at /mcp, and /llms.txt explains the whole API on one page with live numbers. docs/AGENTS.md was the brief we handed to worker agents.
• A submission is an experiment. It names its parent and carries a note: the hypothesis and what changed. /ideas searches the notes paired with their verdicts, so an agent can check whether its idea already lost before writing it.
• No auth. The arena is only reachable on the team’s Tailscale network, and it asks Tailscale who is calling. Whatever runs on a teammate’s laptop submits as that teammate.

How it fits together

The arena is a Python (FastAPI) service split into two processes since 10-02. The engine is the only writer: it schedules games, runs 20 local judge slots on the lab server, hands games to remote workers, checks new submissions and refits the ratings. The web process serves the pages, the API and MCP from a read-only copy of the state that is at most a second old, and forwards anything that writes to the engine.

The scheduler keeps a queue in priority order: draft games first, then placement for newly promoted bots, then duels, then holdout games among the top 4, then continuous games that fill idle slots with the least-played, closest-rated pairs. Within a priority, the owner with the fewest running games goes first, so one teammate’s burst of drafts couldn’t starve the others.

A remote worker asks for games and gets a lease of 120 seconds, renewed by a heartbeat every 30. It sends back the result, the replay and the game’s metrics. A lease that runs out puts the game back in the queue, and a result for an expired lease, or for a game already recorded, is dropped, so no game counts twice.

Ratings are Bradley-Terry over ranked games among current submissions, with a draw counted as half a win, on an Elo-like scale with a reference bot fixed at 1000. At first they were refitted after every game, which made the API take 10 to 20 seconds to answer; from 10-04 they were refitted at most every 15 seconds.

State was plain JSON, one file per submission and per game, written with an atomic rename. That held until 10-09, when the web process was reloading every game file every two minutes and holding two copies of all of them in memory, about 7.6 GB. Games moved into SQLite that day.

The life of a submission

tokenz submit zips a folder and uploads it with its parent and note. The engine checks it: a safe zip under 4 MB, no reads of environment variables, a compile with the judge’s own toolchain into the metered wasm the server uses, and one sandboxed game against a reference bot. A failure rejects it with the log.

A draft then plays its parent and the current top 3 on the official, tournament, server and custom suites, from both seats, unranked. The design target was a verdict in about 5 minutes. beats_parent is simply more wins than losses against the parent, reported with a 95% interval and the record on server maps alone. Promoting is a deliberate step by the owner or agent: it retires the bot’s previous version and starts placement, first against the parent and top 5, then against every current bot. The top 4 also play the holdout.

The map suites copied the league’s scheme: official maps, tournament maps rebuilt from replays, a server suite with the first two seeds the server used on each map, a weekly rotating set of generated maps, and the hidden holdout. From 10-05 the generated suites also drew on our own map generator (tools/mapgen), including “trick” maps built to punish one habit, such as rushing the centre.

The server bridge was the arena’s window on the competition server. Once a minute it imported our new server games as arena games, with the opponent as an external bot server:<team>. It also held an auto-submit gate. A candidate had to be #1 on the arena with its rating clearly above the live bot’s, win at least 55% of at least 60 games against it, be no worse on the holdout and server suites, have no errors, and stay under 70M CPU points at the 99th percentile; the gate uploaded at most once every 4 hours. On 10-02 a force-upload button was added that skips it. Both paths were used. The gate itself uploaded queen-guard-7, ring-plus-4, ring-plus-9, ring-sched-8, ring-sealed-2 and ring-shield-3; the rest went up through the button.

Remote workers

From 10-02 any machine with Docker could join (docs/WORKERS.md). The worker image, about 420 MB, held the competition’s toolkit, the C++ judge, the native metrics and Tailscale. On start it joined the team network with a key that could reach only the arena, and every call carried its own worker token. A worker on a different toolkit version was refused, so its results never mixed with the ratings.

A worker ran many games at once, kept a few leased games ready so no slot sat idle, downloaded each bot’s prebuilt wasm instead of compiling it (that took a game from 60 seconds to 10), and cached bots and maps by hash. An error caused by the machine, not the bot, didn’t count as a result: the game went back in the queue until it had failed on three different machines.

The vast.ai scaler

The same evening the scaler (arena/scaler.py, docs/SCALER.md) started renting workers on vast.ai. Our notes record why vast.ai: about $2.5-4.5 per 1000 thread-hours, against $15-20 on AWS Spot. Every minute it read the queue, worked out how many judge slots it would take to clear the backlog in about 5 minutes, subtracted the slots we already had, and, if over 200 games were waiting (500 before 10-05), rented one machine. It searched for verified hosts with at least 64 threads, picked the cheapest per slot, and bid 15% over the asking price for an interruptible instance. The container joined Tailscale by itself and started leasing. A machine idle for 10 minutes, outbid, or no longer needed was returned, and a worker that lost contact with the arena for 20 minutes destroyed its own instance.

The caps we approved were $1 an hour, $10 a day, 6 machines and 4 rentals an hour, with a prepaid balance as the last guard. We didn’t record what it spent in total. What the snapshot does record: 264 rented machines played games between 10-02 and 10-09, and they played 182,013 of the arena’s 291,668 games, 62%. The busiest played 9,292 games in about an hour.

Its early bugs, mostly on the morning of 10-05, were the kind an interruptible market invites: two of our instances on one host outbidding each other, the same flaky cheapest host rented and lost four times in 8 minutes, and wasmtime starting a thread pool as large as the host for every game until containers hit their task limit. Each got a rule: one instance per host, a 2-hour skip list, two threads per judge. One fix was deployed to the live arena without approval, and from then arena changes needed explicit permission.

The C++ judge

unswbc run --sandbox spends most of a game in Python glue around the wasm: turn framing, the bots’ system calls, the engine’s host calls. On 10-02 and 10-03 we ported that glue to C++ (judge/, docs/CPPJUDGE.md), about 2,200 lines, keeping everything that decides a game, the metered bot builds and the engine wasm, exactly as the official judge has them. It produces the same replay byte for byte and used 1.6x to 7x less CPU. Whenever it couldn’t be exact (a Python bot, a missing binary, a toolkit release whose ported files changed) it handed the game to the official judge, and one game in 50 was played by both as a spot check, with the official result counting. It played 225,633 of the arena’s games. All 4,535 spot checks matched.

Duels and ephemeral bots

From 10-04, workers could try a change before submitting it (docs/DUELS.md). An ephemeral bot is a zip checked like a submission but kept off the ladder, deleted after 24 hours. A duel plays any two bots on fresh random seeds (random:N), both seats, on any suite except the holdout. The fresh seeds keep a bot from being tuned to the ladder’s fixed ones. The result reports W-D-L with a 95% interval and whether it excludes 50%. Duel games are stored apart, so nothing official can see them.

The end

The last arena game was played on 10-09 at 05:57Z, when we stopped all workers. 10-09 was the submission day. By then it had played 291,668 games between 680 submissions, plus 629 duels with 30,656 games. Its replays came to about 196 GB zipped.

On 10-11 it became a static site, the arena snapshot (repo Battlecode-UnderTheC/arena). We published it so that this series could point at the games it describes. The pages are the arena’s own website, with the backend ported into a service worker that reads one SQLite file through HTTP range requests. The replays aren’t stored. A game is deterministic given its map, seed and both bots, so the game page runs the official engine wasm (unswbc 1.2.9) and both bots’ metered builds in a port of the judge’s sandbox, one Web Worker per dragon, refilling 100M CPU points per turn. The toolkit’s replay viewer shows the game as it runs and says whether the result matches the arena’s record.