1. RL

Our first plan for the dragons game was reinforcement learning, because the NYPC winner had used it. It lived in syeyoung-rl and its game submodule dragons-env, and it ran from the first commit on 2026-09-22 at 20:09 to the last training commit on 09-27. On 09-28 a scripted bot beat the policy 79-1-0, and that was the end of RL for this project.

The design documents state what the policy was supposed to be. The implementation spec says: “Strategies must emerge from team-outcome RL; no tactical teaching curriculum.” COMPETITIVE_RL_DESIGN.md lists the behaviours we hoped would emerge (harvesting circuits, body blocking, selective splitting, cheap head trades, recovering corpse pearls), and calls them “hypotheses to test, not rules to hard-code into the actor”.

The simulator

The rules lived only in the official wasm engine. Driving it in-process gave about 20k dragon-turns per second on one thread, which is slow for self-play. On 09-24 dragons-env was rewritten as a vectorised C++ port of the engine, checked byte for byte against the unswbc 1.0.0 wasm build. On one 14-core machine, with every thread busy, it stepped 7.7M dragon-turns per second. It could also hand a critic a summary of the true board, which no bot ever sees.

The model

The policy runs once per dragon per turn. Our C++ layer turned the protocol text into what that dragon remembered (memory is private, and a child created by a split starts with none), then packed it into fixed-size inputs:

• a 17x17 window of the remembered map around the head: for each tile, when it was last seen, pearls, bed countdowns, who occupies it, the kind of each edge and where its portal leads;
• an 8x8 coarse grid summarising the whole remembered board: per cell, how much was seen, pearls, beds, allies, enemies, kelp and portals;
• up to 49 visible dragons, the last 64 sonar words received, a few status values and the previous command.

Each input has its own small encoder: convolutions and per-tile MLPs over the window, MLPs with mean and max pooling over sets such as the visible dragons and the inbox. The outputs are joined into 512 numbers and fused to 192. A GRU with a 192-wide state is the dragon’s memory between turns. It lives as long as the dragon does, and a split child starts from zero. The final actor had 675,798 parameters.

The decoder builds a command in steps, each conditioned on the last:

First it picks MOVE or SPLIT. A split scores every legal size. A move is a sequence of up to 8 direction tokens: a small GRU proposes one step at a time, scoring north, east, south, west and stop from the encoded tile in each direction, and walks its own remembered map, portals included, to know where the next step starts. Several tokens make a sprint. Last, each of the four sonar directions picks one of 257 options, silence or one of 256 words. The vocabulary was fixed, and what the words meant was left for training to work out.

The critic was used only in training. It saw the true board as an 8x8 grid, every living dragon with a summary of its actor state, and both teams’ lengths, and predicted the result for one team. It had 325,217 parameters.

This was the third design in a week. The first (09-22) gave each dragon a 42-channel memory map of the whole board, read by a multiscale ConvGRU that passed messages between the two ends of each portal. The second (09-25) used graph message passing. Both were replaced because each decision cost too much, not because they had been shown to play worse.

Training

Training was PPO with self-play on the four RTX 2080 Ti GPUs of a lab server, poscatgpu2.

JAX was used only to define the model and export it as compiled programs, one to act and others for gradients. A single C++ program, dragons_train, ran them through PJRT. Two collection workers per GPU each stepped hundreds of matches in our engine, batching every waiting decision into one GPU call per step. Finished 500-round matches went to a queue, were replayed to rebuild their observations, and fed the learner, which computed advantages, ran the gradient programs on all four GPUs and stepped Adam on the host. New weights went back to the workers. Half the matches were self-play, with both sides learning; the other half were against a snapshot from a league saved every 20 updates. The maps were freshly generated, 7 to 64 tiles a side, with the 11 contest maps making up a tenth of the pool.

Each update used at least 65,536 decisions: one epoch in four minibatches, a clip of 0.1, a stop when the policy moved more than 0.02 in KL, and Adam at 3e-4. There was no GAE. Every decision of every dragon on a team got the same final return G, minus the critic’s value at that point, plus that dragon’s own shaping rewards discounted by 0.95 per decision. In the last version, G was the result plus half the longest-dragon margin plus a quarter of the total-length margin. The shaping paid +0.05 for a pearl and +0.1 for an enemy that died ramming our body, and a death cost 0.1 plus 0.05 per segment. An earlier version paid 0.3 per ram, and the policy spent its dragons ramming.

Deploying it

A submitted bot gets 100M wasm CPU points and 48 MB of memory a turn, in a zip of at most 4 MB. The policy was rewritten as plain C++ with only the standard library, doing its arithmetic in half precision where the trained model did; it matched the JAX model’s commands on 1,199 of 1,200 test decisions. The 2.7 MB of weights first shipped as text, but decoding them blew the CPU budget, so they were compiled into the binary. A decision then cost about 23M points at the median. That policy, at update 28193, was our second server submission, v2.

What it cost

syeyoung-rl has 150 commits across its branches. By day: 10 on 09-22, 18, 12, 43, 47, 15, and 5 on 09-28. Most of it went into making training fit on the GPUs: the 105 commits from 09-25 to 09-27 are mostly throughput and memory fixes. We never recorded how long the deployed run trained.

The README’s “Not done” section lists where this left us: “Any training run long enough to evaluate, ablations or promotion gates.” No RL run was ever evaluated against outside bots.

The scripted bot

At 00:31 on 09-28, under five hours after the last reward change, a commit added a scripted C++ bot in its own strategy/ folder. By then RL wasn’t looking promising, and we had decided on a loop of LLM agents instead; this bot was something for the loop to start from. The commit describes it as: “Explore, gather with many small dragons, feed them into leaders late, trade head-to-head when it pays.” It had a board memory (tiles, beds, kelp, portal pairs), breadth-first pathing with dead-end checks, value-per-round targeting and splitting out of traps, and it shared an agent interface with the RL policy so the two could play each other in the simulator.

strategy/EXPERIMENTS.md records every run, all against the RL policy. “Pool 40” is the first 40 maps of the RL training pool, played from both sides (80 games); “official” is the 11 contest maps (22 games).

Change	Maps	W-D-L
first version	official	10-0-12
when trapped, split and walk the child out	official	18-0-4
sweep baseline (split from length 8)	pool 40	69-0-11
split from length 6	pool 40	74-0-6
survivor fleeing turned off	pool 40	78-0-2
split from length 4	official	22-0-0
split from length 4	pool 40	79-1-0

The first version lost on small maps. The file’s own finding on the remaining losses: they “were small crowded maps (arena, Colosseum) where the RL swarm wins the one-for-one head-to-head attrition. Splitting at 4 wins the count race there; fleeing hurts.” The 79-1-0 configuration was committed at 01:31 on 09-28, an hour after the bot’s first commit. It became strategy-v1 and was submitted as server v3.

By the afternoon of 09-28 the last commit in the repository was a rule that strategy configurations are committed code, “results are recorded by the league runner”. strategy-v1 became the first variant in the league described in the next stage, and the league’s design document says in its opening section: “Nothing here depends on RL (abandoned 2026-09-28).”