NES
PLAYER

First encounter with a game

Castlevania, absent from every training run. Instincts score 700 points; the random agent and the transferred network score zero. The agent read the score off the screen itself — cross-checked against emulator memory: 700 = 700.

Few-shot on unseen games

Three episodes on a new game versus training from scratch: Gradius 0.707 → 0.787, Battletoads 0.594 → 0.671. On Balloon Fight transfer hurts (−3.5 pp) — recorded as-is.

Fifty-nine clears are worth no more than six

Eighty more planner runs produced 59 clears, and clones trained on nested subsets of 6, 12, 24 and 59 are indistinguishable: −182 [−415, +34] between the ends of the ladder. The whole distance from zero to about 1900 is bought by the first handful; the next fifty buy nothing.

Retracted 10 September. The first version of this card said −83 [−247, +71]. Those numbers were measured on collapsed data: on macOS cp -R src/ dst/ copies the directory’s contents, so the episodes overwrote each other and every rung trained on a single episode. The conclusion survived the rerun; the measurement did not.

The second and third epochs buy variance, not skill

The obvious next step was demonstrations recorded where the clone fails: the planner takes the wheel wherever the clone stalls, and the rescues go into the training set. That looked worth +278 px on one training run and −240 px on three. Playing every epoch of every run explained it. The mean across six runs does not rise with epochs (1736 → 1753 → 1507) while the spread between runs quadruples, from 340 px after one epoch to 1527 px after three. One clone that scored 1885 after a single epoch was trained down to 563 by the third, while its loss fell the whole way.

At one epoch, where the instrument is steady, the rescues are worth +169 [−107, +445] over six training runs, positive in five of six. Zero is inside, so this stays a lead rather than a result. The first three seeds agreed on +221, +277 and +239, and that agreement is exactly what made it look certain; three fresh seeds gave −355, +382 and +252. Three numbers that agree are a coincidence that happens often.

Not the depth of the network, but the hint about where to look

To price the representation we removed it: a policy with one weight per pixel per action and nothing in between. Three arms, six training runs each, one epoch, the same 59 demonstrations and the same 32 evaluation seeds.

Pixels straight to buttons, 985. The conv net with the attention loss off, 1288. With it, 1641. The representation is worth +656 [+282, +1031], and the attention loss +353 [+180, +526], winning six of six. It is the first result here measured at the training-run level rather than the evaluation seed that survives being measured that way, and the steadiest of the three: supervising where the network looks removes much of the run-to-run spread as well. The loss was added long ago, measured once, and never checked across runs.

The linear policy does not need probing — it can be looked at, because its weights reshape into a picture. And for the sixth time running, fitting the demonstrations best did not mean playing best.

The curve saturates exactly where the default already sat

The weight of the attention loss had never been chosen by measurement: 1.0 was a guess that stuck. Four weights, six training runs each, the same demonstrations and the same 32 evaluation seeds.

0 → 1288, 0.5 → 1493, 1.0 → 1641, 2.0 → 1636. The first half of the weight buys +204 [+67, +342], winning six of six; the second half is not separable from noise; doubling past 1.0 buys nothing and doubles the spread between training runs, 138 against 309.

The same shape as the number of epochs: pushing a term harder than it wants to go widens the lottery more than it moves the mean. Nothing to change — but the default now rests on evidence rather than inheritance, and nothing to change is a result when the alternative was to keep not knowing.

With exact dynamics, the planner finishes the level

Replace the world model with the emulator itself — rewind the console with save states and see where each of six plans ends — and Mario clears 1-1 in 20 runs out of 32 against none for the reactive policy. Median progress 7117 against 787, half the deaths. Not a ceiling but the best measured arm: a longer tail, two-step search and an oracle continuing itself are untested.

One future is not a plan's value

The continuation is played by a sampling policy, so a single rollout is a future that happened, not a value: two scorings of the same plan differed by 22 px while different plans differed by 21. Averaging four continuations moved the planner's median 3121 → 7117 and its clears 12/32 → 20/32 without changing a single candidate. The largest methodological error found in this project.

Noise in the score reverses the sign of a search

A CEM search over macro-actions loses 2041 px on a noisy score and gains 2168 on a clean one — same search, same seeds. The best of thirty-eight noisy estimates is the luckiest draw, not the best plan. Caveat: the comparison changes both the noise and the continuation policy, so the optimiser's curse is compatible with the data rather than isolated by it.

The present, as we can describe it, does not decide the choice

Seven inputs — from two scalars to exact level geometry, sprites, action history and the policy's own probabilities — trained identically and tested on a level never seen. All capture the same ~20% of the planner's per-decision gain; rich inputs only improve picks among near-ties. The useful remainder was reachable only by simulating the future.

Half the simulations, 86% of the gain

The one lever that worked without learning: the oracle at two draws instead of four keeps 86% of its advantage at 54% of the compute (median 5116 against 7117, 16 clears against 20). The compute-progress curve is measured at three points, and the second extra batch of simulations costs twice the first per pixel of progress.

The 2→4 draw step cannot be resolved by 32 seeds

The adaptive rule — escalate where two draws are unstable on the winner and the point's own noise — keeps 94% of the saving at 69% of the cost offline with thresholds frozen on other levels, yet online it showed nothing, twice, at +29% compute. A rerun on common random numbers (every continuation draw seeded by the decision's state, so arms play byte-identical games until a genuine policy difference) exposed why: even always-escalating against never-escalating comes out at +254 [−919, +1441] — the whole segment between the fixed budgets sits inside the noise. The fine economics is measurable only offline, on the stored draw matrices.

Four games, one planner, zero tuning

The same oracle with the same six templates on every dry-land SMB level and three new games: 2-1…6-1 give +509…+2483, Contra (J) +2270, Contra (U) +2134, Super C +1210 — every interval clear of zero, 371 of 384 paired seeds to the planner. On Contra the policy is inert (zero on every seed) while the oracle walks nine screens: B+RIGHT means "run" in Mario and "advance firing" in Contra. Super C had no position in its RAM map at all — the camera was found by scanning RAM in minutes, and the method opens the side-scroller library at a few minutes per game.

Long plans are toxic; three pairs repair almost everything

The control the two-step search was missing: the same five behaviours stretched to monolithic 96 frames collapse the planner — −1849 [−2937, −698] against the 48-frame champion, deaths doubled. An open-loop "run" held for 96 frames is a commitment nothing can rescue. Three hand-written pairs ("run up → jump", "back off → jump") buy back +1563 [+529, +2564] and match the full 25-pair search at a third of the cost — yet the 48-frame horizon replanned every 16 frames remains the load-bearing local optimum.

Contra: level 1 from power-on, 12 of 32

The same planner, 32 paired seeds, 7000 frames. 23 of 32 reach the final wall, 12 clear the level — by the game's own level counter, not a metric, and one clear checked frame by frame: cannons and sensor exploding, the intermission, the "AREA 2" card. What it took: fire by tapping, not holding (read off a causal scan of the console), five jump-timing compositions distilled from the two-step search's choices (the cliff before the wall: plain 5/32, with them 22/32), and the wall's damage from the game's object table — by type, not by address. An earlier claim of "hits" was wrong and was retracted before this.

Contra, level 2: the first room, 17 of 32

A stage that does not scroll. Six objectives in a row gave 0/8 and every one of them fired as designed; the fault was a term silently returning zero — the hero was lost the moment he turned toward the door, and the next sensor appearing counted as a loss. Reading candidate scores per decision found it in ten minutes. With two notes from the manual ("destroy the sensor", "the door is in the middle") and no number chosen by hand, the room opens in 17 of 32 runs with a random enemy phase (CI 36–69%). One crossing checked frame by frame. One room, not the stage: in the next one the soldier parks in a corner again.

A clone trained on the planner's clears plays Contra at last

Distilling the planner's choice failed in five roles, which is measured and still stands. But once the planner began finishing the first level, data existed that never had before: twelve runs played to the end. A clone was trained on them with the same recipe as the old one — only the data changed. Playing from pixels and sound alone, no planner and no rewinding, it goes from a median of 0 to 1952 over 32 seeds, +1719 [+1516, +1895], winning every seed; it runs the bridge, jumps the water, shoots. The twelve training seeds were twelve of the thirty-two evaluated, so separately: on the twenty never trained on, a median of 1996 and +1722 [+1475, +1923], 20 of 20. The caveats: the knowledge is the teacher's, and the teacher is far ahead — it clears the level in 12 of 32 and the clone in none. Accuracy predicted none of it: the clone that plays beats its majority baseline by 5.5 points, the inert one beat its own by 56. The first explanation — that the new commands, the wall-killing diagonal among them, were the mechanism — was tested by ablation and withdrawn: a clone trained on the same clears with the diagonals mapped away plays just as well (−181 [−569, +208]). What carries the gain is state coverage and the structure of trajectories that went far, not the vocabulary. Established so far for one game and twelve demonstrations.

Every low-level numeric input Contra needs is found automatically

Buttons and tap-vs-hold by a causal scan; templates from it; camera or 8-bit scroll by agreement with the picture; object tables with hit points by shooting (hit points count down in small steps, a type is wiped in one lump); the player's sprites by which move with the button; the section counter from the fades between rooms across two runs (one transition lets a free-running timer through). What stays human is the manual: what to destroy and where the exit is.

Rush'n Attack: a game with nothing written by hand

Position from a camera scan (the game keeps only an 8-bit scroll with no high byte; candidates are ranked by agreement with the picture's own shift), templates from a causal button scan (knife on tap, forward = RIGHT). The planner gains +145 px [+124, +166], 32 of 32 paired seeds, one death fewer per run. Price of the game: two scans of a few minutes each and zero human minutes in the loop.

Rewinding loses to compositions

The hypothesis "the mistake was made earlier than 144 frames can see": when every candidate is doomed, rewind the console 16, 32, 64 frames and re-plan with a growing horizon. On Contra's cliff: plain 1/8, rollback 0/8 and 0/8, rollback with a composition window 5/8, two-step everywhere 8/8 at the same cost. The rollback's survivors were stalls; the failure was in the candidate set, not the lookahead. Closed by measurement.

Hearing adds 7 pp

Ablation on four seeds out of four: audio on top of video improves action prediction. Separately, sound↔frame matching hits 25.4% against 1.6% by chance.

Making the model look at objects

Only 12.5% of the attention map fell inside object boxes — worse than a uniform gaze (13.8%). After training with a hint from the tracker: 21.5%, and 38.0% on the multi-game base.

Going back to where you got to

On one life the agent reached 650–720 of 3266 and died; the rest of the game was territory it had never been in. Go-Explore remembers places reached and returns by restoring the emulator. Sixty iterations of random button-holds took the frontier to 2352, twice what the trained network manages. A full run cleared 1-1 and 1-2 with no deaths.

The data never jumped at enemies

The network died at the first Goomba with the enemy plainly in its input. The cause was labelling: in our own recorded play, jumping with an enemy ahead sits +1.2 points over its own baseline — nothing — against +11.5 in an expert run. The clone reproduced ours faithfully. Mixing expert data in moved jump probability at contact from 0.02 to 0.5 and halved deaths at that enemy.

It stalled because it could not hold a jump

Of 117 real stalls, 90 were at the first pipe of 1-1 — up to 1089 frames motionless while pressing run-and-jump. Jump height on the NES is how long A is held, and the policy re-decides every frame, releasing the button mid-ascent. Holding for 32 frames cut stalls 114 → 33.

A pit is the one lethal thing that is not a sprite

Everything the perception layer finds moves. A hole in the floor does not move and is not a sprite, so it did not exist for the agent, which ran into one at full speed. Now a column whose lower band is the colour that fills the top of the screen counts as a hole — the same test in daylight and underground.

The sprite table instead of guesswork

The console sends its video chip a table of 64 sprites every frame. That gives exact hardware-sprite positions in any NES game without labels — not game objects: an object can be several sprites, and background obstacles (that very pit) are not in the table. For training only: the motion tracker the agent plays with hits a real sprite only 31% of the time and misses 43% of the objects.

Digits without labels

The agent learns digits on its own, from counter dynamics: transitions of the least significant digit form a ring 0→9. The timer is read with 95.8% exact matches and a correlation of 1.000.

The planner finishes the level with the emulator as its model

A planner that rewinds the console clears 1-1 in 20 runs of 32. But its "world model" is the emulator: it looks at the future rather than predicting it. A learned scorer on the same seeds is worth nothing. "The level was completed" is true; "the agent learned to complete it" is not yet.

Learning the planner's choice did not work

A pre-registered experiment with the criterion fixed in advance and a frozen seed block: a student trained to imitate the planner's choice lost to the policy by -432 px (p=0.0005). It learned not the teacher's decisions but the cheapest way to be wrong — wait in 34% of decisions against the teacher's 5%.

A perfect scorer wrong half the time is worth almost nothing

Corrupting an exact scorer at a fixed rate: 20% wrong leaves a third of the advantage, 50% leaves +181 of +4590. The guide for keeping half is an error rate under 15.7%. The best classifier here is wrong 42% of the time.

The planner transfers; not its choice, but its trajectories

A learner was priced in five roles — value estimator, six-way chooser, binary gate, DAgger student, compute allocator — and failed all five; a place-keyed episodic memory lost to the policy as a sixth. Three mechanisms shown separately: reactive representations do not rank across levels, compounding error drifts into the cheapest behaviour, and a wrongly discarded candidate cannot be bought back. Work moves to allocating the planner's own compute budget. Added 8 September: cloning its successful trajectories — twelve clears of level 1 — takes a pixels-and-audio clone from zero to a median of 1952, the first play in this game. The failure is about the planner's choice, not its demonstrations.

Cloning is finished as a source of gains

A copy at 0.981 validation plays no better than one at 0.567. Behavioural cloning is bounded by whoever produced the data, and it has reached that bound.

Adding up small abilities does not work yet

Jumping a pit, handing over when stuck, and both together: −60, −85, −215 distance. They are kept as switches and measured in combination rather than thrown away.