How NES Player works

The agent learns to play NES games from the same inputs a human gets: a picture and a sound. Its output is gamepad presses. The goal is not to beat a particular game but to make skills carry over to games the agent has never seen.

Principles

No memory reads

The policy has no access to emulator memory. Everything it uses is extracted from pixels and audio. Memory is available to the training loop only, as ground truth for verifying what the agent read off the screen.

Audio is a first-class input

A hit, damage, a pickup, a scene change are often audible before they are visible, and sometimes happen off screen. Measured: adding audio improves action prediction by 7 pp on all four seeds tested.

Transfer beats high scores

We teach concepts rather than the sprites of one game: the controlled object, a threat, a surface, a projectile, progress. The key metric is how much less experience a new game requires.

Pipeline

Emulator (headless)
 ├─ frame 240×224
 ├─ audio 32040 Hz
 └─ press history
        ↓
Perception, no neural nets
 ├─ tracker: camera scroll, moving objects, which one is me
 ├─ object memory: dangerous / rewarding / unknown
 ├─ hearing: spectral events, sound clusters
 └─ screen reading: counter digits, menu prompts
        ↓
Policy (convolutional net, video + audio)
        ↓
World model → planner
        ↓
Gamepad

Perception

Objects from motion

Frames are compared with camera scroll compensated; moving blobs become objects with stable ids and velocities. A separate branch catches small things such as bullets: they pass only if they have their own velocity relative to the background, otherwise flickering stars flood the scene.

Which object am I

The controlled object is found by correlating its world velocity with button presses. The subtlety: when the camera follows the hero, his on-screen velocity is near zero, and without the scroll correction the agent fails to recognise itself.

Meaning of objects and sounds

Collisions and their outcomes accumulate: an object that cost a life becomes labelled dangerous. Sounds are clustered and tied to events — the death jingle, for instance, plays two seconds before the lives counter in memory changes.

Screen reading

Digits are learned without labels. The idea: transitions of any counter's least significant digit form a ring of ten states, 0→1→…→9→0. The ring is found automatically; what remains is locating zero within it.

The anchor comes from shape: the digit one is the thinnest glyph in any font, and zero differs from two by having a closed hole inside. Topology proved more reliable than statistics: in the Castlevania font the two is heavier than the zero, so picking by ink volume was wrong.

Verified against emulator memory: the timer reads with 95.8% exact matches and a correlation of 1.000. Letters are given as an alphabet prior — a human also arrives at a game already able to read; that is enough to understand "PRESS START" and press what the screen asks for.

Training

Where the data comes from

An instinct policy plays on its own, untrained: it calibrates the controls in-game, seeks progress, jumps when stuck, and investigates unfamiliar objects. Headless it runs at a thousand frames per second, so hundreds of episodes take minutes. The network then learns from them.

Behavioural cloning

A stack of four frames plus a 260 ms audio window → convolutions → a decision. Plus an attention hint: the activation map is penalised for looking outside object boxes. Without it the network stares at scenery — measured.

Self-imitation

The agent plays with some randomness, keeps its best runs and re-trains on them. The reward can come from emulator memory or from pixels: accumulated camera scroll correlates with true progress at 0.87.

World model and planner

Early versions predicted the future in a general latent space and ignored actions: the effect of a press — a couple of pixels of hero displacement — drowned in background and scroll. What worked was an ego-centric model: a crop around the controlled object, its velocity and the action → the next displacement.

The planner rolls out behaviour templates sixteen steps ahead and picks the best by prediction. An important lesson: on a weak model planning hurt — the plan lost systematically to plain reaction. It only won once model quality improved.

Results

First encounter with a game

On Castlevania, absent from all training: instincts score 700 points, the random agent and the transferred network score zero. The score was read off the screen and cross-checked against memory.

Transfer

Few-shot with three episodes on unseen games: Gradius 0.707 → 0.787, Battletoads 0.594 → 0.671. On Balloon Fight transfer hurts: its aerial physics is alien to the base.

Trained games

Balloon Fight 0.944, Contra 0.943, Gradius 0.872, Double Dragon 0.864, Super Mario Bros. 0.708, Battletoads 0.702. The multi-game base scores 0.944.

Emulation cores

The core is a setting; frames and audio are normalised, so models need no retraining when it changes.

• fceumm — default; all models were trained on it;
• nestopia, quicknes — work; on some third-party recordings quicknes holds sync twice as long;
• mesen — unsupported, requires frontend services we do not provide;
• FCEUX — unusable: the build crashes at frame 300 of movie playback.

Hardware and platforms

Everything runs on an ordinary laptop: emulation on the CPU, neural networks on whatever accelerator is present. Training a model takes minutes, not hours. The whole model is 16.5 MB, and 89% of that weight sits in the final fully connected layer while vision accounts for just 2%.

• macOS, Apple Silicon — the main development platform, PyTorch through Metal; every number on this site was measured here;
• Linux x86_64 — same install commands, prebuilt emulator wheels exist; CUDA if a card is present, CPU otherwise;
• Windows — through WSL2: stable-retro ships no native Windows build.

Nothing in the code is Mac-specific: device selection (CUDA → Metal → CPU) and emulator core downloads follow the host platform automatically. A GPU is optional — the CPU does the same work, just slower.

The live window holds 60 frames per second with clean audio: game, perception and rendering in one thread, the network in another, sound in a third. Otherwise compute spikes tear the audio apart.