The memorization trap: when a champion learned one district

In August, I staged one exported NEAT champion—a network evolved by mutating its weights and structure—forward through district after district in the trainer. It repeatedly cleared district-01, so I read that as learning the game. Then it completed 0 of 30 episodes on three generated levels.

Updated September 5, 2026: I rebuilt the AI lab, corrected numerical confounds, and ran the completed trainer under a declared benchmark. This article now separates the original experiment from that new evidence.

I own and build this site and the AI Maker Lab channel — this is a build log, not an independent review.

The original result showed overfitting, not its exact mechanism

The August export cleared district-01 6 times in 10 episodes with 81.5% mean progress. Across the other nine authored districts, it managed one isolated completion in 90 episodes. The export was a version-1 policy—the network that maps observations to actions—with no exposure history: my staging notes say which authored district it last trained on, but the file cannot prove that its generated evaluation worlds were truly unseen. I should therefore call those three worlds nominal hold-outs, not guaranteed unseen tests.

The original figure is historical. It compares the same exported champion over ten episodes per world under a clean pass and under sticky actions, where the environment repeats the previous action with 25% probability.

The historical table still records the failure accurately

I added an evaluateModel pass that replayed the export across all ten authored districts and a selected set of procedural levels. The first pass used five generated levels and 3 episodes per level. Because that small sample was noisy on the one district the model could clear, the stable historical table below uses 10 episodes per level and three generated seeds.

Each episode received hashSeed(seed, EVAL_STREAM_TAG, levelIndex, episode), so the table is reproducible. Progress is relative to the episode’s spawn point: (maxPx - startPx) / (goal.x * TILE - startPx), clamped to [0, 1].

The contrast that matters is district-01’s 6/10 completions against 0/30 across the three nominally held-out generated levels.

The authored-district mean was 27.2%. The generated-level mean was 20.5%. district-01 was the only repeatedly cleared route; the other nine authored districts produced 1 completion in 90 episodes in total, and none besides district-03 reached 31% mean progress.

Sticky actions weakened the mastered route, but not decisively. district-01 fell from 81.5% to 78.7% mean progress while completions stayed 6/10. In the earlier 3-episode pass, it fell from 99.8% and 3/3 completions to 67.5% and 1/3. The perturbation also helped some failures: district-03 rose from 20.5% to 47.4%, district-07 from 22.9% to 39.0%, and the authored-district mean from 27.2% to 30.6%. Deaths on district-01 moved from 2.7 to 1.9, while many failed authored districts averaged between 4 and 7 against an episode cap of 8.

Those results support an overfitting diagnosis and show poor transfer. They do not prove that the network ignored its observations or executed an open-loop action sequence. Sticky-action gains can happen when a bad action is extended by chance; proving the mechanism would require an ablation that removes or scrambles observation channels and compares the same policy.

Prior work made memorization plausible, not proven

Hausknecht and Stone showed in 2015 that NEAT could memorize action sequences in deterministic Atari evaluation and then lose performance when mild stochasticity disturbed them. Machado and colleagues later explained why deterministic evaluation cannot by itself distinguish a closed-loop policy from memorization, and standardized the 25% sticky-action protocol I used in the historical figure.

Cobbe and colleagues changed the unit of evaluation from one level to a distribution. CoinRun separates procedurally generated training levels from held-out test levels, and Procgen extends that idea into a benchmark of generated environments. Those studies motivated my test design. They make the memorization interpretation credible, but they do not replace the missing ablation or repair the version-1 export’s unknown ancestry.

The old trainer rewarded one route and also had numerical confounds

The trainer that produced the export optimized one static, hand-authored level at a time. Each genome received one rollout per generation. There was no averaging across episodes and no hold-out set. The engine step itself is deterministic; the only rollout randomness was the casual skill’s 5% action substitution at decision ticks. Moving to another district rebuilt the trainer around that level and seeded it from the current champion, so previous districts no longer contributed to fitness.

The observation is an egocentric 24-by-18 tile window plus 11 scalars, or 443 floats; it contains neither absolute position nor a wall clock. During the overhaul I found that fresh NEAT genomes guaranteed connections from inputs 0–10, even though tiles occupy 0–431 and the scalar suffix is 432–442. The initializer now connects every scalar to every output and samples tile inputs only from the tile range. That was a real confound, but it does not retroactively prove that this particular champion ignored state.

I corrected three other scoring and rollout boundaries. Evolutionary progress fitness now measures max(0, maxPx - startPx), so a checkpoint’s absolute offset cannot inflate progress. DQN, PPO, Baldwinian NEAT, Lamarckian NEAT, and ERL now share newProgressPx / TILE + 2 * newCheckpoints + 20 * completed - 2 * died as their transition reward, while retaining the separate -0.01 decision charge. PPO now defers a terminal transition that arrives just after a full rollout until the cleared buffer can accept it exactly once. These fixes remove avoidable inconsistencies; I cannot claim that they explain all of the August failure.

One world contract now reaches all seven methods

The rebuilt Skyline Run AI lab keeps GA, DQN, PPO, NEAT, Baldwinian NEAT, Lamarckian NEAT, and ERL. Every method now receives the same explicit training and validation world plan. Evolutionary candidates are scored on the complete fixed batch of selected training worlds and episodes before breeding; DQN and PPO rotate through the selected worlds while preserving their own learning state. Validation episodes never enter fitness or replay.

Selection is now a separate operation. At fixed training boundaries, the worker freezes a candidate and evaluates it on one fixed validation plan. The exported, saved, and watched policy is the best evaluated snapshot, not an incomparable raw-fitness champion or whatever DQN/PPO weights happened to be current. If a run has no validation split, the UI says Training-set evaluation rather than implying a hold-out.

A manual test is reporting-only: it cannot replace the selected policy. Version-2 policy files record the training plan, known training and validation exposure, the selected boundary, and its evaluation report. Imported version-1 files remain usable, but their ancestry is marked unknown rather than invented. Generated test worlds are excluded from all exposure that is actually recorded; when ancestry is unknown, an unrecorded world remains unknown, not guaranteed unseen.

This implementation has been verified locally. I am not claiming that it is deployed.

A fixed budget showed learning but no generated-world generalization

I benchmarked all 7 × 3 × 2 = 42 algorithm, seed, and preset runs. Each got 2,000,000 actual training environment steps with Expert actuation, no demonstrations, and no reward trainer. Expert means no reaction delay and no random input corruption. Equal environment-step budgets do not mean equal compute: the methods perform different amounts of optimization work around those steps.

Policy selection used three spawn-start episodes per world in clean and 0.25 sticky-action passes. The separate reporting-only test used all ten campaign districts plus five generated test worlds, again with three spawn-start episodes per world in both passes. No test row participated in selection.

The table reports median normalized progress from each initialized policy to its selected policy. Single has no hold-out, so its column is training-set evaluation; Generalization uses its validation split. These are within-version selection results, not before-versus-after software comparisons and not test results.

All 14 preset-by-algorithm medians met the declared within-version gain rule: higher completion, or equal completion with at least +0.05 mean normalized progress. The narrower old-versus-new comparison improved only Single NEAT, Single ERL, and Generalization NEAT. The pre-overhaul PPO baseline failed, so I did not substitute a comparison. Old multi-level baselines for GA, DQN, Baldwinian NEAT, Lamarckian NEAT, and ERL were unsupported; the new runs demonstrate that those methods can use the common multi-level contract, not that they beat a fabricated past result.

The decisive result is negative. Across the 42 selected policies, generated test completion was exactly 0/630 clean episodes and 0/630 sticky-0.25 episodes. No method established generalization under this benchmark. Single PPO’s median primary completion reached 100%, but only seeds 1 and 3 cleared; seed 2 did not. The preset name Generalization describes its broader world configuration, not an outcome, and it was not universally better.

The open question moved from learning to transfer

The overhaul answered whether the lab can train and select policies under one reproducible contract: all fourteen group medians improved from their own initial policies. It did not answer how to turn that learning into a generated-world clear. My remaining question is concrete: which change to training diversity or inductive bias can produce the first completion on a reporting-only generated test without letting that test influence selection?

The next build-log chapter, Procedural districts: teaching the generator game grammar, explains how I built the world distribution that now makes that question measurable.

Sources

• Hausknecht, Matthew; Stone, Peter. “The Impact of Determinism on Learning Atari 2600 Games.” AAAI Workshop on Learning for General Competency in Video Games, 2015. https://www.cs.utexas.edu/~pstone/Papers/bib2html-links/AAAI15-hausknecht.pdf. Accessed 2026-08-21.
• Machado, Marlos C.; Bellemare, Marc G.; Talvitie, Erik; Veness, Joel; Hausknecht, Matthew; Bowling, Michael. “Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents.” arXiv:1709.06009, 2018. https://arxiv.org/abs/1709.06009. Accessed 2026-08-21.
• Cobbe, Karl; Klimov, Oleg; Hesse, Chris; Kim, Taehoon; Schulman, John. “Quantifying Generalization in Reinforcement Learning.” arXiv:1812.02341, 2018. https://arxiv.org/abs/1812.02341. Accessed 2026-08-21.
• Cobbe, Karl; Hesse, Christopher; Hilton, Jacob; Schulman, John. “Procgen Benchmark.” OpenAI, 2019. https://openai.com/index/procgen-benchmark/. Accessed 2026-08-21.

I own and build this site and the AI Maker Lab channel — this is a build log, not an independent review.