Teaching the trainer: evolving the reward instead of tuning it

A runner reached 1780 px and still lost to a safer champion at 1828 fitness. The -100 death penalty had punished the farther-reaching behaviour so hard that it was discarded twice. I cut the penalty from 100 to 25 by hand—and exposed a trainer that depended on me reopening the fitness function whenever learning went wrong.

The previous build log fixed Skyline Run’s training rewards before the first training run. The running system showed why that fix could not be the last word. The next build puts tuning inside the experiment: the AI Trainer AI evolves small networks that set rewards for the inner Skyline Run AI lab.

This is not a claim that reward design can be handed to a machine and forgotten. It is a controlled comparison between learned reward schedules and the hand-tuned schedule they are meant to replace.

I own and build this site and the AI Maker Lab channel — this is a build log, not an independent review.

When the hand-tuned fix became the problem

The reward coefficients had looked like ordinary constants: 250 for each checkpoint, 1500 for a completion, 0.5 for every unspent tick, and a penalty for every death. Together they decided which behaviour survived reproduction. Changing one did not merely adjust a chart; it changed the trainer’s definition of a good run.

The hand edit solved one symptom and exposed the larger problem. Every correction so far had been a person changing a constant after watching training fail, and every edit voided the previous baselines. If the reward needs repeated tuning, then reward design belongs inside the learning system, not in configuration outside it.

Research points to rewards that learn

The reward an agent learns from does not have to be the thing its designer ultimately values. Singh, Lewis, and Barto’s “Where Do Rewards Come From?” separates the agent’s primary reward from fitness over the designer’s environment distribution. In that framing, the learning reward is itself a design variable, chosen because the policy it produces maximizes a separate designer objective. The two functions need not even have the same form.

AutoRL turns that framing into an outer evolutionary search. Faust, Francis, and Mehta’s “Evolving Rewards to Automate Reinforcement Learning” treats reward tuning as hyperparameter optimization: a population of inner RL agents trains under candidate rewards, while the outer layer selects the reward that maximizes the task objective. That separation is the direct precedent for scoring our controllers on progress and completion rather than on the shaped fitness they created.

Population Based Training says the useful answer may be a schedule, not a constant. Jaderberg et al.’s “Population Based Training of Neural Networks” jointly adapts models and hyperparameters during training. Its exploit step copies a better population member; its explore step perturbs or resamples hyperparameters. The result is a changing hyperparameter schedule over the course of learning instead of one fixed setting chosen beforehand. PBT is not specifically a reward-search method, but it supports letting reward coefficients change as the inner trainer changes.

Shaping rewards and learning parameters can be evolved together. Elfwing, Uchibe, Doya, and Christensen’s “Co-evolution of Shaping Rewards and Meta-Parameters in Reinforcement Learning” evolves potential-based shaping rewards alongside the learning rate α, discount factor γ, and softmax temperature τ, with experiments in mountain car and transfer from simulation to robotic foraging hardware. The verified abstract establishes that joint optimization. It does not establish whether outer selection used a separately defined ground-truth metric, so I do not use it as evidence for that part of our design.

The search can operate over reward programs, not only coefficients. Niekum, Barto, and Spector’s “Genetic Programming for Reward Function Search” applies genetic programming to alternate reward functions to improve learning performance. The verified DOI record establishes the reward-function search, but does not expose enough method text to determine whether its genetic-programming fitness was a separate task objective or derived from shaped return. It is precedent for the search space, not evidence for our outer scoring rule.

Eureka is the LLM-era version of reward search. Ma et al.’s “Eureka: Human-Level Reward Design via Coding Large Language Models” uses a coding LLM to generate reward code and evolutionary improvement to refine it. Its formal problem searches for a reward R whose learned policy maximizes a separate fitness function F(π), using the ground-truth or sparse task reward for evaluation. The generator changed; the bilevel separation remained.

Reward shaping has a safety condition. Ng, Harada, and Russell’s “Policy Invariance Under Reward Transformations” shows that, beyond positive linear transformations, shaping of the potential-difference form F(s, s') = γΦ(s') - Φ(s) preserves optimal policies in the relevant general setting. An unconstrained reward search can instead change which policy is optimal, not merely make the original task easier to learn. Our [0.25×, 4×] multiplier bounds are a pragmatic guard against extreme coefficients. They are not a proof of policy invariance.

One controller changes four rewards

The design uses bilevel evolution: an outer loop evolves the reward while an inner loop trains against it. The outer population is NEAT, but each genome is not a Skyline Run player. It is a 12-input, 4-output controller that watches one inner NEAT training run and emits multipliers for checkpoint bonus, completion bonus, time bonus, and death penalty. The initial topology is dense and flat: every input connects to every output.

The twelve observations are deliberately small and bounded. At inner generation 0, inputs 1–7 are forced to zero because no generation statistics exist yet. Inputs 8–11 also begin at zero because every current multiplier begins at 1.

Each output becomes 2 ** (2 * tanh(output)). That maps any finite network output into a multiplier between 0.25× and 4×. A zero output gives exactly 1×, so four zero outputs reproduce the hand-tuned coefficients with no special case.

The controller emits rewards before inner generation 0, then emits them again every 5 completed inner generations. NeatTrainer.setRewards applies the new values at the next generation boundary, because shaped fitness is computed only when that generation finishes. It never rescored an in-flight population under rules that changed halfway through its run.

The controller is judged by the agents it produces

Outer selection never sees shaped fitness as its score. A candidate’s meta-fitness—how good a reward function is, scored by the agents it produces—is:

tail-weighted mean over evaluated inner generations of runningMax(bestMaxPx),
  where generation g carries weight g
+ 2000 if any inner generation produced a completion
+ 500 * (final-generation completions / innerPopulationSize)

The first term rewards progress that persists to the end of the run: later generations carry linearly more weight, and every candidate is evaluated 25% past its nominal horizon so the controller trains in the saturated beyond-horizon regime it later deploys into. (The smoke test below predates this revision and was scored with an unweighted mean over the nominal horizon.) The other terms make completion the ground-truth event. The unchanged hand-tuned baseline runs first as candidate 0 in every meta-generation, so its score is known before any controller is evaluated. It and every controller in that meta-generation receive the same hashed inner seed. This use of common random numbers reuses the same seeds across candidates so comparisons are not luck. The first run supplies baselineFitness as a contemporaneous control rather than relying on a baseline measured under an older level or seed.

The seed changed which champion deserved to survive

The inner seed still rotates between meta-generations as innerSeed = hashSeed(seed, metaGen, 0x1177), so raw fitness from different meta-generations is not comparable. The all-time champion is therefore the controller with the largest best - baselineFitness advantage, and species stagnation uses the same scale. This run exposed why: raw-fitness selection would have retained meta-generation 1 at raw 1274.04 and advantage only +25.90, instead of meta-generation 2 at raw 1043.85 and advantage +91.60, purely because meta-generation 1 drew an easier inner seed.

The smoke test found a lead, not a verdict

I ran a deliberately small smoke test on createDistrict01() at skill expert, seed 7, with meta-population 6, 10 inner generations per candidate, controller cadence 5, and inner population 150. Its candidateCount was 7: one hand-tuned baseline followed by six controllers. These are smoke-test measurements, not evidence that the learned trainer generalizes.

The run took 16.92 s for 21 inner trainings and 231781 total ticks. The all-time champion came from meta-generation 2, with bestMetaFitness 1043.85, championBaselineFitness 952.26, and the largest same-generation advantage. Evaluated at inner generation 0 with zeroed telemetry, it emitted [1.05, 0.68, 0.94, 0.81], producing checkpoint bonus 261.64, completion bonus 1016.50, time bonus per tick 0.47, and death penalty 20.34.

The trace also exposed a safe starting condition in the topology. At inner generation 0, the telemetry vector is all zeros, so the emitted multipliers depend only on output biases. Every bias in the seed population is zero. Every controller in meta-generation 0 therefore starts from exactly [1, 1, 1, 1] and can diverge only at its first cadence-5 call. Non-unity generation-0 multipliers appear only after bias mutation; the hand-tuned coefficients are the starting point by construction, not by luck.

The best controller beat its own same-generation hand-tuned control in all three meta-generations. The population mean fell below the baseline only in meta-generation 1. No inner generation produced a completion inside the 10-generation budget. Every champion remained topologically dense and flat, with 0 hidden nodes and 48 connections. A second identical run produced byte-identical MetaGenStats sequences; only wall-clock fields differed.

Three meta-generations are far too short to support a conclusion. The new champion left the checkpoint bonus essentially alone at 1.05×, cut the completion bonus hardest to 0.68×, and softened the death penalty to 0.81×. That direction is a hypothesis about what helped on this seed, not a result about good reward design.

This smoke test decided the evaluation protocol, not whether the learned reward works. It showed deterministic repeatability and found a controller that beat its matched hand-tuned control in three short meta-generations, but no inner generation completed the level. The named next step is a full run on the AI Trainer AI page across repeated, previously unseen inner seeds.

The design will count as evidence only if it produces completions, keeps beating the matched control, and retains that advantage in later meta-generations. Losing on unseen seeds, winning only on progress, or fading during a longer search would falsify the result. The exported controller, configuration, seed set, per-generation baseline, and completion record would all have to remain available so the comparison could be repeated.

The next build log, Should a genome keep what its network learned? Three ways to mix gradients into evolution, asks what should happen to learning after each genome is scored.

Sources

• Singh, Satinder; Lewis, Richard L.; Barto, Andrew G. “Where Do Rewards Come From?” Proceedings of the 31st Annual Conference of the Cognitive Science Society, 2009. https://all.cs.umass.edu/pubs/2009/singh_l_b_09.pdf. Accessed 2026-08-21.
• Faust, Aleksandra; Francis, Anthony; Mehta, Dar. “Evolving Rewards to Automate Reinforcement Learning.” arXiv:1905.07628, May 18, 2019. https://arxiv.org/abs/1905.07628. Accessed 2026-08-21.
• Jaderberg, Max; Dalibard, Valentin; Osindero, Simon; Czarnecki, Wojciech M.; Donahue, Jeff; Razavi, Ali; Vinyals, Oriol; Green, Tim; Dunning, Iain; Simonyan, Karen; Fernando, Chrisantha; Kavukcuoglu, Koray. “Population Based Training of Neural Networks.” arXiv:1711.09846, November 27, 2017. https://arxiv.org/abs/1711.09846. Accessed 2026-08-21.
• Elfwing, Stefan; Uchibe, Eiji; Doya, Kenji; Christensen, Henrik I. “Co-evolution of Shaping Rewards and Meta-Parameters in Reinforcement Learning.” Adaptive Behavior, December 2008. DOI:10.1177/1059712308092835. https://doi.org/10.1177/1059712308092835. Accessed 2026-08-21.
• Niekum, Scott; Barto, Andrew G.; Spector, Lee. “Genetic Programming for Reward Function Search.” IEEE Transactions on Autonomous Mental Development 2(2):83–90, June 2010. DOI:10.1109/TAMD.2010.2051436. https://doi.org/10.1109/TAMD.2010.2051436. Accessed 2026-08-21.
• Ma, Yecheng Jason; Liang, William; Wang, Guanzhi; Huang, De-An; Bastani, Osbert; Jayaraman, Dinesh; Zhu, Yuke; Fan, Linxi; Anandkumar, Anima. “Eureka: Human-Level Reward Design via Coding Large Language Models.” arXiv:2310.12931, October 19, 2023. https://arxiv.org/abs/2310.12931. Accessed 2026-08-21.
• Ng, Andrew Y.; Harada, Daishi; Russell, Stuart. “Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping.” Proceedings of the Sixteenth International Conference on Machine Learning, 1999. http://robotics.stanford.edu/~ang/papers/shaping-icml99.ps. Accessed 2026-08-21.

I own and build this site and the AI Maker Lab channel — this is a build log, not an independent review.