Should a genome keep what its network learned? Three ways to mix gradients into evolution

A genome learns to run, earns its score, and then faces a fork: forget the new weights, inherit them, or pool the run for one shared learner. Each choice changes what evolution is actually selecting. Forgetting rewards a starting point that learns well; inheritance can move species boundaries; pooling lets one learner reuse experience from all 150 genomes.

The three trainers are specified before their first run. Every budget below is a design choice, and nothing in this post is a measured result.

I own and build this site and the AI Maker Lab channel — this is a build log, not an independent review.

Three ways to spend one gradient

The previous build log moved reward tuning inside the experiment. This one puts lifetime learning inside the fitness evaluation. The first AI build log established the evolution-only comparison for Skyline Run; this comparison keeps the game fixed and changes what evolution may reward.

The enabling property is already in the code. Every NEAT genome in this project is acyclic by construction, hidden nodes use tanh, and output nodes are linear. Its phenotype—the behaving network a genome unfolds into—is therefore a differentiable directed acyclic graph that can serve directly as a Q-network without changing the representation. Putting gradient learning inside evaluation is a design choice, not a feasibility question.

The literature refuses to pick a winner

Learning can guide evolution without inheritance. Hinton and Nowlan’s “How Learning Can Guide Evolution” shows how learning reshapes the search space and allows learning organisms to evolve faster even when acquired characteristics are not communicated to the genotype. That supports the Baldwinian variant: learn during life, score the learned performance, then discard the weights. Whether the same effect appears in this game remains an open question.

Baldwinian and Lamarckian search are not interchangeable, and neither wins by default. Lamarckian write-back means copying the learned weights back into the genome. Whitley, Gordon, and Mathias’s “Lamarckian Evolution, the Baldwin Effect and Function Optimization” presents functions where a plain genetic algorithm and Lamarckian evolution converge to the same local optimum while Baldwinian search reaches the global optimum, and cases where Baldwinian search outperforms Lamarckian evolution using the same local search. The result cannot choose a winner here. Instead, it is why all three variants remain separate selectable algorithms.

The Baldwin-effect literature is also a cautionary tale. Turney’s “Myths and Legends of the Baldwin Effect” is the reason I claim no benefit in advance. A named effect is not evidence that one of these planned trainers will improve the game.

Evolving topology while training weights by gradient descent has a precedent. Miikkulainen et al.’s “Evolving Deep Neural Networks” describes CoDeepNEAT, where fitness depends on how well an evolved architecture can be trained by gradient descent and each chromosome is converted to a network and trained for a fixed number of epochs. That precedent supports a fixed inner training budget for each genome. It cannot tell me what the budget should be here.

ERL is the shared-replay design. Evolutionary reinforcement learning, or ERL, pairs a population with a gradient learner and lets them share experience. Khadka and Tumer’s “Evolution-Guided Policy Gradient in Reinforcement Learning” places one reinforcement-learning agent alongside an evolutionary population, trains them in parallel, and shares a replay buffer. ERL recycles experience from the evolutionary population so one learner can extract more information from it and act as a population-driven guide. Here, off-policy Q-learning makes a single buffer fed by 150 different genomes a valid chosen design. How that design performs in Skyline Run is still unmeasured.

CERL broadens shared replay to a portfolio of learners. Khadka et al.’s “Collaborative Evolutionary Reinforcement Learning” succeeds ERL with a portfolio of learners rather than one. I chose the single-learner ERL design to bound its browser cost, so CERL documents a direction not taken. The better design for this game remains unknown.

NEAT supplies the representation this build needs. Stanley and Miikkulainen’s “Evolving Neural Networks through Augmenting Topologies” establishes speciation and innovation-number-aligned crossover across variable topologies. Those mechanics let evolution cross over a gradient-trained genome. They prove the representation can support the loop, not that learning will help in Skyline Run.

Mutating a gradient-trained network is the known failure mode. Bodnar, Day, and Lió’s “Proximal Distilled Evolutionary Reinforcement Learning” traces ERL scalability problems to simple genetic encoding and destructive traditional variation operators that cause catastrophic forgetting of acquired traits. NEAT crossover aligns matching genes by innovation number instead of blending unaligned vectors. That may make an injected genome less fragile than the direct encodings PDERL studied, but it is a local hypothesis, not a finding.

One game, three ways to carry learning forward

The AI lab selector names the planned variants NEAT + DQN (Baldwin), NEAT + backprop (Lamarck), and NEAT + shared DQN (ERL). Every number below is a chosen value unless it says otherwise. None is a measurement.

All three reuse the chosen shaped fitness already used by plain NEAT: furthest progress, 250 per checkpoint, 1500 for a clear, 0.5 per unspent tick under the 3600-tick cap, and 25 per death. Keeping that definition fixed makes their future fitness curves comparable with one another and with plain NEAT. All three are seeded and deterministic by design, so the same seed will reproduce the same generation statistics.

Gradients use a reverse-topological-order backward pass over the genome’s own graph, with one gradient slot for every enabled connection and one for every node bias. Adam applies the updates in place.

Baldwinian: learn, score, forget

The chosen population is 32, with 4 target species. Each genome is fine-tuned in isolation for 2 ε-greedy episodes at a constant ε of 0.2, followed by 1 greedy evaluation episode. The shaped fitness from that evaluation becomes the genome’s score.

The inner Double-DQN uses chosen γ 0.99, learning rate 1e-3, batch size 32, and a replay ring of 4096 transitions that resets for each genome. Learning waits for a warm-up of 128 transitions, refreshes the target network every 128 gradient updates, and uses Huber loss—squared error near zero and linear error for large errors—with δ 1.

The trained weights are then discarded. The genome that reproduces is the genome that was born, so selection rewards a good starting point for learning. The honest consequence is that the exported champion plays with its pre-learning weights.

Lamarckian: learn, score, keep

The population, species target, episodes, exploration, evaluation, Double-DQN settings, replay ring, warm-up, target refresh, and Huber loss use the same chosen values as the Baldwinian loop. One line differs: trained weights and biases are written back into the connection and node genes, clamped to ±4.

Those weights are inherited, so faster generation-to-generation weight progress is the expectation being tested, not a measured result or a claim that this variant is better. NEAT compatibility distance—how different two genomes are, used to group them into species—gives the mean weight difference of matching genes a chosen coefficient of 0.4; moving every weight by gradient descent therefore also moves species boundaries. Speciation is expected to differ from the Baldwinian run on the same seed. Whitley’s result is the warning that Lamarckian inheritance may still lose.

ERL: one buffer, one learner, one injection

The chosen design keeps a full population of 150 running in lockstep exactly as plain NEAT does and adds 1 learner. Every decision by every population member and by the learner enters one shared replay ring of 20 000 transitions with the same per-decision reward already used by the DQN lab.

After each lockstep round, once the chosen warm-up of 1000 transitions exists, the learner takes 1 gradient step with batch size 64. Its chosen γ is 0.99, learning rate is 1e-3, and target network refresh interval is 500 updates. ε anneals from 1 to 0.05 over 50 000 decisions.

The learner trains a clone of the current champion’s topology, preserving a valid NEAT genome and the same innovation numbers. At each generation boundary, its trained weights are written into a genome that replaces one member of the next population. When a new champion appears, the learner re-anchors to that topology and resets its optimizer state, while the replay buffer and exploration schedule survive.

A win must survive unseen seeds

The protocol is fixed before any run so the comparison cannot be retrofitted:

• All four NEAT-family algorithms use the same level, skill tier, seed set, and reward coefficients. The shared instrument is the existing NEAT charts for fitness, progress, species, complexity, and completions, plus one inner-loss series.
• The measurement hazard is explicit: plain NEAT and ERL count one tick as one population-wide lockstep round, while the Baldwinian and Lamarckian trainers run genomes one at a time and count engine ticks. Per-generation curves are directly comparable. Per-tick and wall-clock comparisons are not, and a future sample-efficiency claim must be normalized before it means anything.
• Evidence would require a variant to beat plain NEAT on the same seeds, repeat that result on previously unseen seeds, and produce clears rather than progress-only wins. The exported model, seed, and configuration must remain available so the run can be repeated.

The comparison run still has not happened

The run still to come compares plain NEAT, Baldwinian, Lamarckian, and ERL on the same seeds and then on unseen seeds, with clears—not progress alone—as the standard. It has not been run, and no variant is claimed to be better. The three trainers are specified, not yet built.

The open risks are whether a small per-genome fine-tuning budget can separate learning potential from noise; whether Lamarckian write-back destabilizes speciation enough to hurt; whether ERL injection survives NEAT mutation or is forgotten in the sense PDERL describes; and whether any of these loops is affordable inside a browser tab.

What is decided is what each trainer changes and what evidence could separate them; no performance benefit has been proved. While that comparison remains open, the next build log turns from learning weights to learning mutation itself: A learned gene editor: can a model mutate NEAT better than chance?

Sources

• Hinton, Geoffrey E.; Nowlan, Steven J. “How Learning Can Guide Evolution.” Complex Systems 1:495–502, 1987. https://www.cs.toronto.edu/~hinton/absps/evolution.htm. Accessed 2026-08-21.
• Whitley, Darrell; Gordon, V. Scott; Mathias, Keith. “Lamarckian Evolution, the Baldwin Effect and Function Optimization.” Parallel Problem Solving from Nature — PPSN III, Lecture Notes in Computer Science vol. 866, pp. 6–15, Springer, 1994. https://doi.org/10.1007/3-540-58484-6_245. Accessed 2026-08-21.
• Turney, Peter D. “Myths and Legends of the Baldwin Effect.” arXiv:cs/0212036. https://arxiv.org/abs/cs/0212036. Accessed 2026-08-21.
• Miikkulainen, Risto; Liang, Jason; Meyerson, Elliot; Rawal, Aditya; Fink, Dan; Francon, Olivier; Raju, Bala; Shahrzad, Hormoz; Navruzyan, Arshak; Duffy, Nigel; Hodjat, Babak. “Evolving Deep Neural Networks.” arXiv:1703.00548, March 4, 2017. https://arxiv.org/abs/1703.00548. Accessed 2026-08-21.
• Khadka, Shauharda; Tumer, Kagan. “Evolution-Guided Policy Gradient in Reinforcement Learning.” Advances in Neural Information Processing Systems 31 (NeurIPS 2018), pp. 1196–1208. arXiv:1805.07917. https://arxiv.org/abs/1805.07917. Accessed 2026-08-21.
• Khadka, Shauharda; Majumdar, Somdeb; Nassar, Tarek; Dwiel, Zach; Tumer, Evren; Miret, Santiago; Liu, Yinyin; Tumer, Kagan. “Collaborative Evolutionary Reinforcement Learning.” Proceedings of the 36th International Conference on Machine Learning, PMLR 97, 2019. http://proceedings.mlr.press/v97/khadka19a/khadka19a.pdf. Accessed 2026-08-21.
• Bodnar, Cristian; Day, Ben; Lió, Pietro. “Proximal Distilled Evolutionary Reinforcement Learning.” Proceedings of the AAAI Conference on Artificial Intelligence 34(04), 2020. arXiv:1906.09807. https://arxiv.org/abs/1906.09807. Accessed 2026-08-21.
• Stanley, Kenneth O.; Miikkulainen, Risto. “Evolving Neural Networks through Augmenting Topologies.” Evolutionary Computation 10(2):99–127, 2002. https://doi.org/10.1162/106365602320169811. Accessed 2026-08-21.

I own and build this site and the AI Maker Lab channel — this is a build log, not an independent review.