Can a deterministic game teach a network to run the rooftops?

Skyline Run has a deterministic rooftop district, but a machine-perfect clear can prove only that its rules and geometry agree. It cannot show whether a player can read the lesson. The previous build log measured that clear; now I need something that can play without being told how.

So I am building an AI lab for Skyline Run: neural networks that learn the district in the browser, at selectable skill tiers, trained either by a genetic algorithm or by deep Q-learning, watchable at 1× through max speed or hidden entirely to train faster.

Before the first network ran, however, the environment moved. The checkpoint, hardest gap, and stamina regeneration changed, turning a clean training plan into a test of whether its observations and rewards still describe the game.

I chose every number below before the first training run unless I label it as measured. The later section on the district’s re-tune is measurement throughout. That distinction shows where the plan ends and the evidence begins.

I own and build this site and the AI Maker Lab channel — this is a build log, not an independent review.

The engine was already a training environment

The engine already has the property that matters most for machine learning, and it has it by accident. step(state, input, level) is a pure function at a fixed DT = 1/60. It returns a new state, never mutates its argument, and touches no DOM. Every gameplay timer is an integer tick counter.

That is a reinforcement learning environment. It is deterministic, it has no rendering dependency, and it can be advanced as fast as the CPU allows. The renderer is a separate module that draws a state; it is never required to produce one. Nothing in the game has to change to train against it — a fact worth stating plainly, because the temptation to “add a hook for the AI” is how engines rot.

The requirements sit on top of that:

• Agents at several skill tiers, because the point is to simulate players, not to produce one optimal runner.
• Training that runs in the browser, in a Web Worker, so the page stays responsive.
• Two training methods — reinforcement learning and a genetic algorithm — because they fail differently and comparing them is the interesting part.
• A model that can be downloaded as a file and loaded back.
• A visual mode with speed control, and a headless mode that renders nothing.
• Live analytics: loss, fitness, reward, progress.

Research removed three guesses

Three questions were worth researching rather than guessing: how to stabilise Q-learning, how to size a neuroevolution population, and how often an agent should be allowed to make a decision.

Q-learning needs two crutches, and both are non-negotiable. The first is experience replay: a buffer of past transitions that the network re-samples so it does not learn only from the newest step. Sampling random minibatches from that buffer, as the reference implementations put it, “decorrelates the data and leads to better data efficiency.” The second is a target network: a slowly updated copy used for the prediction target so training does not chase itself. It stays frozen between refreshes every C steps. Two further details came from the same reading and cost nothing to adopt: do not take a gradient step until the buffer holds at least a batch, and use Double DQN — pick the action with the live network, then score it with the target copy — which removes a systematic overestimation for one line of code.

Neuroevolution is a legitimate alternative here, not a toy. Gradient-free genetic algorithms have been shown to evolve deep network weights competitively on Atari and humanoid locomotion. The structural point that made it attractive for this build: in neuroevolution “the network is only trained in between episodes, rather than at every frame” — a phylogenetic rather than ontogenetic approach, meaning learning across generations instead of within a single lifetime. For a browser lab that is a gift, because a whole generation can be simulated in lockstep and rendered as a crowd. Population heuristics cluster around 50–200 individuals; elitism should keep the top performers unmutated so a good genome is never lost to a bad coin flip; and mutation probability has a ceiling — one study reports performance improving with mutation rate and then declining past roughly 0.03, while adaptive schemes sit around 5%.

Decision frequency is a free speed-up until it isn’t. Frame skip has the agent observe every k frames and repeat its action across the gap. It “proportionally shortens the effective horizon,” makes high-level manoeuvres reachable by random exploration instead of exponentially unlikely, and cuts the number of forward passes. It also has a known failure mode: too much skipping causes overcommitment, and tasks needing precise aim degrade. k = 4 is the commonly cited trade-off.

The one place I went against the obvious choice was the math library. TensorFlow.js is the default answer for machine learning in a browser, and its WebGL backend is genuinely powerful — tensors as textures, operations as shaders. That advantage is irrelevant at this scale. An after-the-fact execution of the real constructor measured the [119, 48, 8] network at exactly 6,152 parameters: 119×48 + 48 = 5,760, then 48×8 + 8 = 392. Per-operation dispatch overhead would exceed the arithmetic, and a GPU round-trip per forward pass, tens of times per simulated second, is a pessimisation. The lab uses hand-rolled Float32Array matrix math with manual backpropagation and Adam: faster for this size, no dependency, and trivially portable into a worker. The saved model stores raw weights, so this decision is reversible — swapping in a library later changes two files and no file format.

The training contract, number by number

The observation contract gives the network a useful local view without revealing the whole district: 119 floats combine a forward-weighted tile window with movement state.

Observation. 119 floats, rebuilt into a reused buffer each decision. 108 of them are a 12×9 tile window around the player’s centre tile, spanning 2 tiles behind to 9 ahead and 4 above to 4 below. Tiles are encoded as one scalar each: solid 1, one-way 0.5, checkpoint 0.25, goal 0.75, hazard -1, empty 0. Anything below the level’s bottom edge reads -1, because a pit and a spike deserve the same signal. The window is not mirrored by facing — in this district the goal is always to the right, and pretending otherwise would add a symmetry the level does not have.

The remaining 11 are the state the tile window cannot express: horizontal and vertical velocity, grounded flag, coyote counter, dash timer, dash cooldown, dash energy, facing, sub-tile x and y offsets, and normalised distance to the goal. The two sub-tile offsets exist because a 12×20 px player on a 16 px grid can stand in materially different places inside one cell.

Actions. Eight discrete frames rather than four independent buttons: idle, right, right+jump, right+dash, jump, left, left+jump, right+jump+dash. A discrete set is what a Q-network wants, and these eight cover the district’s vocabulary. One consequence is deliberate: because the engine edge-detects jump against the previous frame, an agent must release jump between decisions to jump again. It cannot hold the button forever and expect repeated jumps. That is exactly the constraint a human hand has.

Decision interval: 3 ticks. Slightly tighter than the cited k = 4. The constants are COYOTE_TICKS = 6 and BUFFER_TICKS = 7, but an after-the-fact execution measured 5 ticks of usable slack for both: a coyote jump succeeds 5 ticks after leaving a ledge and fails at 6, while a buffered press succeeds 5 ticks before landing and fails at 6. A 3-tick interval can be at most 2 ticks late relative to any ideal tick, leaving a 3-tick margin inside either measured window. 20 decisions per second is close to human input rate and still cuts forward passes by two thirds.

Skill tiers are handicaps, trained in. This is the decision I expect to be questioned, so here is the reasoning. The lazy way to make a weak AI is to train one strong policy and then corrupt its output. That produces a good player having a seizure, not a bad player. Instead each tier is an environment property present during training: a reaction delay implemented as a FIFO input queue, and a probability of substituting a random action.

A rookie network learns to play while reacting 150 ms late and fumbling one input in ten. It should develop the behaviour that survives those constraints — more caution, worse gap timing — rather than a corrupted expert’s behaviour. Whether it actually does is the first thing worth measuring.

Genetic algorithm. Population 64, within the cited 50–200 band and small enough to render as a crowd. The top 4 genomes survive unmutated. Parents are chosen by 3-way tournament; 75% of children are uniform crossovers, the rest clones; every child weight then mutates with probability 0.08 by adding a gaussian of σ 0.12, clamped to ±4. The mutation rate sits above the 0.03 ceiling one study reports and near the 5% of adaptive schemes — a deliberate bias toward exploration for a first run on an unexplored problem, and the first hyperparameter I expect to lower.

Fitness is furthest progress plus 250 per checkpoint plus 1,500 for a clear, plus half of every tick left unspent under the 3,600-tick cap, minus 100 per death. Progress is measured as best progress, never current position, so dying and respawning cannot be scored as moving backwards.

Deep Q-network. Stability takes precedence over immediate updates here. γ 0.99, learning rate 1e-3, batch 64, replay capacity 20,000, no gradient steps until 1,000 transitions are stored, hard target sync every 500 train steps, ε annealed linearly from 1.0 to 0.05 over 50,000 decisions, Huber loss with δ 1 on the selected action’s output only. Huber loss uses squared error near zero and becomes linear once the error is large, so outliers do not dominate. It already bounds the gradient, so there is no separate clipping step.

Reward per decision: tiles of new progress, plus 2 per checkpoint, plus 20 for a clear, minus 2 per death, minus 0.01 per decision as a time cost. Episodes end on a clear, at 3,600 ticks, at 8 deaths, or when best progress fails to improve by 4 px in 360 ticks. That last one is the stuck detector, and it is the reason a generation of 64 hopeless agents costs seconds instead of minutes.

Architecture. One worker owns the trainer. It pushes metrics events always and frame snapshots only when the visual mode is on, capped at one per 16 ms. Speed control is worker-side pacing: at fixed multipliers it targets speed × 60 ticks per second against a real-time accumulator; at max it simply runs 12 ms slices back to back. Headless mode is not “rendering skipped” — the worker sends no frames at all, and the end-to-end test asserts a frame count of exactly zero, because a performance mode that quietly still serialises state is a lie.

Models save as JSON tagged aml.skyline-run-ai version 1, carrying the algorithm, the skill tier and its parameters, the observation geometry, the action count, the layer sizes, the raw weights, and the training totals. Loading validates all of it and rejects a mismatch with an error rather than a plausible-looking wrong answer — an observation window of a different shape would load fine as numbers and behave as noise.

The district changed before training began

The decisions above were fixed before the first training run. The game has changed since that record was written, while training has yet to begin. Everything in this section is a measurement taken after the fact, not another chosen value. Stamina now regenerates one pip per 48 grounded running ticks against 96 grounded idle ticks. A checkpoint now stands at column 116. The chained-dash gap narrowed from 10 tiles to 9 tiles. The measured optimal deterministic clear fell from 2218 to 1304 ticks.

The observation-horizon result matters most. The tile window reaches 9 tiles ahead. From the launch tile, where the player’s centre tile is column 121, it covers through column 130. The landing platform’s first solid column is 131, outside the window. The forward-most column, 130, encodes 0 in all 9 cells, exactly like open air.

The platform first enters the window when the centre tile reaches column 122: one tile into the gap, with the player already airborne. Before the easing, the platform began at column 132 and first appeared at centre column 123, two tiles into the gap. The district’s hardest move is committed blind. The easing narrowed that blind margin from 2 tiles to 1. Widening the forward reach is not a free correction: it is baked into the saved model’s observation metadata, so changing it invalidates every model already trained against this contract.

The checkpoint also changes the stuck detector’s meaning. Returning from the old column-80 checkpoint to the launch tile takes 276 ticks, or 76.7% of the 360-tick stuck window; with respawn it takes 294 ticks, or 81.7%. From the new column-116 checkpoint, the return takes 36 ticks, or 10.0%; with respawn it is 54 ticks, or 15.0%. maxPx is best-ever progress, so it does not advance anywhere on a return trip. The stuck timer runs the whole time while the agent is doing exactly the right thing. A detector meant to kill hopeless agents was, on the district’s hardest move, within striking distance of killing agents that were learning it.

The perfect-clear GA fitness increased by 707, or 11.2%: 250 comes from the added checkpoint and 457 from the faster clear. The added checkpoint is worth +2 per DQN clear, while fewer decisions produce the smaller time cost in the table. Fitness numbers from before and after the re-tune are not comparable. Any pre-easing baseline is void.

The regeneration change introduces hidden state. The 11 scalars carry dash energy as sp / SP_MAX, but not the regeneration accumulator. Under the old flat 72-tick rule, that omission hid a fixed clock. Under the dual-rate rule, the accumulator advances at a rate set by the agent’s own recent input. Execution produced states with accumulator values 0 and 94 but bit-identical 119-float observations. Because the encoder never reads the field, any two accumulator values collide; at the extreme, 0 and 95 sit 96 and 1 idle ticks from the next stamina pip — 95 engine ticks apart. That is partial observability introduced by a game-design change, invisible to a feedforward network with no memory and no frame stacking. The honest choices are to add the accumulator as a twelfth scalar and break the saved-model observation contract, or leave it out and accept that the network must infer stamina timing from behaviour it cannot see.

What the design proves—and what training must test

Nothing has been trained yet. The original decision values remain a starting position taken from the literature and from the engine’s measured constants, and the previous build log exists mainly because three carefully reasoned values turned out to be wrong on contact with the running system. The expected corrections remain the mutation rate, the 3-tick decision interval, and the reward’s time cost. The observation window’s forward reach now joins that list with measured evidence already against it.

The specific claim I want to test first is the skill-tier one: that training under a handicap produces recognisably different play rather than a uniformly worse version of the same play. If a rookie network’s runs look like an expert’s with hiccups, the design is wrong and the tiers need to differ structurally, not just in noise.

Two known limitations are already accepted rather than solved. Episodes that end by timeout or stuck detection are stored as terminal transitions, which is not strictly correct — they are truncations, and a value estimate bootstrapped from them is slightly pessimistic. And there is one level. An agent that clears one district has memorised a district; generalisation is not on the table until there is a second one to fail at.

The lab is also not a player study. It measures whether the rules are learnable, which is a narrower and more mechanical question than whether they are legible to a person. The measurements stay in place for the same reason as last time: to say what changed when the next assumption breaks.

So the decision is not that these starting values are right. It is that the deterministic engine, explicit training contracts, and measured caveats make them testable. Next, Teaching the trainer: evolving the reward instead of tuning it asks whether evolution should tune the reward itself instead of leaving me to turn its coefficients by hand.

Sources

• Mnih, Volodymyr et al.; TensorFlow.js examples. snake-dqn: reference browser DQN implementation. https://github.com/tensorflow/tfjs-examples/tree/master/snake-dqn. Accessed 2026-08-21.
• Britz, Denny. “DQN — experience replay and target networks.” reinforcement-learning. https://github.com/dennybritz/reinforcement-learning/blob/master/DQN/README.md. Accessed 2026-08-21.
• van Hasselt, Hado; as summarised in “Techniques to Improve the Performance of a DQN Agent.” Towards Data Science, February 5, 2025. https://towardsdatascience.com/techniques-to-improve-the-performance-of-a-dqn-agent-29da8a7a0a7e/. Accessed 2026-08-21.
• Such, Felipe Petroski et al. “Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning.” arXiv:1712.06567, April 20, 2018. https://arxiv.org/abs/1712.06567. Accessed 2026-08-21.
• Cuccu, Giuseppe; Togelius, Julian; Cudré-Mauroux, Philippe. “Playing Atari with Six Neurons.” arXiv:1806.01363. https://arxiv.org/pdf/1806.01363. Accessed 2026-08-21.
• Lample, Guillaume; Chaplot, Devendra Singh. “Playing FPS Games with Deep Reinforcement Learning.” arXiv:1609.05521. https://arxiv.org/pdf/1609.05521. Accessed 2026-08-21.
• “An Analysis of Frame-skipping in Reinforcement Learning.” arXiv:2102.03718. https://arxiv.org/pdf/2102.03718. Accessed 2026-08-21.
• “Elitist Genetic Algorithm.” Algorithm Afternoon, April 17, 2024. https://algorithmafternoon.com/genetic/elitist_genetic_algorithm/. Accessed 2026-08-21.
• Smilkov, Daniel et al.; as described in “DISCO: A Browser-Based Privacy-Preserving Framework for Distributed Collaborative Learning.” arXiv:2511.19750. https://arxiv.org/pdf/2511.19750. Accessed 2026-08-21.

I own and build this site and the AI Maker Lab channel — this is a build log, not an independent review.