Learning to read the level: replacing lucky jumps with a policy gradient
The previous NEAT champion cleared district-01, then completed 0 of 30 episodes on unseen generated levels. That means thirty attempts on levels the agent had never seen, and zero finishes. The memorization-trap record contains that measured result; the champion looked like a player on one familiar route and like a sequence of lucky jumps everywhere else.
That is the distinction this build is designed to test. An open-loop controller replays a memorized action sequence, while a closed-loop policy reacts to what the screen currently shows by mapping each new observation back to an action. Hausknecht and Stone’s 2015 recurrent-DQN work explains why observation history can matter when the current frame is incomplete. It supplies a mechanism for observation-dependent control, not proof that the old Skyline Run champion ignored its observation or that recurrence is required here.
The next experiment replaces whole-episode evolutionary selection with per-decision policy gradients, trains across a distribution of districts, and can begin from human runs saved in the playable game. The implementation lives in the Skyline Run AI lab; I use the previous post about learning and evolution as the comparison for putting gradient descent inside NEAT.
No PPO training run or behavior-cloning pass has happened. Every number below is a chosen design value, not a measurement, and I am fixing the comparison before its result exists.
I own and build this site and the AI Maker Lab channel — this is a build log, not an independent review.
Seven papers, one unanswered question
Schulman and colleagues’ Proximal Policy Optimization turns policy improvement into repeated, bounded updates. Its clipped surrogate objective limits how far one update may move the policy, even while PPO makes multiple optimization passes over sampled decisions. That gives this build a practical on-policy policy-gradient method. It does not establish that PPO will learn Skyline Run, generalize to new districts, or outperform NEAT here.
Schulman and colleagues’ Generalized Advantage Estimation supplies the per-decision credit signal. GAE, or generalized advantage estimation, blends short- and long-horizon estimates of how much better an action was. It forms an exponentially weighted estimator that trades bias against variance through gamma and lambda, letting later outcomes assign credit to earlier actions. The estimator and its bias–variance relationship are established; whether 0.99 and 0.95 fit this game’s timing is not.
Cobbe and colleagues’ Procgen benchmark makes the training-level distribution part of the algorithmic question. Their experiments show a substantial generalization gap between training and unseen procedurally generated levels, and that performance changes with the number of training levels. The consequence is direct: level diversity must be measured rather than assumed. Their result does not establish that this build’s district generator, chosen training set, or five holdout seeds are sufficient.
Huang and colleagues’ “37 Implementation Details of PPO” shows that the short algorithm description is not the whole implementation. Their reproduction work identifies vectorized environments—many game copies stepping in parallel to fill a batch—along with GAE boundary handling, full-permutation minibatches, per-minibatch advantage normalization, entropy regularization, and gradient-norm clipping among the details that shape a faithful PPO result. I get an implementation lineage and a checklist from that work, not a guarantee that this browser trainer is correct or that these chosen hyperparameters are optimal.
Hester and colleagues’ Deep Q-learning from Demonstrations shows that demonstrations can bootstrap reinforcement learning. DQfD combines demonstration data with temporal-difference objectives and a supervised large-margin loss, retaining demonstrations in replay while the agent learns. That is one successful demonstration-to-RL route for value learning. It does not show that plain behavior cloning before PPO will preserve competence or that one human run is enough.
Schmitt and colleagues’ Kickstarting work shows that a teacher can reduce the experience needed by a new policy. In their tasks, an auxiliary distillation loss let a student match scratch performance with roughly ten times fewer environment steps while continuing to optimize the task reward. The sample-efficiency result belongs to that teacher–student setting. It does not promise a tenfold gain for Skyline Run, where the teacher is a projected human input stream rather than another policy network.
Ecoffet and colleagues’ Go-Explore is the hard-exploration tool deliberately not chosen. Go-Explore first returns to promising archived states and then explores from them, separating reliable state recovery from exploration in sparse, deceptive games. That establishes a powerful strategy for hard exploration, but not that Skyline Run needs its archive-and-return machinery. This game already supplies dense progress reward plus checkpoint rewards, so this build instead uses ordinary multi-level rollouts and random checkpoint starts.
The policy must react, not replay
The actor is a chosen multilayer perceptron with shape [443, 48, 8]: 443 observation floats, one 48-unit tanh hidden layer, and 8 linear action logits. The critic is a separate [443, 48, 1] network with the same hidden activation and one linear value estimate. The actor’s fixed width keeps its visualization and capacity comparable with the existing DQN and genetic-algorithm policies; that convenience is not evidence that 48 units are enough.
The design separates the action choice, the learning process, and the final test so a better training curve cannot be mistaken for transferable play.
| Layer | What it decides | How it is measured |
|---|---|---|
| Policy | Which of 8 discrete actions to take from the current 443-float observation | Action distribution over the 8 logits at each decision tick |
| Training | How sampled decisions update the actor and critic, optionally after behavior cloning | PPO batches use the fixed reward below; no reward curve exists yet |
| Evaluation | Whether the exported actor transfers beyond its training districts | Clears and progress across every campaign district plus 5 generated holdout seeds |
The hyperparameters turn one browser-friendly constraint into the key contrast: 8 parallel environments fill each 1024-decision batch, while evaluation remains separate and scheduled every 10 iterations.
The chosen rollout uses 8 environments and a horizon of 128 decisions per environment, producing a 1024-decision batch. The chosen discount is gamma = 0.99; GAE uses lambda = 0.95; the PPO clip epsilon is 0.2. Each batch receives 4 update epochs with minibatches of 256, learning rate 2.5e-4, entropy coefficient 0.01, value-loss coefficient 0.5, and maximum global gradient norm 0.5. Each browser advance processes at most 64 update samples, and holdout evaluation is scheduled every 10 iterations.
Environment 0 stays pinned to the on-screen district so its runner remains meaningful to the person watching. The other seven environments choose from the training districts with a deterministic round-robin cursor at construction and after each episode. The distribution, not a lucky primary route, is intended to create pressure for a policy that reads the 443-float observation.
The per-decision reward is deliberately identical to the DQN lab’s reward: -0.01 per decision, + progress_px / TILE for new horizontal progress, +2 per checkpoint, +20 on completion, and -2 per death. Holding that definition fixed makes future reward curves comparable; none exists yet.
A saved human demo contains one four-bit nibble for every 60 Hz engine tick: bit 0 is left, bit 1 right, bit 2 jump, and bit 3 dash. Skyline Run’s fixed-step engine is deterministic, so the same nibble stream replays the same run against the same level and engine version. The recorder saves only completed runs, and the loader replays them to verify that they still complete.
Before reinforcement learning, the optional behavior-cloning phase replays each valid demo, copies an observation at each action-decision tick, and labels it with one of the actor’s 8 discrete actions. Free-form four-button human input does not always equal one of those 8 actions, so the projection chooses the action with minimum Hamming distance—the smallest number of differing button bits—and resolves ties by action index. The actor then receives a chosen 10 behavior-cloning epochs, batch size 64, learning rate 1e-3, and a cap of 50,000 samples. Cross-entropy trains the projected action labels; it does not teach the critic or guarantee that human timing survives the projection.
A PPO model file exports only the actor. Loading that file restores its policy weights, then resumes reinforcement learning with a freshly initialized critic and optimizer state. The export is therefore a playable policy snapshot, not a complete training checkpoint.
Unseen-level clears are the bar
The existing evaluator supplies the protocol: every model plays all campaign districts plus 5 generated holdout seeds, with 3 episodes per level. The full pass runs first with sticky-action probability 0, then again at 0.25, where one quarter of decisions repeat the previous action. The model, seed, configuration, and generated-level seeds remain part of the evidence so the comparison can be repeated.
The result that counts is clears on unseen generated levels. Progress fraction remains a useful diagnostic for where a policy stops, but progress without clears does not establish transferable play. A PPO policy that advances farther yet still clears no unseen level has not escaped the standard set by the memorization diagnosis.
The decisive training run has not happened
No PPO iteration, behavior-cloning epoch, or holdout evaluation has run. This post claims no learning curve, completion, speedup, or advantage over another algorithm.
The 48-hidden-unit actor may lack the capacity to turn a 443-float tile observation into robust actions. The browser’s tick budget may not afford enough samples for an on-policy method, even with eight lockstep environments. Behavior cloning may also inject label noise: a human can press any of 16 button combinations, while the policy exposes only 8 actions, so the minimum-Hamming projection can erase intent precisely when two combinations tie.
Those are experiment outcomes, not implementation excuses. A flat return curve, a policy that overfits the training districts, or pretraining that makes PPO worse must remain reportable results under this protocol. The next step is to run the fixed training and evaluation protocol; the next post must answer one exact question: does the trained policy produce clears on unseen levels?
Sources
- Hausknecht, Matthew; Stone, Peter. “Deep Recurrent Q-Learning for Partially Observable MDPs.” AAAI Fall Symposium Series, 2015. arXiv:1507.06527. https://arxiv.org/abs/1507.06527. Accessed 2026-08-21.
- Schulman, John; Wolski, Filip; Dhariwal, Prafulla; Radford, Alec; Klimov, Oleg. “Proximal Policy Optimization Algorithms.” arXiv:1707.06347, 2017. https://arxiv.org/abs/1707.06347. Accessed 2026-08-21.
- Schulman, John; Moritz, Philipp; Levine, Sergey; Jordan, Michael; Abbeel, Pieter. “High-Dimensional Continuous Control Using Generalized Advantage Estimation.” arXiv:1506.02438, 2015. https://arxiv.org/abs/1506.02438. Accessed 2026-08-21.
- Cobbe, Karl; Hesse, Christopher; Hilton, Jacob; Schulman, John. “Leveraging Procedural Generation to Benchmark Reinforcement Learning.” Proceedings of the 37th International Conference on Machine Learning, PMLR 119:2048–2056, 2020. arXiv:1912.01588. https://arxiv.org/abs/1912.01588. Accessed 2026-08-21.
- Huang, Shengyi; Dossa, Rousslan Fernand Julien; Ye, Chang; Braga, João; Chakraborty, Dipam; Mehta, Kinal; Araújo, João G. M. “The 37 Implementation Details of Proximal Policy Optimization.” ICLR Blog Track, 2022. https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/. Accessed 2026-08-21.
- Hester, Todd; Vecerik, Matej; Pietquin, Olivier; Lanctot, Marc; Schaul, Tom; Piot, Bilal; Horgan, Dan; Quan, John; Sendonaris, Andrew; Osband, Ian; Dulac-Arnold, Gabriel; Agapiou, John; Leibo, Joel Z.; Gruslys, Audrunas. “Deep Q-learning from Demonstrations.” Proceedings of the AAAI Conference on Artificial Intelligence 32(1), 2018. arXiv:1704.03732. https://arxiv.org/abs/1704.03732. Accessed 2026-08-21.
- Schmitt, Simon; Hudson, Jonathan J.; Zidek, Augustin; Osindero, Simon; Doersch, Carl; Czarnecki, Wojciech M.; Leibo, Joel Z.; Kuttler, Heinrich; Zisserman, Andrew; Simonyan, Karen; Eslami, S. M. Ali. “Kickstarting Deep Reinforcement Learning.” arXiv:1803.03835, 2018. https://arxiv.org/abs/1803.03835. Accessed 2026-08-21.
- Ecoffet, Adrien; Huizinga, Joost; Lehman, Joel; Stanley, Kenneth O.; Clune, Jeff. “First Return, Then Explore.” Nature 590:580–586, 2021. https://doi.org/10.1038/s41586-020-03157-9. Accessed 2026-08-21.
I own and build this site and the AI Maker Lab channel — this is a build log, not an independent review.