Procedural districts: making Skyline Run test the policy, not the route

Seed 17 turns one route into a test

Seed 17 stretches 14 platforms and 13 engine-accepted hops across a 136-by-30 grid. Orange hazards wait below selected gaps, while P, C, and G mark the spawn, checkpoints, and goal. The numbered arcs show each hop’s dash budget — how many dashes the level can require while remaining clearable — rather than a route drawn by hand.

This artefact exists to stop one layout from becoming the task. A policy evaluated on a fixed route can score by replaying one successful action sequence. Change the layout every episode and that shortcut loses its value; the policy has to respond to what the observation says now.

That is the design consequence of the measured memorization failure that prompted this work. It also follows the experimental structure of Cobbe and colleagues’ CoinRun work: procedural generation creates separate training and test sets, and their agents overfit surprisingly large training sets. ProcGen broadened that protocol to 16 environments and reported that some agents needed as many as 10,000 training levels to close the generalization gap.

A distribution is not proof of generalization. It is the minimum setup that makes a hold-out set — levels kept out of training so a score on them cannot be inflated by practice — meaningful.

I own and build this site and the AI Maker Lab channel — this is a build log, not an independent review.

Measured physics sets the safe edges

A varied training set is useless if some samples are impossible. The generator therefore starts from measurements of the shipped physics, not from a visually plausible sketch.

The engine spec measures a full-hold jump apex of 56 px — 3.5 tiles — on tick 14. A one-tick hold reaches 16.56 px. Holds from one through six ticks produce the strictly increasing ladder 16.56, 22.42, 27.72, 32.62, 37.02, and 40.97 px. A separate engine test demonstrates a ground dash-jump crossing a seven-tile flat gap without an air dash.

The broader flat-gap envelope pins jump-only reach at five tiles, one-air-dash reach at eight, and two-air-dash reach at eleven. Those are measured maxima on level ground under the test search, not analytical promises for a platform above the launch point. That is why the generator samples below them — at most 4, 6, or 9 gap tiles for budgets 0, 1, or 2 — and still asks the engine to prove every individual hop.

Sampling turns those bounds into districts

generateSpec(seed) is a pure seed-to-spec function. It creates a 30-row grid with 10–16 platforms, which means 9–15 main-line hops. Platform widths are 3–8 tiles. The first platform starts after a three-tile left margin, the level ends with four tiles to the right of the last platform, and surface rows are clamped to 6–26.

Those bounds serve the physics rather than visual variety alone. The platform-count range supplies several decisions per episode while keeping the engine search bounded. Widths of 3–8 tiles leave a real landing interval instead of a single-point target. The vertical clamp leaves headroom above the route and reserves rows below it for hazards. The asymmetric margins keep the spawn and goal away from the map edge.

For each hop, the generator samples a dash budget with probabilities 60% for 0, 30% for 1, and 10% for 2. Budget 0 gets a 2–4 tile gap, budget 1 gets 5–6, and budget 2 gets 7–9. The common case therefore asks for ordinary movement; expensive two-dash transitions remain rare. All three bands sit inside the measured 5/8/11-tile flat limits, leaving margin for discrete timing and elevation changes.

The sampled row delta is -2..+4. Because row numbers increase downward, that means at most a two-row rise but as much as a four-row drop. Rising landings consume vertical ability, so the sampler is deliberately stricter upward than downward. The final surface row is clamped, so a sampled delta can be shortened near the limits.

Each gap independently has a 30% chance of receiving ^ hazard tiles on row 28. The hazard belongs to the gap, never to a main-line platform surface. Version one emits . empty space, # solids, ^ hazards, and the P, C, and G markers; it emits no - one-way tiles. The parser then enforces exactly one spawn P, exactly one goal G, and at least one checkpoint C before a candidate can be evaluated.

The important contrast is simple: broad sampled variety enters on the left, but every candidate still has to pass the concrete checks in the middle before the separate test levels on the right can tell us anything.

The shipped game vetoes impossible hops

I use an acceptance oracle — an automated check that rejects a generated level before a player ever sees it — instead of trusting a sketch. canLandOnSegment does not approximate a parabola or compare two distances. launchState places a real GameState at the source boundary, grounded, moving at RUN_SPEED, with the full three-point SP pool. scriptLands then sends input frames through the shipped step function for at most 180 ticks and accepts only after a jump has occurred and the player becomes grounded at the destination height while overlapping its surface.

The search sweeps jump launch ticks 0–8. Once jump begins, that script holds it; this oracle does not sweep the variable jump-hold durations measured above. With a one-dash budget it also sweeps one dash tick from 0–40. With two dashes it sweeps first-dash ticks 0–19, computes the earliest legal second dash from DASH_TICKS and DASH_COOLDOWN_TICKS — including the special case where jump cancels the first dash — and tries 20 second-dash offsets. A hop ships only when at least one of those real-engine scripts lands.

When a hop fails, one repair pass shrinks its gap by one tile toward a floor of 2 and, if the destination rises, also reduces that rise by one row. Those adjustments happen in the same repair pass, not as two sequential phases. After at most 10 repair passes for a hop, or when neither value can change, the candidate is rejected. The generator then rebuilds the whole plan from hashSeed(seed, attempt), for at most eight total generation attempts. Here the hashSeed namespace is a tag mixed into the seed so two different uses of the same seed never draw the same numbers. Exhausting the attempt bound throws level-gen: unsolvable seed <seed>; there is no silent fallback.

The measured seed 0–19 run is useful because it records the cost rather than only the bound. The accepted candidate for every seed needed zero repair passes. Across rejected candidates there was one repair operation total, and seeds 1, 10, and 18 each regenerated once: one repair operation and three whole-level regenerations across 20 seeds.

Checkpoints keep every accepted route playable

Reachable hops are necessary but not sufficient. Dashes spend SP, and checkpoints refill it. During construction, a checkpoint is inserted before a hop that would push cumulative dash budget above the three-point pool, or when four platform intervals have passed since the previous checkpoint. The spawn counts as the first checkpoint. The candidate builder then audits the route by resetting SP to 3 on checkpoint-bearing source platforms and rejecting any hop that would make it negative.

Other properties follow from the layout. Markers sit one row above the center of a solid platform. Hazards exist only on row 28 in horizontal gaps, while platform surfaces stop at row 26. That keeps spawn, checkpoints, and goal directly supported and gives their 3×3 neighbourhoods no hazards. Main-line platform surfaces are solid and hazard-free by construction.

The spec checks more than those arguments for seeds 0–19. It verifies the 30-row height; 9–15-hop chain; platform count, margins, widths, and row bounds; valid linked segment indexes; a successful canLandOnSegment result for every hop; spawn and goal on the ends of the main line; solid support and hazard-free neighbourhoods for spawn, checkpoints, and goal; no hazards in the two cells above every referenced main-line surface; non-negative SP throughout the route; and the absence of one-way tiles. Determinism is also checked for representative ordinary and tagged seeds. The “checkpoint at least every four platforms” cadence is established by checkpointPlatforms; the spec proves its energy consequence over seeds 0–19 rather than asserting that cadence directly.

Separate seed namespaces make memorization visible

Training seeds are derived as hashSeed(seed, GEN_TRAIN_TAG, i) with GEN_TRAIN_TAG = 0x6e01. Hold-out seeds use hashSeed(seed, GEN_HOLDOUT_TAG, i) with GEN_HOLDOUT_TAG = 0x6e02. The trainer never intentionally asks the training generator for the hold-out derivation, so the two lists are separated at the protocol level.

That is not a mathematical collision-proof guarantee: hashSeed returns a 32-bit value, and the current generator spec exercises representative values from both tagged spaces but does not prove that the two families can never collide. The honest claim is separate derivation namespaces, not impossible leakage under every 32-bit input. An explicit collision check would be required for that stronger guarantee.

The hold-out chart in the trainer should be read as a diagnostic. At each generation it takes that generation’s best training genome, runs it once on each hold-out level, and plots mean spawn-relative progress plus the number of completed hold-out levels. Validation ticks do not enter the training tick count or fitness. A training curve rising while hold-out stays flat is the warning this design was built to expose; both rising is evidence worth investigating, not proof that the policy will generalize beyond the selected hold-out family.

The generator and that diagnostic are now available in the live Skyline Run AI lab. The generator now guarantees that every accepted main-line hop has a real-engine landing script, checkpoint energy never goes negative, and the training and hold-out lists use separate derivations. It does not yet guarantee better generalization: no hardened training run is reported here, and an explicit collision check remains the next step for any stronger namespace claim.

The next experiment asks whether a policy can act on the level in front of it rather than repeat lucky timing: Learning to read the level: replacing lucky jumps with a policy gradient.

Sources

• Cobbe, Karl; Klimov, Oleg; Hesse, Chris; Kim, Taehoon; Schulman, John. “Quantifying Generalization in Reinforcement Learning.” arXiv:1812.02341, 2018. https://arxiv.org/abs/1812.02341. Accessed 2026-08-21.
• Cobbe, Karl; Hesse, Christopher; Hilton, Jacob; Schulman, John. “Procgen Benchmark.” OpenAI, 2019; companion paper arXiv:1912.01588. https://openai.com/index/procgen-benchmark/. Accessed 2026-08-21.

I own and build this site and the AI Maker Lab channel — this is a build log, not an independent review.