300 Million Steps And A Flagpole. Or How I Taught Machines Super Mario

It’s-a me!!#

Almmost anyone who grew up in the 80s can beat World 1-1 of Super Mario Bros in their sleep. This level was the rite of passage that taught a generation where jump buttons belong. So when I set out to teach a reinforcement learning agent to finish it, I expected a long afternoon and a short victory lap.

Instead I got days of training, a laptop that doubled as a space heater, and a genuine lesson about how badly I had misunderstood my own childhood.

The hardware I had wasn’t glamorous, especially in a world of Nvidia DGX sparks and RTX6000s. A laptop RTX 3070m with 8GB VRAM, shared with everything else running on the host. Inside it, an emulator running the actual NES ROM through stable-retro, with PPO from stable-baselines3 doing the learning.

PS: Note that I wasn’t doing this training alone. I was aided by my trusted sidekick Deepseek V4 Flash, (the darling of the world in AI models right now in the August of 2026, history should note). Flash, or Buddy as I started calling him, ran a ton of experiments for me to tune various hyperparameters when I got the chance to take a few winks of sleep.

A machine learning to jump#

For a machine, the first enemy is a wall. I’d have loved if my agent could just stroll towards the goomba and hop over it at the right moment. But over and over, it walked straight into it. After days of training, it still cleared the level about 1 in every 100 tries. 1%. A human toddler with a controller does better.

My agent saw the game as most agents do, as frames: 84x84 grayscale snapshots, 4 of them stacked so it could sense motion. Agents aren’t humans who can process a much richer and bigger screen in the blink of an eye. From this limited info, it had to invent what a jump is, what a pit is, and which button gets you from one side of a gap to the other.

The first lesson I learnt in this is one that of patience. A Mario jump takes several frames to complete, and at 1 action per frame the agent kept producing single-frame hops that ended in the pit. Repeating each decision for 4 frames turned those stumbles into real jumps. Our little guy finally made it past the first goomba, then the pipe, then stopped, helpless, at the 1st gap he could not see. It could not clear the level yet even a single time, but it could at least get past the 1st enemy.

A short life worth living is worth more than living too long#

Entropy is the truth of life. Chaos is our friend. And that’s what makes a model learn as it performs random actions and sees what lets him get closer to the goal. It of course started giving me a sense of progress as I could see a few random runs after several hours worth of training where Mario would clear a run and reach the flag near the castle.

But it remained random for some reason, never finding a meaning in it.

While reviewing the logs bleary eyed one early morning, I realized that I had been making the “episodes” too long. Mario, after all, has 3 lives. Every time he gets dunked by a goomba, or falls into a pit, or runs out of time sticking to a wall, he just starts again, ready to go. stable-retro of course doesn’t have a done signal when a life is lost, it sends that out only when the game is over, all 3 lives gone. This caused a bit of a drift in my calculations which thought mario was stuck for a while, and also because the max_x (how far he had traveled in that run) remained the same even if he died at the 1st goomba in the next life. Making my episodes 1-life runs, fixed this and took me suddenly from completely random to a 0.2% clear run (2 in a 1000 runs cleared the level). It was not much, but it was not completely random anymore either. I’d take it.

2 quiet mistakes#

To up the winning rate, I (and Buddy) spent a lot of time tweaking every hyperparameter we could think of, and trying various techniques. Different learning rates, annealing, distilling knowledge from the master (a composition of the clear runs) to the student, ensemble of specialists (the runs that could clear and the runs that could do the initial parts well on their own), and so many more.

Things clicked. And then they didn’t. I trickled over to 1%, then 1.2%, then 1.5% giving me a false sense of progress, and then crashing back down to 1% again.

Late one night I then decided to go back to the basics and made 2 major revisions.

The reward was the first. I had handed the agent 1 enormous prize at the end of a clear, a terminal bonus worth roughly 37× the entire path reward of the level, plus small credit only for breaking a furthest-distance record. Everything in between, every careful jump, collected coin, and second of survival earned almost nothing. So 1 clear in 100 was a lottery ticket. There was no stairway of small rewards leading up to the flag, no reason for a solid run to feel better than a short one.

The observation was the second. The fastest path had quietly become the near-sighted one. The model was playing on fog, reading coordinates instead of looking at the world. In search of speed I had, at one point, switched to reading the game’s memory directly instead of seeing the grayscale snapshots: the player’s x-coordinate, lives, coins, a tidy list of numbers. It was fast, roughly 6× faster than learning from pixels, and the model raced through the easy stretch. And I had experience with it. Of course I had taught machines Kung Fu the same way because bits and bytes of the RAM are much easier to understand I thought, than those tiny pixels. Then it hit the 1st hole and stalled. A list of numbers carries no sense of the space you need to clear, so the agent kept walking off the edge. Progress collapsed because speed without vision is just a faster way to fall.

I rebuilt both. The new reward was dense: a small step for every bit of forward progress, a small coin bonus, a small score bonus, a gentle tug of time pressure, and a modest flag bonus at the end instead of the casino payout. My experience from kung-fu masters helped a lot as I tweaked and re-tweaked balancing of various components of the rewards. (This is always very interesting, btw. Because this is the same way game developers tweak rewards for games played by humans as well)

And I went back to pixels, letting the network see the pipes and pits and goombas the way we did as kids, the same 4 stacked frames I started with.

That recipe change sounded one swift move, it was anything but. I had to keep tweaking the rewards over and over again but once I got things right, in one fell swoop (or a few million steps) it moved the clear rate from 1% to 54%.

The walls we could name#

54% was honest progress and nowhere near enough at the same time. The next stretch taught me that the agent’s failures were not random. It died at the same few obstacles each time, most often around world position 1785, a spot I learned to name like a person we kept running into on the street. I remember prompting “Hey Buddy, did it cross 1785 this time?”

I also realized that I had forgotten about our own favorite tool that we had talked about earlier. I had done all the best practices but had removed everything yet again as I reset and started from the scratch again. Entropy, the knob that keeps an agent exploring, can also quietly drive the whole thing off a cliff. Let it run hot too long and the policy collapses, right back to dying at the first goomba, undoing a night of work. The fix was to catch the strong model just before that collapse and rehearse it gently: a lower learning rate, modest steady exploration, and never let exploration taper all the way to nothing, since that produced a strange freeze at zero clears.

That careful chain took us 54, then 68, then 84, and finally 97.

The numbers#

Over a hundred 1-life runs, the final model cleared the level 97 times. Over 300 runs, 92.3%. Over 500, 469 clears, or 93.8%. Its best stretch was 58 clears in a row. And when I locked the model to play the same best move every time, it cleared the level 300 times out of 300, running the identical reliable trajectory each time and reaching a furthest position of 3204, 1 hop from the flag. I am quite proud of these results also because I was able to reproduce this in a deterministic evaluation.

The strongest published figure I found for this exact task, from a Stanford course project, was 96.32%,¹ and that run leaned on a top-3 voting trick to steady its sampling. The best plain 1-life result I found out there was 17.4%,² in the same emulator family as the Stanford work, though not byte-for-byte our harness.

You can get the code and the final checkpoints from here to see it in action yourself: https://github.com/shantanugoel/super-mario-bros-rl

Fin#

The best model is genuinely good, and 97 or 100 in 100 fresh runs is not a lucky streak. But I would be misleading you if I implied this method is predictable. Deep learning loves determinism, and starting this recipe from scratch is a lottery. Across different random seeds I saw results from 94% down to a humbling 50. Fresh training would climb beautifully and then collapse no matter how carefully I tuned the learning rate or the exploration, and no dial on earth stopped it. The working answer was to catch the best checkpoint before the collapse and preserve it like a photograph. The top 3 seeds averaged about 90%. So read the headline the way I do: a real and repeatable best, not a reliable average.

As a kid I played this level a hundred times and never asked why the first goomba waits exactly where it does, or why that first gap is just small enough to clear. The answer, taught to me by a machine that needed 300 million steps to cross it, is that someone lined it up so it could be done. I had been crossing that gap by feel since I was 6 without ever noticing what a neat piece of design it was.

My agent will never know it finished something a generation did in an afternoon. But I do.