# Physical Atari: A Robust and Accessible Platform for Real-time Reinforcement Learning on Robots

arXiv: https://arxiv.org/abs/2606.19357 · HTML: https://mlsanity.com/paper/2606.19357/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 29 May 2026.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation](https://mlsanity.com/paper/1811.02790/) (2018)
- [Learning to Walk via Deep Reinforcement Learning](https://mlsanity.com/paper/1812.11103/) (2018)
- [Bigger, Better, Faster: Human-level Atari with human-level efficiency](https://mlsanity.com/paper/2305.19452/) (2023)
- [When to use parametric models in reinforcement learning?](https://mlsanity.com/paper/1906.05243/) (2019)
- [A Deeper Look at Experience Replay](https://mlsanity.com/paper/1712.01275/) (2017)
- [Real-Time Reinforcement Learning](https://mlsanity.com/paper/1911.04448/) (2019)
- [Setting up a Reinforcement Learning Task with a Real-World Robot](https://mlsanity.com/paper/1803.07067/) (2018)
- [Discounted Reinforcement Learning Is Not an Optimization Problem](https://mlsanity.com/paper/1910.02140/) (2019)
- [Reactive Reinforcement Learning in Asynchronous Environments](https://mlsanity.com/paper/1802.06139/) (2018)

## Built on by (most cited first, 1 of 1)

- [Measure the Sim-to-Real Gap: Designing an Affordable Real-World Benchmark Platform for Reinforcement Learning in AIoT Systems](https://mlsanity.com/paper/2607.10309/) (2026)

## Abstract (arXiv)

We built a robot called the Robotroller that actuates an Atari CX40+ controller and a device called the Atari Devbox that renders the game frame and the reward signal from the Arcade Learning Environment on a screen. The Robotroller and the Atari Devbox, together with an off-the-shelf camera and a desktop computer, constitute a system that can be used to study reinforcement learning algorithms in the physical world. We call the full system Physical Atari. In this paper, we detail the key decisions that make Physical Atari a robust and accessible platform. To make the system robust, we designed the Robotroller so that all movement is done through bearings, which reduces wear. Additionally, we wrote software that monitors the state of the servos at a high frequency and intervenes to limit stress. To make the system accessible, we used affordable off-the-shelf components and parts that can be manufactured using consumer 3D printers. Physical Atari can be built for under 1,000 and has been used for weeks of non-stop reinforcement learning experiments without any mechanical failures. We used it to validate that reinforcement learning algorithms can learn directly on robots and show that even small distribution shifts between learning and deployment can significantly degrade the performance of policies. Our results underscore the importance of on-device adaptation for strong performance on robots.