# Thinking While Moving: Deep Reinforcement Learning with Concurrent Control

arXiv: https://arxiv.org/abs/2004.06089 · HTML: https://mlsanity.com/paper/2004.06089/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2020.

## Built on by (most cited first, 30 of 31)

- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Real-Time Execution of Action Chunking Flow Policies](https://mlsanity.com/paper/2506.07339/) (2025)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions](https://mlsanity.com/paper/2309.10150/) (2023)
- [Unsupervised Domain Adaptation of Object Detectors: A Survey](https://mlsanity.com/paper/2105.13502/) (2021)
- [Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models](https://mlsanity.com/paper/2211.11736/) (2022)
- [Acting in Delayed Environments with Non-Stationary Markov Policies](https://mlsanity.com/paper/2101.11992/) (2021)
- [Token Turing Machines](https://mlsanity.com/paper/2211.09119/) (2022)
- [Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization](https://mlsanity.com/paper/2109.14549/) (2021)
- [Revisiting State Augmentation methods for Reinforcement Learning with Stochastic Delays](https://mlsanity.com/paper/2108.07555/) (2021)
- [DVFO: Learning-Based DVFS for Energy-Efficient Edge-Cloud Collaborative Inference](https://mlsanity.com/paper/2306.01811/) (2023)
- [Dynamic Decision Frequency with Continuous Options](https://mlsanity.com/paper/2212.04407/) (2022)
- [DEER: A Delay-Resilient Framework for Reinforcement Learning with Variable Delays](https://mlsanity.com/paper/2406.03102/) (2024)
- [PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2210.08217/) (2022)
- [Multiscale Sensor Fusion and Continuous Control with Neural CDEs](https://mlsanity.com/paper/2203.08715/) (2022)
- [Offline Reinforcement Learning at Multiple Frequencies](https://mlsanity.com/paper/2207.13082/) (2022)
- [DiscreteRTC: Discrete Diffusion Policies are Natural Asynchronous Executors](https://mlsanity.com/paper/2604.25050/) (2026)
- [Learning to Recover: Dynamic Reward Shaping with Wheel-Leg Coordination for Fallen Robots](https://mlsanity.com/paper/2506.05516/) (2025)
- [Finding the Time to Think: Learning Planning Budgets in Real-Time RL](https://mlsanity.com/paper/2606.26463/) (2026)
- [Latency-Tolerant Cloud-Edge Collaborative Vision-Language-Action Models via Emergent Representational Specialization](https://mlsanity.com/paper/2608.00569/) (2026)
- [Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latency](https://mlsanity.com/paper/2608.23831/) (2026)
- [Off-Beat Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2205.13718/) (2022)
- [RT-HCP: Dealing with Inference Delays and Sample Efficiency to Learn Directly on Robotic Platforms](https://mlsanity.com/paper/2509.06714/) (2025)
- [Think at 5 Hz, Act at 20 Hz: Asynchronous Fast-Slow Vision-Language-Action Inference for Closed-Loop Driving](https://mlsanity.com/paper/2607.15621/) (2026)
- [SmoothRL: Online Reinforcement Learning During Asynchronous Execution](https://mlsanity.com/paper/2608.29768/) (2026)
- [World Models Under Asynchronous Sensor Observations](https://mlsanity.com/paper/2609.07299/) (2026)
- [Non-Parametric Stochastic Policy Gradient with Strategic Retreat for Non-Stationary Environment](https://mlsanity.com/paper/2203.14905/) (2022)
- [Real-Time Execution with Autoregressive Policies](https://mlsanity.com/paper/2606.13355/) (2026)

## Abstract (arXiv)

We study reinforcement learning in settings where sampling an action from the policy must be done concurrently with the time evolution of the controlled system, such as when a robot must decide on the next action while still performing the previous action. Much like a person or an animal, the robot must think and move at the same time, deciding on its next action before the previous one has completed. In order to develop an algorithmic framework for such concurrent control problems, we start with a continuous-time formulation of the Bellman equations, and then discretize them in a way that is aware of system delays. We instantiate this new class of approximate dynamic programming methods via a simple architectural extension to existing value-based deep reinforcement learning algorithms. We evaluate our methods on simulated benchmark tasks and a large-scale robotic grasping task where the robot must "think while moving".