# R^3: On-device Real-Time Deep Reinforcement Learning for Autonomous Robotics

arXiv: https://arxiv.org/abs/2308.15039 · HTML: https://mlsanity.com/paper/2308.15039/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 82 references that are in the map.
- First posted on arXiv 29 Aug 2023.

## Builds on (most cited first, 30 of 36)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Revisiting Fundamentals of Experience Replay](https://mlsanity.com/paper/2007.06700/) (2020)
- [DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills](https://mlsanity.com/paper/1804.02717/) (2018)
- [Self-supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation](https://mlsanity.com/paper/1709.10489/) (2017)
- [NICGSlowDown: Evaluating the Efficiency Robustness of Neural Image Caption Generation Models](https://mlsanity.com/paper/2203.15859/) (2022)
- [Real-Time Reinforcement Learning](https://mlsanity.com/paper/1911.04448/) (2019)
- [Robust Multi-Agent Reinforcement Learning with State Uncertainty](https://mlsanity.com/paper/2307.16212/) (2023)
- [Sibling-Attack: Rethinking Transferable Adversarial Attacks against Face Recognition](https://mlsanity.com/paper/2303.12512/) (2023)
- [Few-Shot Continual Active Learning by a Robot](https://mlsanity.com/paper/2210.04137/) (2022)
- [Revisiting Self-Training for Few-Shot Learning of Language Model](https://mlsanity.com/paper/2110.01256/) (2021)
- [Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size](https://mlsanity.com/paper/2211.11092/) (2022)
- [PolicyCleanse: Backdoor Detection and Mitigation in Reinforcement Learning](https://mlsanity.com/paper/2202.03609/) (2022)
- [Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework](https://mlsanity.com/paper/2210.16798/) (2022)
- [DeepPerform: An Efficient Approach for Performance Testing of Resource-Constrained Neural Networks](https://mlsanity.com/paper/2210.05370/) (2022)
- [Continual Learning In Environments With Polynomial Mixing Times](https://mlsanity.com/paper/2112.07066/) (2021)
- [Dynamic Transformers Provide a False Sense of Efficiency](https://mlsanity.com/paper/2305.12228/) (2023)
- [NVRadarNet: Real-Time Radar Obstacle and Free Space Detection for Autonomous Driving](https://mlsanity.com/paper/2209.14499/) (2022)
- [Learning to Reverse DNNs from AI Programs Automatically](https://mlsanity.com/paper/2205.10364/) (2022)

## Built on by (most cited first, 5 of 5)

- [DLoRA: Distributed Parameter-Efficient Fine-Tuning Solution for Large Language Model](https://mlsanity.com/paper/2404.05182/) (2024)
- [Your Compiler is Backdooring Your Model: Understanding and Exploiting Compilation Inconsistency Vulnerabilities in Deep Learning Compilers](https://mlsanity.com/paper/2509.11173/) (2025)
- [Visual SLAM with 3D Gaussian Primitives and Depth Priors Enabling Novel View Synthesis](https://mlsanity.com/paper/2408.05635/) (2024)
- [Knowledge capture, adaptation and composition (KCAC): A framework for cross-task curriculum learning in robotic manipulation](https://mlsanity.com/paper/2505.10522/) (2025)
- [TetraRL: A Self-Adaptive Runtime for On-Device Deep Reinforcement Learning Systems](https://mlsanity.com/paper/2606.13891/) (2026)

## Abstract (arXiv)

Autonomous robotic systems, like autonomous vehicles and robotic search and rescue, require efficient on-device training for continuous adaptation of Deep Reinforcement Learning (DRL) models in dynamic environments. This research is fundamentally motivated by the need to understand and address the challenges of on-device real-time DRL, which involves balancing timing and algorithm performance under memory constraints, as exposed through our extensive empirical studies. This intricate balance requires co-optimizing two pivotal parameters of DRL training -- batch size and replay buffer size. Configuring these parameters significantly affects timing and algorithm performance, while both (unfortunately) require substantial memory allocation to achieve near-optimal performance. This paper presents R^3, a holistic solution for managing timing, memory, and algorithm performance in on-device real-time DRL training. R^3 employs (i) a deadline-driven feedback loop with dynamic batch sizing for optimizing timing, (ii) efficient memory management to reduce memory footprint and allow larger replay buffer sizes, and (iii) a runtime coordinator guided by heuristic analysis and a runtime profiler for dynamically adjusting memory resource reservations. These components collaboratively tackle the trade-offs in on-device DRL training, improving timing and algorithm performance while minimizing the risk of out-of-memory (OOM) errors. We implemented and evaluated R^3 extensively across various DRL frameworks and benchmarks on three hardware platforms commonly adopted by autonomous robotic systems. Additionally, we integrate R^3 with a popular realistic autonomous car simulator to demonstrate its real-world applicability. Evaluation results show that R^3 achieves efficacy across diverse platforms, ensuring consistent latency performance and timing predictability with minimal overhead.