# Offline Reinforcement Learning for Visual Navigation

arXiv: https://arxiv.org/abs/2212.08244 · HTML: https://mlsanity.com/paper/2212.08244/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 16 Dec 2022.

## Placed in

- Offline RL and imitation learning from data / [Language-conditioned policies and VLAs](https://mlsanity.com/offlinerl/language-conditioned-policies-and-vlas/)

## Builds on (most cited first, 30 of 30)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [Learning to Explore using Active Neural SLAM](https://mlsanity.com/paper/2004.05155/) (2020)
- [Offline Reinforcement Learning with Fisher Divergence Critic Regularization](https://mlsanity.com/paper/2103.08050/) (2021)
- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [Semi-parametric Topological Memory for Navigation](https://mlsanity.com/paper/1803.00653/) (2018)
- [Planning with Goal-Conditioned Policies](https://mlsanity.com/paper/1911.08453/) (2019)
- [SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks](https://mlsanity.com/paper/1609.05130/) (2016)
- [Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills](https://mlsanity.com/paper/2104.07749/) (2021)
- [Rapid Exploration for Open-World Navigation with Latent Goal Models](https://mlsanity.com/paper/2104.05859/) (2021)
- [IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data](https://mlsanity.com/paper/1911.05321/) (2019)
- [COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning](https://mlsanity.com/paper/2010.14500/) (2020)
- [ViNG: Learning Open-World Navigation with Visual Goals](https://mlsanity.com/paper/2012.09812/) (2020)
- [Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes](https://mlsanity.com/paper/2110.06192/) (2021)
- [PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planning](https://mlsanity.com/paper/1710.03937/) (2017)
- [Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement](https://mlsanity.com/paper/2002.11089/) (2020)
- [gradSLAM: Automagically differentiable SLAM](https://mlsanity.com/paper/1910.10672/) (2019)
- [Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation](https://mlsanity.com/paper/1810.07167/) (2018)

## Built on by (most cited first, 29 of 29)

- [RVT: Robotic View Transformer for 3D Object Manipulation](https://mlsanity.com/paper/2306.14896/) (2023)
- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis](https://mlsanity.com/paper/2312.08782/) (2023)
- [TorchRL: A data-driven decision-making library for PyTorch](https://mlsanity.com/paper/2306.00577/) (2023)
- [TrackVLA: Embodied Visual Tracking in the Wild](https://mlsanity.com/paper/2505.23189/) (2025)
- [Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RL](https://mlsanity.com/paper/2404.09857/) (2024)
- [EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videos](https://mlsanity.com/paper/2504.11732/) (2025)
- [FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing](https://mlsanity.com/paper/2304.09831/) (2023)
- [TrackVLA++: Unleashing Reasoning and Memory Capabilities in VLA Models for Embodied Visual Tracking](https://mlsanity.com/paper/2510.07134/) (2025)
- [Learning Speed Adaptation for Flight in Clutter](https://mlsanity.com/paper/2403.04586/) (2024)
- [One-4-All: Neural Potential Fields for Embodied Navigation](https://mlsanity.com/paper/2303.04011/) (2023)
- [From reactive to cognitive: brain-inspired spatial intelligence for embodied agents](https://mlsanity.com/paper/2508.17198/) (2025)
- [Scaling Vision-and-Language Navigation With Offline RL](https://mlsanity.com/paper/2403.18454/) (2024)
- [Beyond Human Demonstrations: Diffusion-Based Reinforcement Learning to Generate Data for VLA Training](https://mlsanity.com/paper/2509.19752/) (2025)
- [Feudal Networks for Visual Navigation](https://mlsanity.com/paper/2402.12498/) (2024)
- [Visual Semantic Navigation with Real Robots](https://mlsanity.com/paper/2311.16623/) (2023)
- [Urban-Focused Multi-Task Offline Reinforcement Learning with Contrastive Data Sharing](https://mlsanity.com/paper/2406.14054/) (2024)
- [Run, Ruminate, and Regulate: A Dual-process Thinking System for Vision-and-Language Navigation](https://mlsanity.com/paper/2511.14131/) (2025)
- [Mildly Constrained Evaluation Policy for Offline Reinforcement Learning](https://mlsanity.com/paper/2306.03680/) (2023)
- [Avoidance Navigation Based on Offline Pre-Training Reinforcement Learning](https://mlsanity.com/paper/2308.01551/) (2023)
- [VAPOR: Legged Robot Navigation in Outdoor Vegetation Using Offline Reinforcement Learning](https://mlsanity.com/paper/2309.07832/) (2023)
- [EnCoMP: Enhanced Covert Maneuver Planning with Adaptive Threat-Aware Visibility Estimation using Offline Reinforcement Learning](https://mlsanity.com/paper/2403.20016/) (2024)
- [SCALE: Self-Correcting Visual Navigation for Mobile Robots via Anti-Novelty Estimation](https://mlsanity.com/paper/2404.10675/) (2024)
- [Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach](https://mlsanity.com/paper/2505.06482/) (2025)
- [Inference Time Policy Optimization for Offline RL with Differentiable World Models](https://mlsanity.com/paper/2603.22430/) (2026)
- [Learning from Demonstration with Failure Awareness for Safe Robot Navigation](https://mlsanity.com/paper/2604.23360/) (2026)
- [FeudalNav: A Simple Framework for Visual Navigation](https://mlsanity.com/paper/2602.06974/) (2026)
- [120 Minutes and a Laptop: Minimalist Image-goal Navigation via Unsupervised Exploration and Offline RL](https://mlsanity.com/paper/2603.26441/) (2026)
- [CST-WM: A Causally Structured World Model for Embodied Visual Tracking](https://mlsanity.com/paper/2609.06302/) (2026)

## Abstract (arXiv)

Reinforcement learning can enable robots to navigate to distant goals while optimizing user-specified reward functions, including preferences for following lanes, staying on paved paths, or avoiding freshly mowed grass. However, online learning from trial-and-error for real-world robots is logistically challenging, and methods that instead can utilize existing datasets of robotic navigation data could be significantly more scalable and enable broader generalization. In this paper, we present ReViND, the first offline RL system for robotic navigation that can leverage previously collected data to optimize user-specified reward functions in the real-world. We evaluate our system for off-road navigation without any additional data collection or fine-tuning, and show that it can navigate to distant goals using only offline training from this dataset, and exhibit behaviors that qualitatively differ based on the user-specified reward function.