# Self-supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation

arXiv: https://arxiv.org/abs/1709.10489 · HTML: https://mlsanity.com/paper/1709.10489/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Sep 2017.

## Built on by (most cited first, 30 of 77)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Large Language Models for Robotics: A Survey](https://mlsanity.com/paper/2311.07226/) (2023)
- [QUAR-VLA: Vision-Language-Action Model for Quadruped Robots](https://mlsanity.com/paper/2312.14457/) (2023)
- [VTNet: Visual Transformer Network for Object Goal Navigation](https://mlsanity.com/paper/2105.09447/) (2021)
- [OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigation](https://mlsanity.com/paper/2509.19480/) (2025)
- [Fully Autonomous Real-World Reinforcement Learning with Applications to Mobile Manipulation](https://mlsanity.com/paper/2107.13545/) (2021)
- [Hierarchical Object-to-Zone Graph for Object Navigation](https://mlsanity.com/paper/2109.02066/) (2021)
- [Unsupervised Skill Discovery with Bottleneck Option Learning](https://mlsanity.com/paper/2106.14305/) (2021)
- [Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey](https://mlsanity.com/paper/2110.01411/) (2021)
- [Self-Supervised Euphemism Detection and Identification for Content Moderation](https://mlsanity.com/paper/2103.16808/) (2021)
- [A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions](https://mlsanity.com/paper/2109.03540/) (2021)
- [ExAug: Robot-Conditioned Navigation Policies via Geometric Experience Augmentation](https://mlsanity.com/paper/2210.07450/) (2022)
- [In-Distribution Barrier Functions: Self-Supervised Policy Filters that Avoid Out-of-Distribution States](https://mlsanity.com/paper/2301.12012/) (2023)
- [XR-1: Towards Versatile Vision-Language-Action Models via Learning Unified Vision-Motion Representations](https://mlsanity.com/paper/2511.02776/) (2025)
- [Learning Robotic Navigation from Experience: Principles, Methods, and Recent Results](https://mlsanity.com/paper/2212.06759/) (2022)
- [Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directions](https://mlsanity.com/paper/2502.15336/) (2025)
- [Learning to Drive Anywhere with Model-Based Reannotation](https://mlsanity.com/paper/2505.05592/) (2025)
- [Mutual Information Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/2210.07484/) (2022)
- [Probabilistic Constraint for Safety-Critical Reinforcement Learning](https://mlsanity.com/paper/2306.17279/) (2023)
- [A Vision Based Deep Reinforcement Learning Algorithm for UAV Obstacle Avoidance](https://mlsanity.com/paper/2103.06403/) (2021)
- [Agent-Centric Relation Graph for Object Visual Navigation](https://mlsanity.com/paper/2111.14422/) (2021)
- [Generalization in Deep Reinforcement Learning for Robotic Navigation by Reward Shaping](https://mlsanity.com/paper/2209.14271/) (2022)
- [Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learning](https://mlsanity.com/paper/2411.12155/) (2024)
- [AsyncVLA: An Asynchronous VLA for Fast and Robust Navigation on the Edge](https://mlsanity.com/paper/2602.13476/) (2026)
- [Self-Consistent Models and Values](https://mlsanity.com/paper/2110.12840/) (2021)
- [Towards Disturbance-Free Visual Mobile Manipulation](https://mlsanity.com/paper/2112.12612/) (2021)
- [Learning Forward Dynamics Model and Informed Trajectory Sampler for Safe Quadruped Navigation](https://mlsanity.com/paper/2204.08647/) (2022)
- [Improving and Benchmarking Offline Reinforcement Learning Algorithms](https://mlsanity.com/paper/2306.00972/) (2023)

## Abstract (arXiv)

Enabling robots to autonomously navigate complex environments is essential for real-world deployment. Prior methods approach this problem by having the robot maintain an internal map of the world, and then use a localization and planning method to navigate through the internal map. However, these approaches often include a variety of assumptions, are computationally intensive, and do not learn from failures. In contrast, learning-based methods improve as the robot acts in the environment, but are difficult to deploy in the real-world due to their high sample complexity. To address the need to learn complex policies with few samples, we propose a generalized computation graph that subsumes value-based model-free methods and model-based methods, with specific instantiations interpolating between model-free and model-based. We then instantiate this graph to form a navigation model that learns from raw images and is sample efficient. Our simulated car experiments explore the design decisions of our navigation model, and show our approach outperforms single-step and N-step double Q-learning. We also evaluate our approach on a real-world RC car and show it can learn to navigate through a complex indoor environment with a few hours of fully autonomous, self-supervised training. Videos of the experiments and code can be found at github.com/gkahn13/gcg