# Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping

arXiv: https://arxiv.org/abs/1910.02787 · HTML: https://mlsanity.com/paper/1910.02787/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 1 Oct 2019.

## Placed in

- Offline RL and imitation learning from data / [Real-world robot offline RL](https://mlsanity.com/offlinerl/real-world-robot-offline-rl/)

## Built on by (most cited first, 29 of 29)

- [Efficient Risk-Averse Reinforcement Learning](https://mlsanity.com/paper/2205.05138/) (2022)
- [RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents](https://mlsanity.com/paper/2102.08159/) (2021)
- [An Analysis of Quantile Temporal-Difference Learning](https://mlsanity.com/paper/2301.04462/) (2023)
- [The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learning](https://mlsanity.com/paper/2305.15703/) (2023)
- [More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning](https://mlsanity.com/paper/2402.07198/) (2024)
- [Discovering Diverse Solutions in Deep Reinforcement Learning by Maximizing State-Action-Based Mutual Information](https://mlsanity.com/paper/2103.07084/) (2021)
- [Risk-Conditioned Distributional Soft Actor-Critic for Risk-Sensitive Navigation](https://mlsanity.com/paper/2104.03111/) (2021)
- [The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation](https://mlsanity.com/paper/2305.18388/) (2023)
- [Learning Risk-Aware Quadrupedal Locomotion using Distributional Reinforcement Learning](https://mlsanity.com/paper/2309.14246/) (2023)
- [The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learning](https://mlsanity.com/paper/2207.07570/) (2022)
- [All the Feels: A dexterous hand with large-area tactile sensing](https://mlsanity.com/paper/2210.15658/) (2022)
- [Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2402.07598/) (2024)
- [Octopus v3: Technical Report for On-device Sub-billion Multimodal AI Agent](https://mlsanity.com/paper/2404.11459/) (2024)
- [Risk-Averse Finetuning of Large Language Models](https://mlsanity.com/paper/2501.06911/) (2025)
- [Safe Reinforcement Learning with Dead-Ends Avoidance and Recovery](https://mlsanity.com/paper/2306.13944/) (2023)
- [Distributional Bellman Operators over Mean Embeddings](https://mlsanity.com/paper/2312.07358/) (2023)
- [Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model](https://mlsanity.com/paper/2410.20312/) (2024)
- [Dejavu: Towards Experience Feedback Learning for Embodied Intelligence](https://mlsanity.com/paper/2510.10181/) (2025)
- [Statistical Efficiency and Inference of Quantile Distributional Reinforcement Learning](https://mlsanity.com/paper/2607.08444/) (2026)
- [Conformal Prediction Beyond the Horizon: Distribution-Free Inference for Policy Evaluation](https://mlsanity.com/paper/2510.26026/) (2025)
- [NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models](https://mlsanity.com/paper/2603.09542/) (2026)
- [RoboSwap: A GAN-driven Video Diffusion Framework For Unsupervised Robot Arm Swapping](https://mlsanity.com/paper/2506.08632/) (2025)
- [Online Inference for Quantile Temporal Difference Learning in Distributional Reinforcement Learning](https://mlsanity.com/paper/2608.12973/) (2026)
- [Robustifying a Policy in Multi-Agent RL with Diverse Cooperative Behaviors and Adversarial Style Sampling for Assistive Tasks](https://mlsanity.com/paper/2403.00344/) (2024)
- [Bellman Unbiasedness: Toward Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation](https://mlsanity.com/paper/2407.21260/) (2024)
- [Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity](https://mlsanity.com/paper/2602.03778/) (2026)
- [Risk-Aware Reinforcement Learning for Mobile Manipulation](https://mlsanity.com/paper/2603.04579/) (2026)
- [Distributional Off-Policy Evaluation with Deep Quantile Process Regression](https://mlsanity.com/paper/2604.18143/) (2026)
- [A Finite-Sample Analysis of Quantile Temporal-Difference Learning](https://mlsanity.com/paper/2608.27313/) (2026)

## Abstract (arXiv)

The distributional perspective on reinforcement learning (RL) has given rise to a series of successful Q-learning algorithms, resulting in state-of-the-art performance in arcade game environments. However, it has not yet been analyzed how these findings from a discrete setting translate to complex practical applications characterized by noisy, high dimensional and continuous state-action spaces. In this work, we propose Quantile QT-Opt (Q2-Opt), a distributional variant of the recently introduced distributed Q-learning algorithm for continuous domains, and examine its behaviour in a series of simulated and real vision-based robotic grasping tasks. The absence of an actor in Q2-Opt allows us to directly draw a parallel to the previous discrete experiments in the literature without the additional complexities induced by an actor-critic architecture. We demonstrate that Q2-Opt achieves a superior vision-based object grasping success rate, while also being more sample efficient. The distributional formulation also allows us to experiment with various risk distortion metrics that give us an indication of how robots can concretely manage risk in practice using a Deep RL control policy. As an additional contribution, we perform batch RL experiments in our virtual environment and compare them with the latest findings from discrete settings. Surprisingly, we find that the previous batch RL findings from the literature obtained on arcade game environments do not generalise to our setup.