# Coarse-to-fine Q-attention with Tree Expansion

arXiv: https://arxiv.org/abs/2204.12471 · HTML: https://mlsanity.com/paper/2204.12471/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 26 Apr 2022.

## Builds on (most cited first, 23 of 23)

- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [Transporter Networks: Rearranging the Visual World for Robotic Manipulation](https://mlsanity.com/paper/2010.14406/) (2020)
- [Asymmetric Actor Critic for Image-Based Robot Learning](https://mlsanity.com/paper/1710.06542/) (2017)
- [Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks](https://mlsanity.com/paper/1812.07252/) (2018)
- [Sim-to-Real Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/1806.07851/) (2018)
- [Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning](https://mlsanity.com/paper/1803.09956/) (2018)
- [Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation](https://mlsanity.com/paper/2106.12534/) (2021)
- [Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach](https://mlsanity.com/paper/1804.05172/) (2018)
- [Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations](https://mlsanity.com/paper/1912.04344/) (2019)
- [Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Clutter](https://mlsanity.com/paper/2101.01132/) (2021)
- [Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation](https://mlsanity.com/paper/2105.14829/) (2021)
- [Composable Deep Reinforcement Learning for Robotic Manipulation](https://mlsanity.com/paper/1803.06773/) (2018)
- [MoreFusion: Multi-object Reasoning for 6D Pose Estimation from Volumetric Fusion](https://mlsanity.com/paper/2004.04336/) (2020)
- [Monte-Carlo Tree Search for Efficient Visually Guided Rearrangement Planning](https://mlsanity.com/paper/1904.10348/) (2019)
- [3D Simulation for Robot Arm Control with Deep Q-Learning](https://mlsanity.com/paper/1609.03759/) (2016)
- [Visual Foresight Trees for Object Retrieval from Clutter with Nonprehensile Rearrangement](https://mlsanity.com/paper/2105.02857/) (2021)
- [Extended Tree Search for Robot Task and Motion Planning](https://mlsanity.com/paper/2103.05456/) (2021)
- [Coarse-to-Fine Q-attention with Learned Path Ranking](https://mlsanity.com/paper/2204.01571/) (2022)
- [ReorientBot: Learning Object Reorientation for Specific-Posed Placement](https://mlsanity.com/paper/2202.11092/) (2022)
- [Bingham Policy Parameterization for 3D Rotations in Reinforcement Learning](https://mlsanity.com/paper/2202.03957/) (2022)
- [SafePicking: Learning Safe Object Extraction via Object-Level Mapping](https://mlsanity.com/paper/2202.05832/) (2022)
- [Hierarchical Policy for Non-prehensile Multi-object Rearrangement with Deep Reinforcement Learning and Monte Carlo Tree Search](https://mlsanity.com/paper/2109.08973/) (2021)

## Built on by (most cited first, 9 of 9)

- [Masked World Models for Visual Control](https://mlsanity.com/paper/2206.14244/) (2022)
- [Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulation](https://mlsanity.com/paper/2403.03890/) (2024)
- [Multi-View Masked World Models for Visual Robotic Manipulation](https://mlsanity.com/paper/2302.02408/) (2023)
- [Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets](https://mlsanity.com/paper/2304.08742/) (2023)
- [PerAct2: Benchmarking and Learning for Robotic Bimanual Manipulation Tasks](https://mlsanity.com/paper/2407.00278/) (2024)
- [BiGym: A Demo-Driven Mobile Bi-Manual Manipulation Benchmark](https://mlsanity.com/paper/2407.07788/) (2024)
- [On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning](https://mlsanity.com/paper/2206.03271/) (2022)
- [Continuous Control with Coarse-to-fine Reinforcement Learning](https://mlsanity.com/paper/2407.07787/) (2024)
- [Redundancy-aware Action Spaces for Robot Learning](https://mlsanity.com/paper/2406.04144/) (2024)

## Abstract (arXiv)

Coarse-to-fine Q-attention enables sample-efficient robot manipulation by discretizing the translation space in a coarse-to-fine manner, where the resolution gradually increases at each layer in the hierarchy. Although effective, Q-attention suffers from "coarse ambiguity" - when voxelization is significantly coarse, it is not feasible to distinguish similar-looking objects without first inspecting at a finer resolution. To combat this, we propose to envision Q-attention as a tree that can be expanded and used to accumulate value estimates across the top-k voxels at each Q-attention depth. When our extension, Q-attention with Tree Expansion (QTE), replaces standard Q-attention in the Attention-driven Robot Manipulation (ARM) system, we are able to accomplish a larger set of tasks; especially on those that suffer from "coarse ambiguity". In addition to evaluating our approach across 12 RLBench tasks, we also show that the improved performance is visible in a real-world task involving small objects.