# PoseAgent: Budget-Constrained 6D Object Pose Estimation via Reinforcement Learning

arXiv: https://arxiv.org/abs/1612.03779 · HTML: https://mlsanity.com/paper/1612.03779/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Dec 2016.

## Built on by (most cited first, 9 of 9)

- [Tuning computer vision models with task rewards](https://mlsanity.com/paper/2302.08242/) (2023)
- [ReAgent: Point Cloud Registration using Imitation and Reinforcement Learning](https://mlsanity.com/paper/2103.15231/) (2021)
- [Reinforced Attention for Few-Shot Learning and Beyond](https://mlsanity.com/paper/2104.04192/) (2021)
- [PFRL: Pose-Free Reinforcement Learning for 6D Pose Estimation](https://mlsanity.com/paper/2102.12096/) (2021)
- [A Visual Navigation Perspective for Category-Level Object Pose Estimation](https://mlsanity.com/paper/2203.13572/) (2022)
- [SporeAgent: Reinforced Scene-level Plausibility for Object Pose Refinement](https://mlsanity.com/paper/2201.00239/) (2022)
- [Collaborative Multi-Agent Video Fast-Forwarding](https://mlsanity.com/paper/2305.17569/) (2023)
- [CMR-Agent: Learning a Cross-Modal Agent for Iterative Image-to-Point Cloud Registration](https://mlsanity.com/paper/2408.02394/) (2024)
- [Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling](https://mlsanity.com/paper/2608.30821/) (2026)

## Abstract (arXiv)

State-of-the-art computer vision algorithms often achieve efficiency by making discrete choices about which hypotheses to explore next. This allows allocation of computational resources to promising candidates, however, such decisions are non-differentiable. As a result, these algorithms are hard to train in an end-to-end fashion. In this work we propose to learn an efficient algorithm for the task of 6D object pose estimation. Our system optimizes the parameters of an existing state-of-the art pose estimation system using reinforcement learning, where the pose estimation system now becomes the stochastic policy, parametrized by a CNN. Additionally, we present an efficient training algorithm that dramatically reduces computation time. We show empirically that our learned pose estimation procedure makes better use of limited resources and improves upon the state-of-the-art on a challenging dataset. Our approach enables differentiable end-to-end training of complex algorithmic pipelines and learns to make optimal use of a given computational budget.