# Mobile Robot Manipulation using Pure Object Detection

arXiv: https://arxiv.org/abs/2201.12437 · HTML: https://mlsanity.com/paper/2201.12437/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 71 references that are in the map.
- First posted on arXiv 28 Jan 2022.

## Builds on (most cited first, 30 of 42)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [CORe50: a New Dataset and Benchmark for Continuous Object Recognition](https://mlsanity.com/paper/1705.03550/) (2017)
- [Frustratingly Simple Few-Shot Object Detection](https://mlsanity.com/paper/2003.06957/) (2020)
- [Few-shot Object Detection via Feature Reweighting](https://mlsanity.com/paper/1812.01866/) (2018)
- [Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation](https://mlsanity.com/paper/1908.07433/) (2019)
- [Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours](https://mlsanity.com/paper/1509.06825/) (2015)
- [Meta R-CNN : Towards General Solver for Instance-level Few-shot Learning](https://mlsanity.com/paper/1909.13032/) (2019)
- [Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks](https://mlsanity.com/paper/1812.07252/) (2018)
- [FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding](https://mlsanity.com/paper/2103.05950/) (2021)
- [Few-Shot Object Detection and Viewpoint Estimation for Objects in the Wild](https://mlsanity.com/paper/2007.12107/) (2020)
- [TossingBot: Learning to Throw Arbitrary Objects with Residual Physics](https://mlsanity.com/paper/1903.11239/) (2019)
- [DeFRCN: Decoupled Faster R-CNN for Few-Shot Object Detection](https://mlsanity.com/paper/2108.09017/) (2021)
- [Incremental Few-Shot Object Detection](https://mlsanity.com/paper/2003.04668/) (2020)
- [Self-supervised 6D Object Pose Estimation for Robot Manipulation](https://mlsanity.com/paper/1909.10159/) (2019)
- [Generalized Few-Shot Object Detection without Forgetting](https://mlsanity.com/paper/2105.09491/) (2021)
- [RepMet: Representative-based metric learning for classification and one-shot object detection](https://mlsanity.com/paper/1806.04728/) (2018)
- [Beyond Max-Margin: Class Margin Equilibrium for Few-shot Object Detection](https://mlsanity.com/paper/2103.04612/) (2021)
- [Semantic Relation Reasoning for Shot-Stable Few-Shot Object Detection](https://mlsanity.com/paper/2103.01903/) (2021)
- [Interactive Perception: Leveraging Action in Perception and Perception in Action](https://mlsanity.com/paper/1604.03670/) (2016)
- [LSTD: A Low-Shot Transfer Detector for Object Detection](https://mlsanity.com/paper/1803.01529/) (2018)
- [Query Adaptive Few-Shot Object Detection with Heterogeneous Graph Convolutional Networks](https://mlsanity.com/paper/2112.09791/) (2021)
- [Dense Relation Distillation with Context-aware Aggregation for Few-Shot Object Detection](https://mlsanity.com/paper/2103.17115/) (2021)
- [Hallucination Improves Few-Shot Object Detection](https://mlsanity.com/paper/2105.01294/) (2021)

## Built on by (most cited first, 8 of 8)

- [InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policy](https://mlsanity.com/paper/2510.13778/) (2025)
- [Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalization](https://mlsanity.com/paper/2601.12993/) (2026)
- [Detect Everything with Few Examples](https://mlsanity.com/paper/2309.12969/) (2023)
- [ST4VLA: Spatially Guided Training for Vision-Language-Action Models](https://mlsanity.com/paper/2602.10109/) (2026)
- [Kinematically-Decoupled Impedance Control for Fast Object Visual Servoing and Grasping on Quadruped Manipulators](https://mlsanity.com/paper/2307.04918/) (2023)
- [OpenNav: Efficient Open Vocabulary 3D Object Detection for Smart Wheelchair Navigation](https://mlsanity.com/paper/2408.13936/) (2024)
- [Auto-Labeling Data for Object Detection](https://mlsanity.com/paper/2506.02359/) (2025)
- [Feedback-driven object detection and iterative model improvement](https://mlsanity.com/paper/2411.19835/) (2024)

## Abstract (arXiv)

This paper addresses the problem of mobile robot manipulation using object detection. Our approach uses detection and control as complimentary functions that learn from real-world interactions. We develop an end-to-end manipulation method based solely on detection and introduce Task-focused Few-shot Object Detection (TFOD) to learn new objects and settings. Our robot collects its own training data and automatically determines when to retrain detection to improve performance across various subtasks (e.g., grasping). Notably, detection training is low-cost, and our robot learns to manipulate new objects using as few as four clicks of annotation. In physical experiments, our robot learns visual control from a single click of annotation and a novel update formulation, manipulates new objects in clutter and other mobile settings, and achieves state-of-the-art results on an existing visual servo control and depth estimation benchmark. Finally, we develop a TFOD Benchmark to support future object detection research for robotics: https://github.com/griffbr/tfod.