# Visually Grounded Task and Motion Planning for Mobile Manipulation

arXiv: https://arxiv.org/abs/2202.10667 · HTML: https://mlsanity.com/paper/2202.10667/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2022.

## Built on by (most cited first, 5 of 5)

- [RoCo: Dialectic Multi-Robot Collaboration with Large Language Models](https://mlsanity.com/paper/2307.04738/) (2023)
- [Grounding Classical Task Planners via Vision-Language Models](https://mlsanity.com/paper/2304.08587/) (2023)
- [DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planning](https://mlsanity.com/paper/2406.17659/) (2024)
- [ORLA*: Mobile Manipulator-Based Object Rearrangement with Lazy A Star](https://mlsanity.com/paper/2309.13707/) (2023)
- [Future Predictive Success-or-Failure Classification for Long-Horizon Robotic Tasks](https://mlsanity.com/paper/2404.03415/) (2024)

## Abstract (arXiv)

Task and motion planning (TAMP) algorithms aim to help robots achieve task-level goals, while maintaining motion-level feasibility. This paper focuses on TAMP domains that involve robot behaviors that take extended periods of time (e.g., long-distance navigation). In this paper, we develop a visual grounding approach to help robots probabilistically evaluate action feasibility, and introduce a TAMP algorithm, called GROP, that optimizes both feasibility and efficiency. We have collected a dataset that includes 96,000 simulated trials of a robot conducting mobile manipulation tasks, and then used the dataset to learn to ground symbolic spatial relationships for action feasibility evaluation. Compared with competitive TAMP baselines, GROP exhibited a higher task-completion rate while maintaining lower or comparable action costs. In addition to these extensive experiments in simulation, GROP is fully implemented and tested on a real robot system.