# Visually Grounding Language Instruction for History-Dependent Manipulation

arXiv: https://arxiv.org/abs/2012.08977 · HTML: https://mlsanity.com/paper/2012.08977/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Dec 2020.

## Built on by (most cited first, 5 of 5)

- [R2-Bench: Benchmarking the Robustness of Referring Perception Models under Perturbations](https://mlsanity.com/paper/2403.04924/) (2024)
- [CodeGraphVLP: Code-as-Planner Meets Semantic-Graph State for Non-Markovian Vision-Language-Action Models](https://mlsanity.com/paper/2604.22238/) (2026)
- [PARTNR: Pick and place Ambiguity Resolving by Trustworthy iNteractive leaRning](https://mlsanity.com/paper/2211.08304/) (2022)
- [tagE: Enabling an Embodied Agent to Understand Human Instructions](https://mlsanity.com/paper/2310.15605/) (2023)
- [Attribute-based Object Grounding and Robot Grasp Detection with Spatial Reasoning](https://mlsanity.com/paper/2509.08126/) (2025)

## Abstract (arXiv)

This paper emphasizes the importance of a robot's ability to refer to its task history, especially when it executes a series of pick-and-place manipulations by following language instructions given one by one. The advantage of referring to the manipulation history can be categorized into two folds: (1) the language instructions omitting details but using expressions referring to the past can be interpreted, and (2) the visual information of objects occluded by previous manipulations can be inferred. For this, we introduce a history-dependent manipulation task which objective is to visually ground a series of language instructions for proper pick-and-place manipulations by referring to the past. We also suggest a relevant dataset and model which can be a baseline, and show that our model trained with the proposed dataset can also be applied to the real world based on the CycleGAN. Our dataset and code are publicly available on the project website: https://sites.google.com/view/history-dependent-manipulation.