# Understanding Natural Language Instructions for Fetching Daily Objects Using GAN-Based Multimodal Target-Source Classification

arXiv: https://arxiv.org/abs/1906.06830 · HTML: https://mlsanity.com/paper/1906.06830/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2019.

## Built on by (most cited first, 16 of 16)

- [Language-Conditioned Imitation Learning with Base Skill Priors under Unstructured Data](https://mlsanity.com/paper/2305.19075/) (2023)
- [CrossMap Transformer: A Crossmodal Masked Path Transformer Using Double Back-Translation for Vision-and-Language Navigation](https://mlsanity.com/paper/2103.00852/) (2021)
- [Moment-based Adversarial Training for Embodied Language Comprehension](https://mlsanity.com/paper/2204.00889/) (2022)
- [Target-dependent UNITER: A Transformer-Based Multimodal Language Comprehension Model for Domestic Service Robots](https://mlsanity.com/paper/2107.00811/) (2021)
- [Talk-to-Resolve: Combining scene understanding and spatial dialogue to resolve granular task ambiguity for a collocated robot](https://mlsanity.com/paper/2111.11099/) (2021)
- [Multimodal Diffusion Segmentation Model for Object Segmentation from Manipulation Instructions](https://mlsanity.com/paper/2307.08597/) (2023)
- [Incremental Object Grounding Using Scene Graphs](https://mlsanity.com/paper/2201.01901/) (2022)
- [Learning Visual-Audio Representations for Voice-Controlled Robots](https://mlsanity.com/paper/2109.02823/) (2021)
- [DoRO: Disambiguation of referred object for embodied agents](https://mlsanity.com/paper/2207.14205/) (2022)
- [Switching Head-Tail Funnel UNITER for Dual Referring Expression Comprehension with Fetch-and-Carry Tasks](https://mlsanity.com/paper/2307.07166/) (2023)
- [Learning-To-Rank Approach for Identifying Everyday Objects Using a Physical-World Search Engine](https://mlsanity.com/paper/2312.15844/) (2023)
- [Natural Language Instructions for Intuitive Human Interaction with Robotic Assistants in Field Construction Work](https://mlsanity.com/paper/2307.04195/) (2023)
- [Predicting and Attending to Damaging Collisions for Placing Everyday Objects in Photo-Realistic Simulations](https://mlsanity.com/paper/2102.06507/) (2021)
- [RAIDER: Tool-Equipped Large Language Model Agent for Robotic Action Issue Detection, Explanation and Recovery](https://mlsanity.com/paper/2503.17703/) (2025)
- [Prototypical Contrastive Transfer Learning for Multimodal Language Understanding](https://mlsanity.com/paper/2307.05942/) (2023)
- [Instruct2Act: From Human Instruction to Actions Sequencing and Execution via Robot Action Network for Robotic Manipulation](https://mlsanity.com/paper/2602.09940/) (2026)

## Abstract (arXiv)

In this paper, we address multimodal language understanding for unconstrained fetching instruction in domestic service robots context. A typical fetching instruction such as "Bring me the yellow toy from the white shelf" requires to infer the user intention, that is what object (target) to fetch and from where (source). To solve the task, we propose a Multimodal Target-source Classifier Model (MTCM), which predicts the region-wise likelihood of target and source candidates in the scene. Unlike other methods, MTCM can handle regionwise classification based on linguistic and visual features. We evaluated our approach that outperformed the state-of-the-art method on a standard data set. In addition, we extended MTCM with Generative Adversarial Nets (MTCM-GAN), and enabled simultaneous data augmentation and classification.