# A Multimodal Classifier Generative Adversarial Network for Carry and Place Tasks from Ambiguous Language Instructions

arXiv: https://arxiv.org/abs/1806.03847 · HTML: https://mlsanity.com/paper/1806.03847/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2018.

## Built on by (most cited first, 7 of 7)

- [CrossMap Transformer: A Crossmodal Masked Path Transformer Using Double Back-Translation for Vision-and-Language Navigation](https://mlsanity.com/paper/2103.00852/) (2021)
- [Unified Questioner Transformer for Descriptive Question Generation in Goal-Oriented Visual Dialogue](https://mlsanity.com/paper/2106.15550/) (2021)
- [LatteGAN: Visually Guided Language Attention for Multi-Turn Text-Conditioned Image Manipulation](https://mlsanity.com/paper/2112.13985/) (2021)
- [DM2RM: Dual-Mode Multimodal Ranking for Target Objects and Receptacles Based on Open-Vocabulary Instructions](https://mlsanity.com/paper/2408.07910/) (2024)
- [3D Instruction Ambiguity Detection](https://mlsanity.com/paper/2601.05991/) (2026)
- [Predicting and Attending to Damaging Collisions for Placing Everyday Objects in Photo-Realistic Simulations](https://mlsanity.com/paper/2102.06507/) (2021)
- [ESearch-R1: Learning Cost-Aware MLLM Agents for Interactive Embodied Search via Reinforcement Learning](https://mlsanity.com/paper/2512.18571/) (2025)

## Abstract (arXiv)

This paper focuses on a multimodal language understanding method for carry-and-place tasks with domestic service robots. We address the case of ambiguous instructions, that is, when the target area is not specified. For instance "put away the milk and cereal" is a natural instruction where there is ambiguity regarding the target area, considering environments in daily life. Conventionally, this instruction can be disambiguated from a dialogue system, but at the cost of time and cumbersome interaction. Instead, we propose a multimodal approach, in which the instructions are disambiguated using the robot's state and environment context. We develop the Multi-Modal Classifier Generative Adversarial Network (MMC-GAN) to predict the likelihood of different target areas considering the robot's physical limitation and the target clutter. Our approach, MMC-GAN, significantly improves accuracy compared with baseline methods that use instructions only or simple deep neural networks.