# A Multimodal Target-Source Classifier with Attention Branches to Understand Ambiguous Instructions for Fetching Daily Objects

arXiv: https://arxiv.org/abs/1912.10675 · HTML: https://mlsanity.com/paper/1912.10675/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2019.

## Built on by (most cited first, 6 of 6)

- [Target-dependent UNITER: A Transformer-Based Multimodal Language Comprehension Model for Domestic Service Robots](https://mlsanity.com/paper/2107.00811/) (2021)
- [Switching Head-Tail Funnel UNITER for Dual Referring Expression Comprehension with Fetch-and-Carry Tasks](https://mlsanity.com/paper/2307.07166/) (2023)
- [Case Relation Transformer: A Crossmodal Language Generation Model for Fetching Instructions](https://mlsanity.com/paper/2107.00789/) (2021)
- [Predicting and Attending to Damaging Collisions for Placing Everyday Objects in Photo-Realistic Simulations](https://mlsanity.com/paper/2102.06507/) (2021)
- [Prototypical Contrastive Transfer Learning for Multimodal Language Understanding](https://mlsanity.com/paper/2307.05942/) (2023)
- [Learning Multimodal Confidence for Intention Recognition in Human-Robot Interaction](https://mlsanity.com/paper/2405.14116/) (2024)

## Abstract (arXiv)

In this study, we focus on multimodal language understanding for fetching instructions in the domestic service robots context. This task consists of predicting a target object, as instructed by the user, given an image and an unstructured sentence, such as "Bring me the yellow box (from the wooden cabinet)." This is challenging because of the ambiguity of natural language, i.e., the relevant information may be missing or there might be several candidates. To solve such a task, we propose the multimodal target-source classifier model with attention branches (MTCM-AB), which is an extension of the MTCM. Our methodology uses the attention branch network (ABN) to develop a multimodal attention mechanism based on linguistic and visual inputs. Experimental validation using a standard dataset showed that the MTCM-AB outperformed both state-of-the-art methods and the MTCM. In particular the MTCM-AB accuracy on average was 90.1% while human performance was 90.3% on the PFN-PIC dataset.