# Target-dependent UNITER: A Transformer-Based Multimodal Language Comprehension Model for Domestic Service Robots

arXiv: https://arxiv.org/abs/2107.00811 · HTML: https://mlsanity.com/paper/2107.00811/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 2 Jul 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering](https://mlsanity.com/paper/1712.00377/) (2017)
- [Visual Entailment: A Novel Task for Fine-Grained Image Understanding](https://mlsanity.com/paper/1901.06706/) (2019)
- [GuessWhat?! Visual object discovery through multi-modal dialogue](https://mlsanity.com/paper/1611.08481/) (2016)
- [A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input](https://mlsanity.com/paper/1410.0210/) (2014)
- [Attention Branch Network: Learning of Attention Mechanism for Visual Explanation](https://mlsanity.com/paper/1812.10025/) (2018)
- [CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressions](https://mlsanity.com/paper/1901.00850/) (2019)
- [Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions](https://mlsanity.com/paper/1710.06280/) (2017)
- [Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods](https://mlsanity.com/paper/1907.09358/) (2019)
- [Comprehension-guided referring expressions](https://mlsanity.com/paper/1701.03439/) (2017)
- [Understanding Natural Language Instructions for Fetching Daily Objects Using GAN-Based Multimodal Target-Source Classification](https://mlsanity.com/paper/1906.06830/) (2019)
- [A Multimodal Target-Source Classifier with Attention Branches to Understand Ambiguous Instructions for Fetching Daily Objects](https://mlsanity.com/paper/1912.10675/) (2019)
- [Alleviating the Burden of Labeling: Sentence Generation by Attention Branch Encoder-Decoder Network](https://mlsanity.com/paper/2007.04557/) (2020)

## Built on by (most cited first, 7 of 7)

- [Moment-based Adversarial Training for Embodied Language Comprehension](https://mlsanity.com/paper/2204.00889/) (2022)
- [Multimodal Diffusion Segmentation Model for Object Segmentation from Manipulation Instructions](https://mlsanity.com/paper/2307.08597/) (2023)
- [Switching Head-Tail Funnel UNITER for Dual Referring Expression Comprehension with Fetch-and-Carry Tasks](https://mlsanity.com/paper/2307.07166/) (2023)
- [Learning-To-Rank Approach for Identifying Everyday Objects Using a Physical-World Search Engine](https://mlsanity.com/paper/2312.15844/) (2023)
- [Natural Language Instructions for Intuitive Human Interaction with Robotic Assistants in Field Construction Work](https://mlsanity.com/paper/2307.04195/) (2023)
- [Naming Objects for Vision-and-Language Manipulation](https://mlsanity.com/paper/2303.02871/) (2023)
- [Prototypical Contrastive Transfer Learning for Multimodal Language Understanding](https://mlsanity.com/paper/2307.05942/) (2023)

## Abstract (arXiv)

Currently, domestic service robots have an insufficient ability to interact naturally through language. This is because understanding human instructions is complicated by various ambiguities and missing information. In existing methods, the referring expressions that specify the relationships between objects are insufficiently modeled. In this paper, we propose Target-dependent UNITER, which learns the relationship between the target object and other objects directly by focusing on the relevant regions within an image, rather than the whole image. Our method is an extension of the UNITER-based Transformer that can be pretrained on general-purpose datasets. We extend the UNITER approach by introducing a new architecture for handling the target candidates. Our model is validated on two standard datasets, and the results show that Target-dependent UNITER outperforms the baseline method in terms of classification accuracy.