mlsanityData to 14 Oct 2026

Case Relation Transformer: A Crossmodal Language Generation Model for Fetching Instructions

arXiv 2107.00789 ↗First posted 2 Jul 2021cs.RO cs.CL cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

There have been many studies in robotics to improve the communication skills of domestic service robots. Most studies, however, have not fully benefited from recent advances in deep neural networks because the training datasets are not large enough. In this paper, our aim is to augment the datasets based on a crossmodal language generation model. We propose the Case Relation Transformer (CRT), which generates a fetching instruction sentence from an image, such as "Move the blue flip-flop to the lower left box." Unlike existing methods, the CRT uses the Transformer to integrate the visual features and geometry features of objects in the image. The CRT can handle the objects because of the Case Relation Block. We conducted comparison experiments and a human evaluation. The experimental results show the CRT outperforms baseline methods.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 23 of its 30 references that are in the map. First posted on arXiv 2 Jul 2021.

Builds on

23 of 30 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  10. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
11 more
  1. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 198
  2. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  3. 2019Image Captioning: Transforming Objects into Wordscited by 171
  4. 2014Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)cited by 94
  5. 2017Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructionscited by 62
  6. 2019Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methodscited by 55
  7. 2021On the Evaluation of Vision-and-Language Navigation Instructionscited by 45
  8. 2017Stack-Captioning: Coarse-to-Fine Learning for Image Captioningcited by 27
  9. 2019A Multimodal Target-Source Classifier with Attention Branches to Understand Ambiguous Instructions for Fetching Daily Objectscited by 6
  10. 2020Alleviating the Burden of Labeling: Sentence Generation by Attention Branch Encoder-Decoder Networkcited by 6
  11. 2019Multimodal Attention Branch Network for Perspective-Free Sentence Generationcited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 1.

4 papers.

  1. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
  2. 2023JaSPICE: Automatic Evaluation Metric Using Predicate-Argument Structures for Image Captioning Modelscited by 6
  3. 2022Relational Future Captioning Model for Explaining Likely Collisions in Daily Taskscited by 2
  4. 2023Fully Automated Task Management for Generation, Execution, and Evaluation: A Framework for Fetch-and-Carry Tasks with Natural Language Instructions in Continuous Spacecited by 0