mlsanityData to 13 Oct 2026

Which One Are You Referring To? Multimodal Object Identification in Situated Dialogue

arXiv 2302.14680 ↗First posted 28 Feb 2023cs.CL cs.AI cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

The demand for multimodal dialogue systems has been rising in various domains, emphasizing the importance of interpreting multimodal inputs from conversational and situational contexts. We explore three methods to tackle this problem and evaluate them on the largest situated dialogue dataset, SIMMC 2.1. Our best method, scene-dialogue alignment, improves the performance by ~20% F1-score compared to the SIMMC 2.1 baselines. We provide analysis and discussion regarding the limitation of our methods and the potential directions for future works. Our code is publicly available at https://github.com/holylovenia/multimodal-object-identification.

Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 34 of its 52 references that are in the map. First posted on arXiv 28 Feb 2023.

Builds on

34 of 52 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  5. 2020End-to-End Object Detection with Transformerscited by 7,396
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,855
  7. 2015VQA: Visual Question Answeringcited by 2,495
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,450
  9. 2016Modeling Context in Referring Expressionscited by 1,057
  10. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 898
  11. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 815
  12. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 648
22 more
  1. 2016Visual Dialogcited by 377
  2. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  3. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 213
  4. 2015End-to-end people detection in crowded scenescited by 140
  5. 2015Natural Language Object Retrievalcited by 124
  6. 2019Audio-Visual Scene-Aware Dialogcited by 122
  7. 2018Image Chat: Engaging Grounded Conversationscited by 76
  8. 2019CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressionscited by 73
  9. 2017Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generationcited by 59
  10. 2021SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversationscited by 55
  11. 2018Visual Coreference Resolution in Visual Dialog using Neural Module Networkscited by 54
  12. 2019CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialogcited by 43
  13. 2021Multimodal End-to-End Sparse Model for Emotion Recognitioncited by 40
  14. 2021Discourse Probing of Pretrained Language Modelscited by 38
  15. 2020Situated and Interactive Multimodal Conversationscited by 35
  16. 2021Multimodal Dialogue Response Generationcited by 32
  17. 2017Visual Reference Resolution using Attention Memory for Visual Dialogcited by 28
  18. 2022Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioningcited by 25
  19. 2021Pragmatic competence of pre-trained language models through the lens of discourse connectivescited by 16
  20. 2022Multimodal Conversational AI: A Survey of Datasets and Approachescited by 16
  21. 2022Every picture tells a story: Image-grounded controllable stylistic story generationcited by 5
  22. 2018ScoutBot: A Dialogue System for Collaborative Navigationcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80