mlsanityData to 7 Oct 2026

SPRING: Situated Conversation Agent Pretrained with Multimodal Questions from Incremental Layout Graph

arXiv 2301.01949 ↗First posted 5 Jan 2023cs.CL cs.AI cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Existing multimodal conversation agents have shown impressive abilities to locate absolute positions or retrieve attributes in simple scenarios, but they fail to perform well when complex relative positions and information alignments are involved, which poses a bottleneck in response quality. In this paper, we propose a Situated Conversation Agent Petrained with Multimodal Questions from INcremental Layout Graph (SPRING) with abilities of reasoning multi-hops spatial relations and connecting them with visual attributes in crowded situated scenarios. Specifically, we design two types of Multimodal Question Answering (MQA) tasks to pretrain the agent. All QA pairs utilized during pretraining are generated from novel Incremental Layout Graphs (ILG). QA pair difficulty labels automatically annotated by ILG are used to promote MQA-based Curriculum Learning. Experimental results verify the SPRING's effectiveness, showing that it significantly outperforms state-of-the-art approaches on both SIMMC 1.0 and SIMMC 2.0 datasets.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 27 of its 49 references that are in the map. First posted on arXiv 5 Jan 2023.

Builds on

27 of 49 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  12. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
15 more
  1. 2016Visual Dialogcited by 377
  2. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  3. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  4. 2016GuessWhat?! Visual object discovery through multi-modal dialoguecited by 112
  5. 2021GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injectioncited by 102
  6. 2021SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversationscited by 54
  7. 2019Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systemscited by 52
  8. 2022S2SQL: Injecting Syntax to Question-Schema Interaction Graph Encoder for Text-to-SQL Parserscited by 41
  9. 2020Situated and Interactive Multimodal Conversationscited by 35
  10. 2022Proton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsingcited by 30
  11. 2022SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understandingcited by 28
  12. 2017Evaluating Visual Conversational Agents via Cooperative Human-AI Gamescited by 17
  13. 2021Towards Socially Intelligent Agents with Mental State Transition and Human Utilitycited by 14
  14. 2021SocAoG: Incremental Graph Parsing for Social Relation Inference in Dialoguescited by 9
  15. 2021Multimodal Interactions Using Pretrained Unimodal Models for SIMMC 2.0cited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 0; 2026: 3.

6 papers.

  1. 2023PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Expertscited by 19
  2. 2026Evaluating Scene-based In-Situ Item Labeling for Immersive Conversational Recommendationcited by 4
  3. 2024MCF-VC: Mitigate Catastrophic Forgetting in Class-Incremental Learning for Multimodal Video Captioningcited by 1
  4. 2026Extended to Reality: Prompt Injection in 3D Environmentscited by 1
  5. 2026AMATA: Adaptive Multi-Agent Trajectory Alignment for Knowledge-Intensive Question Answeringcited by 1
  6. 2023Multimodal Recommendation Dialog with Subjective Preference: A New Challenge and Benchmarkcited by 0