mlsanityData to 7 Oct 2026

Multimodal Contextualized Semantic Parsing from Speech

arXiv 2406.06438 ↗First posted 10 Jun 2024cs.CL cs.CV cs.HCcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce Semantic Parsing in Contextual Environments (SPICE), a task designed to enhance artificial agents' contextual awareness by integrating multimodal inputs with prior contexts. SPICE goes beyond traditional semantic parsing by offering a structured, interpretable framework for dynamically updating an agent's knowledge with new information, mirroring the complexity of human communication. We develop the VG-SPICE dataset, crafted to challenge agents with visual scene graph construction from spoken conversational exchanges, highlighting speech and visual data integration. We also present the Audio-Vision Dialogue Scene Parser (AViD-SP) developed for use on VG-SPICE. These innovations aim to improve multimodal information processing and integration. Both the VG-SPICE dataset and the AViD-SP model are publicly available.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 49 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

27 of 49 references.

References · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  9. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  10. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  11. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 376
  12. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
15 more
  1. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 191
  2. 2019CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databasescited by 180
  3. 2020TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Trackingcited by 130
  4. 2018The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselinescited by 90
  5. 2018A Survey on Semantic Parsingcited by 62
  6. 2021SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversationscited by 54
  7. 2021Orthogonal Projection Losscited by 44
  8. 2020Conversational Semantic Parsing for Dialog State Trackingcited by 40
  9. 2018Large-Scale Visual Relationship Understandingcited by 33
  10. 2020Weakly Supervised Visual Semantic Parsingcited by 28
  11. 2021Slot Self-Attentive Dialogue State Trackingcited by 27
  12. 2022STOP: A dataset for Spoken Task Oriented Semantic Parsingcited by 22
  13. 2020Context Dependent Semantic Parsing: A Surveycited by 10
  14. 2021Contextual Semantic Parsing for Multilingual Task-Oriented Dialoguescited by 10
  15. 2022Visual Semantic Parsing: From Images to Abstract Meaning Representationcited by 4

Built on by

Not cited yet.