# A Simple Baseline for Audio-Visual Scene-Aware Dialog

arXiv: https://arxiv.org/abs/1904.05876 · HTML: https://mlsanity.com/paper/1904.05876/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2019.

## Built on by (most cited first, 30 of 52)

- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic](https://mlsanity.com/paper/2111.14447/) (2021)
- [Vision Transformers are Parameter-Efficient Audio-Visual Learners](https://mlsanity.com/paper/2212.07983/) (2022)
- [CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2403.04640/) (2024)
- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs](https://mlsanity.com/paper/2101.12059/) (2021)
- [Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback](https://mlsanity.com/paper/2412.15838/) (2024)
- [Perceptual Score: What Data Modalities Does Your Model Perceive?](https://mlsanity.com/paper/2110.14375/) (2021)
- [MeaCap: Memory-Augmented Zero-shot Image Captioning](https://mlsanity.com/paper/2403.03715/) (2024)
- [ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishing](https://mlsanity.com/paper/2303.02437/) (2023)
- [AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistant](https://mlsanity.com/paper/2203.04203/) (2022)
- [On-the-fly Modulation for Balanced Multimodal Learning](https://mlsanity.com/paper/2410.11582/) (2024)
- [Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks](https://mlsanity.com/paper/2311.05152/) (2023)
- [Progressive Spatio-temporal Perception for Audio-Visual Question Answering](https://mlsanity.com/paper/2308.05421/) (2023)
- [Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time](https://mlsanity.com/paper/2407.01851/) (2024)
- [Zero-Shot Video Captioning with Evolving Pseudo-Tokens](https://mlsanity.com/paper/2207.11100/) (2022)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [Object-aware Adaptive-Positivity Learning for Audio-Visual Question Answering](https://mlsanity.com/paper/2312.12816/) (2023)
- [Learning to Retrieve Videos by Asking Questions](https://mlsanity.com/paper/2205.05739/) (2022)
- [Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser](https://mlsanity.com/paper/2305.17343/) (2023)
- [Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering](https://mlsanity.com/paper/2404.12020/) (2024)
- [Boosting Audio Visual Question Answering via Key Semantic-Aware Cues](https://mlsanity.com/paper/2407.20693/) (2024)
- [Latent Space Explanation by Intervention](https://mlsanity.com/paper/2112.04895/) (2021)
- [Patch-level Sounding Object Tracking for Audio-Visual Question Answering](https://mlsanity.com/paper/2412.10749/) (2024)
- [Question-Aware Gaussian Experts for Audio-Visual Question Answering](https://mlsanity.com/paper/2503.04459/) (2025)
- [Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialogues](https://mlsanity.com/paper/2103.00820/) (2021)
- [Video and Text Matching with Conditioned Embeddings](https://mlsanity.com/paper/2110.11298/) (2021)
- [CAD -- Contextual Multi-modal Alignment for Dynamic AVQA](https://mlsanity.com/paper/2310.16754/) (2023)
- [NarrativeBridge: Enhancing Video Captioning with Causal-Temporal Narrative](https://mlsanity.com/paper/2406.06499/) (2024)
- [Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamics Audio-Visual Scenarios](https://mlsanity.com/paper/2305.12397/) (2023)

## Abstract (arXiv)

The recently proposed audio-visual scene-aware dialog task paves the way to a more data-driven way of learning virtual assistants, smart speakers and car navigation systems. However, very little is known to date about how to effectively extract meaningful information from a plethora of sensors that pound the computational engine of those devices. Therefore, in this paper, we provide and carefully analyze a simple baseline for audio-visual scene-aware dialog which is trained end-to-end. Our method differentiates in a data-driven manner useful signals from distracting ones using an attention mechanism. We evaluate the proposed approach on the recently introduced and challenging audio-visual scene-aware dataset, and demonstrate the key features that permit to outperform the current state-of-the-art by more than 20% on CIDEr.