mlsanityData to 14 Oct 2026

Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videos

arXiv 2110.05122 ↗First posted 11 Oct 2021cs.CVcited by 86

Not on the map. Why?

Abstract (arXiv)

360^circ videos convey holistic views for the surroundings of a scene. It provides audio-visual cues beyond pre-determined normal field of views and displays distinctive spatial relations on a sphere. However, previous benchmark tasks for panoramic videos are still limited to evaluate the semantic understanding of audio-visual relationships or spherical spatial property in surroundings. We propose a novel benchmark named Pano-AVQA as a large-scale grounded audio-visual question answering dataset on panoramic videos. Using 5.4K 360^circ video clips harvested online, we collect two types of novel question-answer pairs with bounding-box grounding: spherical spatial relation QAs and audio-visual relation QAs. We train several transformer-based models from Pano-AVQA, where the results suggest that our proposed spherical spatial embeddings and multimodal training objectives fairly contribute to a better semantic understanding of the panoramic surroundings on the dataset.

Key facts. Cited by 86 papers in the map (data through 14 Oct 2026). Builds on 49 of its 66 references that are in the map. First posted on arXiv 11 Oct 2021.

Builds on

49 of 66 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2016Layer Normalizationcited by 5,669
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  12. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
37 more
  1. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  2. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  3. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  4. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  5. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  6. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  7. 2019Clotho: An Audio Captioning Datasetcited by 358
  8. 2017Embodied Question Answeringcited by 336
  9. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  10. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  11. 2018The Sound of Pixelscited by 259
  12. 2017Objects that Soundcited by 256
  13. 2017IQA: Visual Question Answering in Interactive Environmentscited by 155
  14. 2019The Sound of Motionscited by 146
  15. 2019WoodScape: A multi-task, multi-camera fisheye dataset for autonomous drivingcited by 144
  16. 2019Audio-Visual Scene-Aware Dialogcited by 123
  17. 2018Building Generalizable Agents with a Realistic and Rich 3D Environmentcited by 109
  18. 2018Learning to Navigate in Cities Without a Mapcited by 104
  19. 2019Embodied Question Answering in Photorealistic Environments with Point Cloud Perceptioncited by 103
  20. 2018Self-Supervised Generation of Spatial Audio for 360 Videocited by 93
  21. 2017DeepStory: Video Story QA by Deep Embedded Memory Networkscited by 68
  22. 2018Cube Padding for Weakly-Supervised Saliency Prediction in 360deg Videoscited by 66
  23. 20182.5D Visual Soundcited by 63
  24. 2019Self-supervised Moving Vehicle Tracking with Stereo Soundcited by 63
  25. 2020Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representationcited by 59
  26. 2016Leveraging Video Descriptions to Learn Video Question Answeringcited by 55
  27. 2019KnowIT VQA: Answering Knowledge-Based Questions about Videoscited by 53
  28. 2018Learning Conditioned Graph Structures for Interpretable Visual Question Answeringcited by 45
  29. 2017Video Question Answering via Attribute-Augmented Attention Network Learningcited by 39
  30. 2019Temporal Reasoning via Audio Question Answeringcited by 39
  31. 2018A Memory Network Approach for Story-based Temporal Summarization of 360deg Videoscited by 29
  32. 2020Semantic Object Prediction and Spatial Sound Super-Resolution with Binaural Soundscited by 26
  33. 2016Pano2Vid: Automatic Cinematography for Watching 360circ Videoscited by 22
  34. 2017Deep 360 Pilot: Learning a Deep Agent for Piloting through 360deg Sports Videocited by 22
  35. 2018ODSQA: Open-domain Spoken Question Answering Datasetcited by 21
  36. 2018A Deep Ranking Model for Spatio-Temporal Highlight Detection from a 360 Videocited by 16
  37. 2020Self-supervised Neural Audio-Visual Sound Source Localization via Probabilistic Spatial Modelingcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 10; 2023: 18; 2024: 23; 2025: 17; 2026: 17.

86 papers.

Sort:

By area:

  1. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  2. 2022Multimodal Learning with Transformers: A Surveycited by 400
  3. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 191
  4. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  5. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
  6. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 102
  7. 2022Video Question Answering: Datasets, Algorithms and Challengescited by 84
  8. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  9. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  10. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?cited by 47
  11. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  12. 20213D Question Answeringcited by 41
38 more
  1. 2023STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Eventscited by 41
  2. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 39
  3. 2022Deep Learning for Omnidirectional Vision: A Survey and New Perspectivescited by 38
  4. 2024BAT: Learning to Reason about Spatial Sounds with Large Language Modelscited by 36
  5. 2025Audio-centric Video Understanding Benchmark without Text Shortcutcited by 36
  6. 2024On-the-fly Modulation for Balanced Multimodal Learningcited by 32
  7. 2023Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Taskscited by 31
  8. 2023Progressive Spatio-temporal Perception for Audio-Visual Question Answeringcited by 29
  9. 2023Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answeringcited by 29
  10. 2022Bloom Library: Multimodal Datasets in 300+ Languages for a Variety of Downstream Taskscited by 28
  11. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Timecited by 26
  12. 2024Towards Open-Vocabulary Audio-Visual Event Localizationcited by 26
  13. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 25
  14. 2023Connecting Vision and Language with Video Localized Narrativescited by 24
  15. 2025video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Modelcited by 24
  16. 2025Are Multimodal Large Language Models Ready for Omnidirectional Spatial Reasoning?cited by 23
  17. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  18. 2023Audio-Visual Instance Segmentationcited by 21
  19. 2024Advancing Weakly-Supervised Audio-Visual Video Parsing via Segment-wise Pseudo Labelingcited by 20
  20. 2023Object-aware Adaptive-Positivity Learning for Audio-Visual Question Answeringcited by 19
  21. 2023Improving Audio-Visual Video Parsing with Pseudo Visual Labelscited by 18
  22. 2023Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parsercited by 18
  23. 2024Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answeringcited by 18
  24. 2025Learning Musical Representations for Music Performance Question Answeringcited by 17
  25. 2022Panoramic Vision Transformer for Saliency Detection in 360deg Videoscited by 15
  26. 2025SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearingcited by 15
  27. 2024Boosting Audio Visual Question Answering via Key Semantic-Aware Cuescited by 14
  28. 2023Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoningcited by 13
  29. 2024Patch-level Sounding Object Tracking for Audio-Visual Question Answeringcited by 13
  30. 2022Exploring Fine-Grained Audiovisual Categorization with the SSW60 Datasetcited by 12
  31. 2023A Review of ChatGPT Applications in Education, Marketing, Software Engineering, and Healthcare: Benefits, Drawbacks, and Research Directionscited by 12
  32. 2025Question-Aware Gaussian Experts for Audio-Visual Question Answeringcited by 12
  33. 2023CAD -- Contextual Multi-modal Alignment for Dynamic AVQAcited by 10
  34. 2024CoLeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised Audio-Visual Video Parsingcited by 10
  35. 2024NarrativeBridge: Enhancing Video Captioning with Causal-Temporal Narrativecited by 10
  36. 2025Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptationcited by 10
  37. 2024Video Question Answering for People with Visual Impairments Using an Egocentric 360-Degree Cameracited by 9
  38. 2023Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamics Audio-Visual Scenarioscited by 8

50 most cited of 86 All 86 (JSON)