mlsanityData to 7 Oct 2026

Audio-Visual Instance Segmentation

arXiv 2310.18709 ↗First posted 28 Oct 2023cs.CV cs.LG cs.MMcited by 21

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a new multi-modal task, termed audio-visual instance segmentation (AVIS), which aims to simultaneously identify, segment and track individual sounding object instances in audible videos. To facilitate this research, we introduce a high-quality benchmark named AVISeg, containing over 90K instance masks from 26 semantic categories in 926 long videos. Additionally, we propose a strong baseline model for this task. Our model first localizes sound source within each frame, and condenses object-specific contexts into concise tokens. Then it builds long-range audio-visual dependencies between these tokens using window-based attention, and tracks sounding objects among the entire video sequences. Extensive experiments reveal that our method performs best on AVISeg, surpassing the existing methods from related tasks. We further conduct the evaluation on several multi-modal large models. Unfortunately, they exhibits subpar performance on instance-level sound source localization and temporal perception. We expect that AVIS will inspire the community towards a more comprehensive multi-modal understanding. Dataset and code is available at https://github.com/ruohaoguo/avis.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 64 of its 97 references that are in the map. First posted on arXiv 28 Oct 2023.

Builds on

64 of 97 references.

References · 64

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2023Segment Anythingcited by 8,158
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2017Mask R-CNNcited by 7,259
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
38 more
  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  2. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  5. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  6. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  7. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  8. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  9. 2020HOTA: A Higher Order Metric for Evaluating Multi-Object Trackingcited by 546
  10. 2019Hybrid Task Cascade for Instance Segmentationcited by 479
  11. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  12. 2019Video Instance Segmentationcited by 391
  13. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  14. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 320
  15. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  16. 2023Tracking Anything with Decoupled Video Segmentationcited by 197
  17. 2018Learning to Localize Sound Source in Visual Scenescited by 188
  18. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  19. 2021Localizing Visual Sounds the Hard Waycited by 169
  20. 2021Mask2Former for Video Instance Segmentationcited by 169
  21. 2021Occluded Video Instance Segmentation: A Benchmarkcited by 154
  22. 2022Audio-Visual Segmentationcited by 148
  23. 2019Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagationcited by 125
  24. 2020STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videoscited by 121
  25. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  26. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  27. 2020SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentationcited by 108
  28. 2021Video Instance Segmentation using Inter-Frame Communication Transformerscited by 105
  29. 2021SeqFormer: Sequential Transformer for Video Instance Segmentationcited by 98
  30. 2022VITA: Video Instance Segmentation via Object Token Associationcited by 94
  31. 2018Self-Supervised Generation of Spatial Audio for 360 Videocited by 93
  32. 2021SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentationcited by 87
  33. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  34. 2021Crossover Learning for Fast Online Video Instance Segmentationcited by 83
  35. 2023DVIS: Decoupled Video Instance Segmentation Frameworkcited by 77
  36. 2023Audio-Visual Segmentation with Semanticscited by 73
  37. 2021Positive Sample Propagation along the Audio-Visual Event Linecited by 69
  38. 2023AVSegFormer: Audio-Visual Segmentation with Transformercited by 66

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 13.

21 papers.

Sort:

By area:

  1. 2024Towards Open-Vocabulary Audio-Visual Event Localizationcited by 26
  2. 2024Dense Audio-Visual Event Localization under Cross-Modal Consistency and Multi-Temporal Granularity Collaborationcited by 12
  3. 2025Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptationcited by 10
  4. 2024Segment Anything for Videos: A Systematic Surveycited by 5
  5. 2025SimToken: A Simple Baseline for Referring Audio-Visual Segmentationcited by 5
  6. 2025Learning What To Hear: Boosting Sound-Source Association For Robust Audiovisual Instance Segmentationcited by 4
  7. 2026Audit After Segmentation: Reference-Free Mask Quality Assessment for Language-Referred Audio-Visual Segmentationcited by 2
  8. 2025TEn-CATG:Text-Enriched Audio-Visual Video Parsing with Multi-Scale Category-Aware Temporal Graphcited by 1
  9. 2025Teacher-Guided Pseudo Supervision and Cross-Modal Alignment for Audio-Visual Video Parsingcited by 1
  10. 2026SeaVIS: Sound-Enhanced Association for Online Audio-Visual Instance Segmentationcited by 1
  11. 2026Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentationcited by 1
  12. 2026Face-Guided Sentiment Boundary Enhancement for Weakly-Supervised Temporal Sentiment Localizationcited by 1
9 more
  1. 20262nd of the 5th PVUW MeViS-Audio Track: ASR-SaSaSa2VAcited by 1
  2. 2026CAE-AV: Improving Audio-Visual Learning via Cross-modal Interactive Enrichmentcited by 0
  3. 2026AV-Unified: A Unified Framework for Audio-visual Scene Understandingcited by 0
  4. 2026RA-SSU: Towards Fine-Grained Audio-Visual Learning with Region-Aware Sound Source Understandingcited by 0
  5. 2026AVTrack: Audio-Visual Tracking in Human-centric Complex Scenescited by 0
  6. 2026AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMscited by 0
  7. 2026Hear to See: Discerning Stateful Listening for Audio-Visual Instance Segmentationcited by 0
  8. 2026Motion-Aware Reasoning from Speech to Mask Tracks: Runner-up Solution for the MeViS-Audio Track of the 8th LSVOS Challenge 2026cited by 0
  9. 2026Binaural Audio-Visual Instance Segmentationcited by 0