mlsanityData to 7 Oct 2026

AVSegFormer: Audio-Visual Segmentation with Transformer

arXiv 2307.01146 ↗First posted 3 Jul 2023cs.CV cs.LG cs.SDcited by 66

Not on the map. Why?

Abstract (arXiv)

The combination of audio and vision has long been a topic of interest in the multi-modal community. Recently, a new audio-visual segmentation (AVS) task has been introduced, aiming to locate and segment the sounding objects in a given video. This task demands audio-driven pixel-level scene understanding for the first time, posing significant challenges. In this paper, we propose AVSegFormer, a novel framework for AVS tasks that leverages the transformer architecture. Specifically, we introduce audio queries and learnable queries into the transformer decoder, enabling the network to selectively attend to interested visual features. Besides, we present an audio-visual mixer, which can dynamically adjust visual features by amplifying relevant and suppressing irrelevant spatial channels. Additionally, we devise an intermediate mask loss to enhance the supervision of the decoder, encouraging the network to produce more accurate intermediate predictions. Extensive experiments demonstrate that AVSegFormer achieves state-of-the-art results on the AVS benchmark. The code is available at https://github.com/vvvb-github/AVSegFormer.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Builds on 46 of its 62 references that are in the map. First posted on arXiv 3 Jul 2023.

Builds on

46 of 62 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2017Mask R-CNNcited by 7,259
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  9. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  11. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  12. 2015VQA: Visual Question Answeringcited by 2,489
34 more
  1. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  2. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  3. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  4. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  5. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  6. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  7. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  8. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  9. 2019Panoptic Feature Pyramid Networkscited by 644
  10. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  11. 2019Hybrid Task Cascade for Instance Segmentationcited by 479
  12. 2017Look, Listen and Learncited by 452
  13. 2015Neural Module Networkscited by 375
  14. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 341
  15. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327
  16. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  17. 2017Objects that Soundcited by 254
  18. 2021Associating Objects with Transformers for Video Object Segmentationcited by 225
  19. 2018Learning to Localize Sound Source in Visual Scenescited by 188
  20. 2019UPSNet: A Unified Panoptic Segmentation Networkcited by 175
  21. 2021Localizing Visual Sounds the Hard Waycited by 169
  22. 2022Audio-Visual Segmentationcited by 148
  23. 2020Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsingcited by 146
  24. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  25. 2020Multiple Sound Sources Localization from Coarse to Finecited by 118
  26. 2021Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformerscited by 109
  27. 2021SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentationcited by 106
  28. 2020Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matchingcited by 104
  29. 2019Dual-modality seq2seq network for audio-visual event localizationcited by 72
  30. 2021Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Predictioncited by 57
  31. 2020Making a Case for 3D Convolutions for Object Segmentation in Videoscited by 56
  32. 2019Self-Supervised Audio-Visual Co-Segmentationcited by 44
  33. 2015Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sourcescited by 42
  34. 2022DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Groundingcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 16; 2025: 29; 2026: 15.

66 papers.

Sort:

By area:

  1. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perceptioncited by 39
  2. 2025Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaborationcited by 38
  3. 2023Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizercited by 35
  4. 2023Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learningcited by 32
  5. 2024Ref-AVS: Refer and Segment Objects in Audio-Visual Scenescited by 32
  6. 2023Contrastive Conditional Latent Diffusion for Audio-visual Segmentationcited by 30
  7. 2024Audio-Visual Segmentation via Unlabeled Frame Exploitationcited by 27
  8. 2023Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentationcited by 25
  9. 2025Multimodal Referring Segmentation: A Surveycited by 22
  10. 2023Audio-Visual Instance Segmentationcited by 21
  11. 2024Stepping Stones: A Progressive Training Strategy for Audio-Visual Semantic Segmentationcited by 21
  12. 2023QDFormer: Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decompositioncited by 19
38 more
  1. 2025Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localizationcited by 17
  2. 2024Open-Vocabulary Audio-Visual Semantic Segmentationcited by 16
  3. 2025Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperationcited by 16
  4. 2025SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenescited by 16
  5. 2025AVS-Mamba: Exploring Temporal and Multi-modal Mamba for Audio-Visual Segmentationcited by 15
  6. 2024Can Textual Semantics Mitigate Sounding Object Segmentation Preference?cited by 13
  7. 2024CPM: Class-conditional Prompting Machine for Audio-visual Segmentationcited by 11
  8. 2025Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptationcited by 10
  9. 2025Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentationcited by 10
  10. 2025Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semanticscited by 8
  11. 2024Bootstrapping Audio-Visual Segmentation by Strengthening Audio Cuescited by 7
  12. 2025Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignmentcited by 7
  13. 2024R2-Bench: Benchmarking the Robustness of Referring Perception Models under Perturbationscited by 6
  14. 2024Multi-modal Transfer Learning between Biological Foundation Modelscited by 6
  15. 2024Extending Segment Anything Model into Auditory and Temporal Dimensions for Audio-Visual Segmentationcited by 5
  16. 2024Towards Adversarial Robustness via Debiased High-Confidence Logit Alignmentcited by 5
  17. 2025Do Audio-Visual Segmentation Models Truly Segment Sounding Objects?cited by 5
  18. 2025Revisiting Audio-Visual Segmentation with Vision-Centric Transformercited by 5
  19. 2024SAVE: Segment Audio-Visual Easy way using Segment Anything Modelcited by 4
  20. 2024AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentationcited by 4
  21. 2025Implicit Counterfactual Learning for Audio-Visual Segmentationcited by 4
  22. 2025Learning What To Hear: Boosting Sound-Source Association For Robust Audiovisual Instance Segmentationcited by 4
  23. 2025Complementary and Contrastive Learning for Audio-Visual Segmentationcited by 4
  24. 2025TAViS: Text-bridged Audio-Visual Segmentation with Foundation Modelscited by 3
  25. 2025From Waveforms to Pixels: A Survey on Audio-Visual Segmentationcited by 3
  26. 2025Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentationcited by 3
  27. 2025AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Promptingcited by 2
  28. 2025Agentic Design Review Systemcited by 2
  29. 2025MoLT: Mixture of Layer-Wise Tokens for Efficient Audio-Visual Learningcited by 2
  30. 2026How Do Optical Flow and Textual Prompts Collaborate to Assist in Audio-Visual Semantic Segmentation?cited by 2
  31. 2026Audit After Segmentation: Reference-Free Mask Quality Assessment for Language-Referred Audio-Visual Segmentationcited by 2
  32. 2024TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorizationcited by 1
  33. 2025Audio Visual Segmentation Through Text Embeddingscited by 1
  34. 2025HAVT-IVD: Heterogeneity-Aware Cross-Modal Network for Audio-Visual Surveillance: Idling Vehicles Detection With Multichannel Audio and Multiscale Visual Cuescited by 1
  35. 2025OpenAVS: Training-Free Open-Vocabulary Audio Visual Segmentation with Foundational Modelscited by 1
  36. 2025Progressive Homeostatic and Plastic Prompt Tuning for Audio-Visual Multi-Task Incremental Learningcited by 1
  37. 2025Towards Agentic AI for Multimodal-Guided Video Object Segmentationcited by 1
  38. 2025Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentationcited by 1

50 most cited of 66 All 66 (JSON)