mlsanityData to 14 Oct 2026

Audio-Visual Class-Incremental Learning

arXiv 2308.11073 ↗First posted 21 Aug 2023cs.CVcited by 33

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce audio-visual class-incremental learning, a class-incremental learning scenario for audio-visual video recognition. We demonstrate that joint audio-visual modeling can improve class-incremental learning, but current methods fail to preserve semantic similarity between audio and visual features as incremental step grows. Furthermore, we observe that audio-visual correlations learned in previous tasks can be forgotten as incremental steps progress, leading to poor performance. To overcome these challenges, we propose AV-CIL, which incorporates Dual-Audio-Visual Similarity Constraint (D-AVSC) to maintain both instance-aware and class-aware semantic similarity between audio-visual modalities and Visual Attention Distillation (VAD) to retain previously learned audio-guided visual attentive ability. We create three audio-visual class-incremental datasets, AVE-Class-Incremental (AVE-CI), Kinetics-Sounds-Class-Incremental (K-S-CI), and VGGSound100-Class-Incremental (VS100-CI) based on the AVE, Kinetics-Sounds, and VGGSound datasets, respectively. Our experiments on AVE-CI, K-S-CI, and VS100-CI demonstrate that AV-CIL significantly outperforms existing class-incremental learning methods in audio-visual class-incremental learning. Code and data are available at: https://github.com/weiguoPian/AV-CIL_ICCV2023.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 79 of its 102 references that are in the map. First posted on arXiv 21 Aug 2023.

Builds on

79 of 102 references.

References · 79

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2016Learning without Forgettingcited by 2,496
  8. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  9. 2017The Kinetics Human Action Video Datasetcited by 1,909
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  11. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  12. 2017Continual Learning with Deep Generative Replaycited by 1,158
38 more
  1. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  2. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  3. 2019Large Scale Incremental Learningcited by 732
  4. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  5. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  6. 2018End-to-End Incremental Learningcited by 597
  7. 2016Conditional Image Synthesis With Auxiliary Classifier GANscited by 565
  8. 2017Look, Listen and Learncited by 458
  9. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  10. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  11. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 283
  12. 2022Masked Autoencoders that Listencited by 273
  13. 2018Learning without Memorizingcited by 250
  14. 2022FOSTER: Feature Boosting and Compression for Class-Incremental Learningcited by 229
  15. 2019Compacting, Picking and Growing for Unforgetting Continual Learningcited by 207
  16. 2018Learning to Localize Sound Source in Visual Scenescited by 189
  17. 2019Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learningcited by 182
  18. 2020PLOP: Learning without Forgetting for Continual Semantic Segmentationcited by 177
  19. 2021Towards Continual Knowledge Learning of Language Modelscited by 174
  20. 2021Localizing Visual Sounds the Hard Waycited by 170
  21. 2022Fine-tuned Language Models are Continual Learnerscited by 156
  22. 2021Self-Supervised Models are Continual Learnerscited by 149
  23. 2019Uncertainty-based Continual Learning with Adaptive Regularizationcited by 146
  24. 2020Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsingcited by 146
  25. 2022Contrastive Audio-Visual Masked Autoencodercited by 146
  26. 2020SS-IL: Separated Softmax for Incremental Learningcited by 126
  27. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  28. 2022Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidationcited by 113
  29. 2020Music Gesture for Visual Sound Separationcited by 112
  30. 2019Overcoming Catastrophic Forgetting with Unlabeled Data in the Wildcited by 111
  31. 2020Continual Learning with Node-Importance based Adaptive Group Sparse Regularizationcited by 105
  32. 2019Continual Learning via Neural Pruningcited by 101
  33. 2018Measuring and regularizing networks in function spacecited by 99
  34. 2021Representational Continuity for Unsupervised Continual Learningcited by 95
  35. 2021RECALL: Replay-based Continual Learning in Semantic Segmentationcited by 83
  36. 2020Semantic Audio-Visual Navigationcited by 82
  37. 2022Localizing Visual Sounds the Easy Waycited by 82
  38. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 73

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 14; 2025: 6; 2026: 8.

33 papers.

Sort:

By area:

  1. 2024Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learningcited by 117
  2. 2024Text-to-Audio Generation Synchronized with Videoscited by 28
  3. 2023Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modelingcited by 24
  4. 2023Class-Incremental Grouping Network for Continual Audio-Visual Learningcited by 22
  5. 2023Weakly-Supervised Audio-Visual Segmentationcited by 19
  6. 2024Audio-visual Generalized Zero-shot Learning the Easy Waycited by 7
  7. 2024Continual Audio-Visual Sound Separationcited by 7
  8. 2025DiffGAP: A Lightweight Diffusion Module in Contrastive Space for Bridging Cross-Model Gapcited by 7
  9. 2024Modality-Inconsistent Continual Learning of Multimodal Large Language Modelscited by 6
  10. 2024Answering Diverse Questions via Text Attached with Key Audio-Visual Cluescited by 5
  11. 2024Semantic Grouping Network for Audio Source Separationcited by 5
  12. 2024Learning Trimodal Relation for Audio-Visual Question Answering with Missing Modalitycited by 4
21 more
  1. 2024Aligning Audio-Visual Joint Representations with an Agentic Workflowcited by 4
  2. 2025PreFM: Online Audio-Visual Event Parsing via Predictive Future Modelingcited by 4
  3. 2024Characterizing Continual Learning Scenarios and Strategies for Audio Analysiscited by 3
  4. 2024Multi-scale Multi-instance Visual Sound Localization and Segmentationcited by 3
  5. 2024Unified Video-Language Pre-training with Synchronized Audiocited by 2
  6. 2026Few-Shot Open-Set Audio Classification Using Attention Information-Fused Prototypescited by 2
  7. 2023STELLA: Continual Audio-Video Pre-training with Spatio-Temporal Localized Alignmentcited by 1
  8. 2024Distribution-Level Memory Recall for Continual Learning: Preserving Knowledge and Avoiding Confusioncited by 1
  9. 2025Multi-Modal Continual Learning via Cross-Modality Adapters and Representation Alignment with Knowledge Preservationcited by 1
  10. 2026Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flowscited by 1
  11. 2023Continual Contrastive Spoken Language Understandingcited by 0
  12. 2024ReFu: Recursive Fusion for Exemplar-Free 3D Class-Incremental Learningcited by 0
  13. 2025TAIL: Text-Audio Incremental Learningcited by 0
  14. 2025Audio-Visual Class-Incremental Learning for Fish Feeding intensity Assessment in Aquaculturecited by 0
  15. 2025MCIGLE: Multimodal Exemplar-Free Class-Incremental Graph Learningcited by 0
  16. 2026Pruned Adaptation Modules: A Simple yet Strong Baseline for Continual Foundation Modelscited by 0
  17. 2026FeDMRA: Federated Incremental Learning with Dynamic Memory Replay Allocationcited by 0
  18. 2026Listen, Look, and Learn: Learning Without Forgetting through SAM-Audiocited by 0
  19. 2026Constrained Hebbian Learning Supports Efficient Representational Allocation under Structural Constraintscited by 0
  20. 2026Regularizing modality contribution drift in multimodal continual learningcited by 0
  21. 2026CE4L: Continual Ego, Exo, and Ego-Exo Learningcited by 0