mlsanityData to 14 Oct 2026

Continual Cross-Modal Generalization

arXiv 2504.00561 ↗First posted 1 Apr 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Cross-modal generalization aims to learn a shared discrete representation space from multimodal pairs, enabling knowledge transfer across unannotated modalities. However, achieving a unified representation for all modality pairs requires extensive paired data, which is often impractical. Inspired by the availability of abundant bimodal data (e.g., in ImageBind), we explore a continual learning approach that incrementally maps new modalities into a shared discrete codebook via a mediator modality. We propose the Continual Mixture of Experts Adapter (CMoE-Adapter) to project diverse modalities into a unified space while preserving prior knowledge. To align semantics across stages, we introduce a Pseudo-Modality Replay (PMR) mechanism with a dynamically expanding codebook, enabling the model to adaptively incorporate new modalities using learned ones as guidance. Extensive experiments on image-text, audio-text, video-text, and speech-text show that our method achieves strong performance on various cross-modal generalization tasks. Code is provided in the supplementary material.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 53 references that are in the map. First posted on arXiv 1 Apr 2025.

Builds on

38 of 53 references.

References · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2016Learning without Forgettingcited by 2,496
  5. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  6. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  7. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  8. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  9. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  10. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  11. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  12. 2019Clotho: An Audio Captioning Datasetcited by 358
26 more
  1. 2018Selective Experience Replay for Lifelong Learningcited by 241
  2. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  3. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  4. 2020Modeling the Background for Incremental Learning in Semantic Segmentationcited by 190
  5. 2018Learning to Localize Sound Source in Visual Scenescited by 189
  6. 2022Language as Queries for Referring Video Object Segmentationcited by 182
  7. 2020Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsingcited by 146
  8. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  9. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
  10. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 66
  11. 2018Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecturecited by 64
  12. 2022Multi-modal Alignment using Representation Codebookcited by 63
  13. 2022Robust Cross-Modal Representation Learning with Progressive Self-Distillationcited by 55
  14. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  15. 2020Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interferencecited by 47
  16. 2022VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMixcited by 45
  17. 2022Task Adaptive Parameter Sharing for Multi-Task Learningcited by 42
  18. 2018Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Datacited by 35
  19. 2021Cross-Modal Discrete Representation Learningcited by 35
  20. 2022Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-trainingcited by 32
  21. 2022XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learningcited by 31
  22. 2018Continual Classification Learning Using Generative Modelscited by 28
  23. 2025Towards Transformer-Based Aligned Generation with Self-Coherence Guidancecited by 14
  24. 2022Multi-Head Distillation for Continual Unsupervised Domain Adaptation in Semantic Segmentationcited by 12
  25. 2024Enhancing Multimodal Unified Representations for Cross Modal Generalizationcited by 12
  26. 2024Semantic Residual for Multimodal Unified Discrete Representationcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations
  2. 2025Multi-Modal Continual Learning via Cross-Modality Adapters and Representation Alignment with Knowledge Preservation
  3. 2024Reconstruct before Query: Continual Missing Modality Learning with Decomposed Prompt Collaboration
  4. 2025Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models
  5. 2026Understanding Cross-Modal Contributions in Continual Vision-Language Models: A Theoretical Perspective
  6. 2022Generalizing Multimodal Variational Methods to Sets
  7. 2026DeLo: Dual Decomposed Low-Rank Experts Collaboration for Continual Missing Modality Learning
  8. 2026Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs