mlsanityData to 7 Oct 2026

CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks

arXiv 2206.09059 ↗First posted 18 Jun 2022cs.CL cs.AI cs.CVcited by 68

Not on the map. Why?

Abstract (arXiv)

Current state-of-the-art vision-and-language models are evaluated on tasks either individually or in a multi-task setting, overlooking the challenges of continually learning (CL) tasks as they arrive. Existing CL benchmarks have facilitated research on task adaptation and mitigating "catastrophic forgetting", but are limited to vision-only and language-only tasks. We present CLiMB, a benchmark to study the challenge of learning multimodal tasks in a CL setting, and to systematically evaluate how upstream continual learning can rapidly generalize to new multimodal and unimodal tasks. CLiMB includes implementations of several CL algorithms and a modified Vision-Language Transformer (ViLT) model that can be deployed on both multimodal and unimodal tasks. We find that common CL methods can help mitigate forgetting during multimodal task learning, but do not enable cross-task knowledge transfer. We envision that CLiMB will facilitate research on a new class of CL algorithms for this challenging multimodal setting.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 42 of its 56 references that are in the map. First posted on arXiv 18 Jun 2022.

Builds on

42 of 56 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
30 more
  1. 2016Matching Networks for One Shot Learningcited by 2,246
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  5. 2014How transferable are features in deep neural networks?cited by 1,216
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  7. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  8. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  9. 2022A Generalist Agentcited by 802
  10. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  11. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  12. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  13. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  14. 2017Learning multiple visual domains with residual adapterscited by 553
  15. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  16. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  17. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  18. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  19. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  20. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  21. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  22. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  23. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  24. 2022The CLEAR Benchmark: Continual LEArning on Real-World Imagerycited by 92
  25. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 87
  26. 2022The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoningcited by 38
  27. 2021Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learningcited by 35
  28. 2019Psycholinguistics meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answeringcited by 33
  29. 2022Continual Sequence Generation with Adaptive Compositional Modulescited by 31
  30. 2022Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-modal Knowledge Transfercited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 19; 2024: 25; 2025: 11; 2026: 8.

68 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2023Class-Incremental Learning: A Surveycited by 209
  3. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 138
  4. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  5. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
  6. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 59
  7. 2023Continual Instruction Tuning for Large Multimodal Modelscited by 49
  8. 2023Continual Vision-Language Representation Learning with Off-Diagonal Informationcited by 40
  9. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  10. 2023TiC-CLIP: Continual Training of CLIP Modelscited by 38
  11. 2023Audio-Visual Class-Incremental Learningcited by 33
  12. 2024Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfercited by 31
38 more
  1. 2024CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Modelscited by 25
  2. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  3. 2022ConStruct-VL: Data-Free Continual Structured VL Concepts Learningcited by 20
  4. 2024DAM: Dynamic Adapter Merging for Continual Video QA Learningcited by 18
  5. 2023CL-MASR: A Continual Learning Benchmark for Multilingual ASRcited by 16
  6. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  7. 2023Revisiting Distillation for Continual Learning on Visual Question Localized-Answering in Robotic Surgerycited by 15
  8. 2024LLMs Can Evolve Continually on Modality for X-Modal Reasoningcited by 12
  9. 2025Continual Text-to-Video Retrieval with Frame Fusion and Task-Aware Routingcited by 12
  10. 2023Task-Attentive Transformer Architecture for Continual Learning of Vision-and-Language Tasks Using Knowledge Distillationcited by 11
  11. 2023Continual Diffusion with STAMINA: STack-And-Mask INcremental Adapterscited by 10
  12. 2024Continual Learning for Smart City: A Surveycited by 9
  13. 2024ViLCo-Bench: VIdeo Language COntinual learning Benchmarkcited by 9
  14. 2024Adapt-∞: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selectioncited by 9
  15. 2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyondcited by 9
  16. 2024Dynamic Transformer Architecture for Continual Learning of Multimodal Taskscited by 8
  17. 2023I2I: Initializing Adapters with Improvised Knowledgecited by 7
  18. 2023POP: Prompt Of Prompts for Continual Learningcited by 7
  19. 2024Enhancing Continual Learning in Visual Question Answering with Modality-Aware Feature Distillationcited by 7
  20. 2024ATLAS: Adapter-Based Multi-Modal Continual Learning with a Two-Stage Learning Strategycited by 7
  21. 2024Continual Audio-Visual Sound Separationcited by 7
  22. 2025Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answeringcited by 7
  23. 2022CL-CrossVQA: A Continual Learning Benchmark for Cross-Domain Visual Question Answeringcited by 6
  24. 2023From MNIST to ImageNet and Back: Benchmarking Continual Curriculum Learningcited by 6
  25. 2024Vision-Language Navigation with Continual Learningcited by 6
  26. 2024One VLM to Keep it Learning: Generation and Balancing for Data-free Continual Visual Question Answeringcited by 6
  27. 2024Cross-domain Multi-modal Few-shot Object Detection via Rich Textcited by 5
  28. 2024Beyond Unimodal Learning: The Importance of Integrating Multiple Modalities for Lifelong Learningcited by 5
  29. 2022Task Formulation Matters When Learning Continually: A Case Study in Visual Question Answeringcited by 4
  30. 2023Visually Grounded Continual Language Learning with Selective Specializationcited by 4
  31. 2024CluMo: Cluster-based Modality Fusion Prompt for Continual Learning in Visual Question Answeringcited by 4
  32. 2025Task-Core Memory Management and Consolidation for Long-term Continual Learningcited by 4
  33. 2026Elastic Weight Consolidation Done Right for Continual Learningcited by 3
  34. 2023GrowCLIP: Data-aware Automatic Model Growing for Large-scale Contrastive Language-Image Pre-trainingcited by 2
  35. 2025Parabolic Continual Learningcited by 2
  36. 2025A Comprehensive Survey on Video Scene Parsing:Advances, Challenges, and Prospectscited by 2
  37. 2024Continuous Unsupervised Domain Adaptation Using Stabilized Representations and Experience Replaycited by 1
  38. 2024Rethinking Teacher-Student Curriculum Learning through the Cooperative Mechanics of Experiencecited by 1

50 most cited of 68 All 68 (JSON)