# COBRA: A Continual Learning Approach to Vision-Brain Understanding

arXiv: https://arxiv.org/abs/2411.17475 · HTML: https://mlsanity.com/paper/2411.17475/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 92 references that are in the map.
- First posted on arXiv 25 Nov 2024.

## Builds on (most cited first, 30 of 58)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1904.00310/) (2019)
- [Co2L: Contrastive Continual Learning](https://mlsanity.com/paper/2106.14413/) (2021)
- [Supermasks in Superposition](https://mlsanity.com/paper/2006.14769/) (2020)
- [Versatile Diffusion: Text, Images and Variations All in One Diffusion Model](https://mlsanity.com/paper/2211.08332/) (2022)
- [OneLLM: One Framework to Align All Modalities with Language](https://mlsanity.com/paper/2312.03700/) (2023)
- [PLOP: Learning without Forgetting for Continual Semantic Segmentation](https://mlsanity.com/paper/2011.11390/) (2020)
- [Using Hindsight to Anchor Past Knowledge in Continual Learning](https://mlsanity.com/paper/2002.08165/) (2020)
- [Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors](https://mlsanity.com/paper/2305.18274/) (2023)
- [Continual Unsupervised Representation Learning](https://mlsanity.com/paper/1910.14481/) (2019)

## Built on by (most cited first, 4 of 4)

- [MANGO: Multimodal Attention-based Normalizing Flow Approach to Fusion Learning](https://mlsanity.com/paper/2508.10133/) (2025)
- [EFC++: Elastic Feature Consolidation with Prototype Re-balancing for Cold Start Exemplar-free Incremental Learning](https://mlsanity.com/paper/2503.10439/) (2025)
- [CLIFF: Continual Learning for Incremental Flake Features in 2D Material Identification](https://mlsanity.com/paper/2508.17261/) (2025)
- [BRAIN: Bias-Mitigation Continual Learning Approach to Vision-Brain Understanding](https://mlsanity.com/paper/2508.18187/) (2025)

## Abstract (arXiv)

Vision-Brain Understanding (VBU) aims to extract visual information perceived by humans from brain activity recorded through functional Magnetic Resonance Imaging (fMRI). Despite notable advancements in recent years, existing studies in VBU continue to face the challenge of catastrophic forgetting, where models lose knowledge from prior subjects as they adapt to new ones. Addressing continual learning in this field is, therefore, essential. This paper introduces a novel framework called Continual Learning for Vision-Brain (COBRA) to address continual learning in VBU. Our approach includes three novel modules: a Subject Commonality (SC) module, a Prompt-based Subject Specific (PSS) module, and a transformer-based module for fMRI, denoted as MRIFormer module. The SC module captures shared vision-brain patterns across subjects, preserving this knowledge as the model encounters new subjects, thereby reducing the impact of catastrophic forgetting. On the other hand, the PSS module learns unique vision-brain patterns specific to each subject. Finally, the MRIFormer module contains a transformer encoder and decoder that learns the fMRI features for VBU from common and specific patterns. In a continual learning setup, COBRA is trained in new PSS and MRIFormer modules for new subjects, leaving the modules of previous subjects unaffected. As a result, COBRA effectively addresses catastrophic forgetting and achieves state-of-the-art performance in both continual learning and vision-brain reconstruction tasks, surpassing previous methods.