mlsanityData to 7 Oct 2026

Fresh-CL: Feature Realignment through Experts on Hypersphere in Continual Learning

arXiv 2501.02198 ↗First posted 4 Jan 2025cs.LG cs.CVcited by 0

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Continual Learning enables models to learn and adapt to new tasks while retaining prior knowledge. Introducing new tasks, however, can naturally lead to feature entanglement across tasks, limiting the model's capability to distinguish between new domain data. In this work, we propose a method called Feature Realignment through Experts on hyperSpHere in Continual Learning (Fresh-CL). By leveraging predefined and fixed simplex equiangular tight frame (ETF) classifiers on a hypersphere, our model improves feature separation both intra and inter tasks. However, the projection to a simplex ETF shifts with new tasks, disrupting structured feature representation of previous tasks and degrading performance. Therefore, we propose a dynamic extension of ETF through mixture of experts, enabling adaptive projections onto diverse subspaces to enhance feature representation. Experiments on 11 datasets demonstrate a 2% improvement in accuracy compared to the strongest baseline, particularly in fine-grained datasets, confirming the efficacy of combining ETF and MoE to improve feature distinction in continual learning scenarios.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 26 of its 38 references that are in the map. First posted on arXiv 4 Jan 2025.

Builds on

26 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters18 Mar

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  5. 2013Describing Textures in the Wildcited by 2,110
  6. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  7. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  8. 2021Learning to Prompt for Continual Learningcited by 938
  9. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  10. 2021Robust fine-tuning of zero-shot modelscited by 785
  11. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  12. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
13 more
  1. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  2. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  3. 2022Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learningcited by 118
  4. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 112
  5. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
  6. 2020Generative Feature Replay For Class-Incremental Learningcited by 109
  7. 2020Memory-Efficient Incremental Learning Through Feature Adaptationcited by 107
  8. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  9. 2022Don't Stop Learning: Towards Continual Learning for the CLIP Modelcited by 61
  10. 2023Class-Incremental Learning using Diffusion Model for Distillation and Replaycited by 31
  11. 2024OrCo: Towards Better Generalization via Orthogonality and Contrast for Few-Shot Class-Incremental Learningcited by 25
  12. 2024Learning Equi-angular Representations for Online Continual Learningcited by 15
  13. 2021In Defense of the Learning Without Forgetting for Task Incremental Learningcited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Similarity-Aware Mixture-of-Experts for Data-Efficient Continual Learning
  2. 2026CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
  3. 2026On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models
  4. 2024Divide and not forget: Ensemble of selectively trained experts in Continual Learning
  5. 2026Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolution
  6. 2022The CLEAR Benchmark: Continual LEArning on Real-World Imagery
  7. 2025LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Models
  8. 2025MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging