mlsanityData to 7 Oct 2026

Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learning

arXiv 2510.16877 ↗First posted 19 Oct 2025cs.LG cs.AI cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Using a nearly-frozen pretrained model, the continual representation learning paradigm reframes parameter updates as a similarity-matching problem to mitigate catastrophic forgetting. However, directly leveraging pretrained features for downstream tasks often suffers from multicollinearity in the similarity-matching stage, and more advanced methods can be computationally prohibitive for real-time, low-latency applications. Inspired by the fly olfactory circuit, we propose Fly-CL, a bio-inspired framework compatible with a wide range of pretrained backbones. Fly-CL substantially reduces training time while achieving performance comparable to or exceeding that of current state-of-the-art methods. We theoretically show how Fly-CL progressively resolves multicollinearity, enabling more effective similarity matching with low time complexity. Extensive simulation experiments across diverse network architectures and data regimes validate Fly-CL's effectiveness in addressing this challenge through a biologically inspired design. Code is available at https://github.com/gfyddha/Fly-CL.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 50 of its 78 references that are in the map. First posted on arXiv 19 Oct 2025.

Builds on

50 of 78 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2016Learning without Forgettingcited by 2,465
  6. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  7. 2022Visual Prompt Tuningcited by 1,674
  8. 2019Natural Adversarial Examplescited by 1,401
  9. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  10. 2021Learning to Prompt for Continual Learningcited by 938
  11. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  12. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
38 more
  1. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  2. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  3. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  4. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  5. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 189
  6. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
  7. 2023RanPAC: Random Projections and Pre-trained Models for Continual Learningcited by 162
  8. 2023A Unified Continual Learning Framework with General Parameter-Efficient Tuningcited by 124
  9. 2023Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimalitycited by 121
  10. 2024Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learningcited by 115
  11. 2023Learning without Forgetting for Vision-Language Modelscited by 79
  12. 2023PILOT: A Pre-Trained Model-Based Continual Learning Toolboxcited by 64
  13. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
  14. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 48
  15. 2022Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interferencecited by 44
  16. 2023When Prompt-based Incremental Learning Does Not Meet Strong Pretrainingcited by 41
  17. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  18. 2024MOS: Model Surgery for Pre-Trained Model-Based Class-Incremental Learningcited by 35
  19. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
  20. 2025DynaPrompt: Dynamic Test-Time Prompt Tuningcited by 31
  21. 2024LLM-Empowered State Representation for Reinforcement Learningcited by 28
  22. 2024Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learningcited by 22
  23. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  24. 2023Supported Trust Region Optimization for Offline Reinforcement Learningcited by 20
  25. 2025Model Predictive Task Sampling for Efficient and Robust Adaptationcited by 20
  26. 2021Can a Fruit Fly Learn Word Embeddings?cited by 17
  27. 2023PromptFusion: Decoupling Stability and Plasticity for Continual Learningcited by 17
  28. 2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppressioncited by 17
  29. 2025Structural features of the fly olfactory circuit mitigate the stability-plasticity dilemma in continual learningcited by 14
  30. 2025LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuningcited by 13
  31. 2024F-OAL: Forward-only Online Analytic Learning with Fast Training and Low Memory Footprint in Class Incremental Learningcited by 12
  32. 2020Bio-Inspired Hashing for Unsupervised Similarity Searchcited by 11
  33. 2025Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environmentscited by 11
  34. 2024Random Representations Outperform Online Continually Learned Representationscited by 10
  35. 2024Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarkscited by 10
  36. 2024Stop Wandering, Find the Keys: LLMs Discriminate Key States for Efficient Multi-Agent Explorationcited by 9
  37. 2021Federated Nearest Neighbor Classification with a Colony of Fruit-Flies: With Supplementcited by 3
  38. 2018Improving Similarity Search with High-dimensional Locality-sensitive Hashingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learningcited by 9
  2. 2026Enhancing Pretrained Model-based Continual Representation Learning via Guided Random Projectioncited by 4
  3. 2026Limited Reasoning Space: The cage of long-horizon reasoning in LLMscited by 3
  4. 2026FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Expertscited by 3
  5. 2026Representation Collapse in Sequential Post-Training of Large Language Modelscited by 2
  6. 2026The Long-Term Effects of Data Selection in LLM Fine-Tuningcited by 1
  7. 2026Online Data Selection Is Implicit Alignmentcited by 1
  8. 2026CDRL: A Reinforcement Learning Framework Inspired by Cerebellar Circuits and Dendritic Computational Strategiescited by 0
  9. 2026SPARCL: Spectral Partitioned Analytic Continual Learningcited by 0
  10. 2026Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Expertscited by 0