mlsanityData to 7 Oct 2026

Dark Experience for General Continual Learning: a Strong, Simple Baseline

arXiv 2004.07211 ↗First posted 15 Apr 2020stat.ML cs.LGcited by 895

Not on the map. Why?

Cited by 38 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Continual Learning has inspired a plethora of approaches and evaluation settings; however, the majority of them overlooks the properties of a practical scenario, where the data stream cannot be shaped as a sequence of tasks and offline training is not viable. We work towards General Continual Learning (GCL), where task boundaries blur and the domain and class distributions shift either gradually or suddenly. We address it through mixing rehearsal with knowledge distillation and regularization; our simple baseline, Dark Experience Replay, matches the network's logits sampled throughout the optimization trajectory, thus promoting consistency with its past. By conducting an extensive analysis on both standard benchmarks and a novel GCL evaluation setting (MNIST-360), we show that such a seemingly simple baseline outperforms consolidated approaches and leverages limited resources. We further explore the generalization capabilities of our objective, showing its regularization being beneficial beyond mere performance.

Key facts. Cited by 895 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 41; 2022: 88; 2023: 191; 2024: 220; 2025: 185; 2026: 170.

895 papers.

Sort:

By area:

  1. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  2. 2021Learning to Prompt for Continual Learningcited by 938
  3. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  4. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  5. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  6. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  7. 2022Memory-Based Model Editing at Scalecited by 443
  8. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  9. 2021Co2L: Contrastive Continual Learningcited by 243
  10. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 242
  11. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  12. 2021New Insights on Reducing Abrupt Representation Change in Online Continual Learningcited by 215
38 more
  1. 2023Class-Incremental Learning: A Surveycited by 209
  2. 2023Continual Pre-training of Language Modelscited by 194
  3. 2023Mitigating the Alignment Tax of RLHFcited by 181
  4. 2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 178
  5. 2023RanPAC: Random Projections and Pre-trained Models for Continual Learningcited by 162
  6. 2021Self-Supervised Models are Continual Learnerscited by 149
  7. 2022Continual Learning and Private Unlearningcited by 124
  8. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
  9. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 121
  10. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  11. 2022Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning Systemcited by 110
  12. 2021Representational Continuity for Unsupervised Continual Learningcited by 95
  13. 2021GCR: Gradient Coreset Based Replay Buffer Selection For Continual Learningcited by 95
  14. 2021Achieving Forgetting Prevention and Knowledge Transfer in Continual Learningcited by 91
  15. 2022Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Timecited by 85
  16. 2023Learning without Forgetting for Vision-Language Modelscited by 79
  17. 2022A Theoretical Study on Solving Continual Learningcited by 78
  18. 2024Continual Learning and Catastrophic Forgettingcited by 73
  19. 2022Representation Compensation Networks for Continual Semantic Segmentationcited by 72
  20. 2023OneNet: Enhancing Time Series Forecasting Models under Concept Drift by Online Ensemblingcited by 71
  21. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  22. 2022CLIP model is an Efficient Continual Learnercited by 64
  23. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  24. 2023Computationally Budgeted Continual Learning: What Does Matter?cited by 61
  25. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Modelcited by 60
  26. 2024Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuningcited by 59
  27. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  28. 2022SparCL: Sparse Continual Learning on the Edgecited by 55
  29. 2022A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learningcited by 54
  30. 2023Better Generative Replay for Continual Federated Learningcited by 53
  31. 2023Online Prototype Learning for Online Continual Learningcited by 53
  32. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
  33. 2024Consistent Prompting for Rehearsal-Free Continual Learningcited by 52
  34. 2023PCR: Proxy-based Contrastive Replay for Online Class-Incremental Continual Learningcited by 50
  35. 2023Introducing Language Guidance in Prompt-based Continual Learningcited by 49
  36. 2023Continual Instruction Tuning for Large Multimodal Modelscited by 49
  37. 2023Real-Time Evaluation in Online Continual Learning: A New Hopecited by 47
  38. 2023AttriCLIP: A Non-Incremental Learner for Incremental Knowledge Learningcited by 46

50 most cited of 895 All 895 (JSON)