mlsanityData to 7 Oct 2026

A Survey of Historical Learning: Learning Models with Learning History

arXiv 2303.12992 ↗First posted 23 Mar 2023cs.LG cs.AI cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

New knowledge originates from the old. The various types of elements, deposited in the training history, are a large amount of wealth for improving learning deep models. In this survey, we comprehensively review and summarize the topic--``Historical Learning: Learning Models with Learning History'', which learns better neural models with the help of their learning history during its optimization, from three detailed aspects: Historical Type (what), Functional Part (where) and Storage Form (how). To our best knowledge, it is the first survey that systematically studies the methodologies which make use of various historical statistics when training deep neural networks. The discussions with related topics like recurrent/memory networks, ensemble learning, and reinforcement learning are demonstrated. We also expose future challenges of this topic and encourage the community to pay attention to the think of historical learning principles when designing algorithms. The paper list related to historical learning is available at urlhttps://github.com/Martinser/Awesome-Historical-Learning.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 181 of its 236 references that are in the map. First posted on arXiv 23 Mar 2023.

Builds on

181 of 236 references.

References · 181

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2017Mask R-CNNcited by 7,259
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
38 more
  1. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  3. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  6. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  8. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  9. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  11. 2019Contrastive Multiview Codingcited by 1,297
  12. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  13. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  14. 2017Hindsight Experience Replaycited by 1,096
  15. 2015Prioritized Experience Replaycited by 1,070
  16. 2021YOLOX: Exceeding YOLO Series in 2021cited by 1,064
  17. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  18. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  19. 2021Robust fine-tuning of zero-shot modelscited by 785
  20. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  21. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  22. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 728
  23. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  24. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  25. 2014Neural Turing Machinescited by 630
  26. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  27. 2016Fully-Convolutional Siamese Networks for Object Trackingcited by 566
  28. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  29. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  30. 2018Born Again Neural Networkscited by 539
  31. 2021Semi-Supervised Semantic Segmentation with Cross Pseudo Supervisioncited by 462
  32. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  33. 2021Conditional DETR for Fast Training Convergencecited by 422
  34. 2020Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detectioncited by 408
  35. 2019Uncertainty-aware Self-ensembling Model for Semi-supervised 3D Left Atrium Segmentationcited by 378
  36. 2015Learning Deconvolution Network for Semantic Segmentationcited by 365
  37. 2014Memory Networkscited by 354
  38. 2017Snapshot Ensembles: Train 1, get M for freecited by 349

50 most cited of 181 All 181 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 0; 2025: 0; 2026: 1.

3 papers.

  1. 2023FedSSA: Semantic Similarity-based Aggregation for Efficient Model-Heterogeneous Personalized Federated Learningcited by 10
  2. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMscited by 5
  3. 2023Improve Deep Forest with Learnable Layerwise Augmentation Policy Schedulecited by 3