mlsanityData to 7 Oct 2026

Mitigating Task-Order Sensitivity and Forgetting via Hierarchical Second-Order Consolidation

arXiv 2602.02568 ↗First posted 31 Jan 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We introduce Hierarchical Taylor Series-based Continual Learning (HTCL), a framework that couples fast local adaptation with conservative, second-order global consolidation to address the high variance introduced by random task ordering. To address task-order effects, HTCL identifies the best intra-group task sequence and integrates the resulting local updates through a Hessian-regularized Taylor expansion, yielding a consolidation step with theoretical guarantees. The approach naturally extends to an L-level hierarchy, enabling multiscale knowledge integration in a manner not supported by conventional single-level CL systems. Across a wide range of datasets and replay and regularization baselines, HTCL acts as a model-agnostic consolidation layer that consistently enhances performance, yielding mean accuracy gains of 7% to 25% while reducing the standard deviation of final accuracy by up to 68% across random task permutations.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 28 of its 38 references that are in the map. First posted on arXiv 31 Jan 2026.

Builds on

28 of 38 references.

References · 28

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  4. 2016Learning without Forgettingcited by 2,465
  5. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  6. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  7. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  8. 2017Continual Learning with Deep Generative Replaycited by 1,152
  9. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  10. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  11. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  12. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
16 more
  1. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  2. 2017Variational Continual Learningcited by 335
  3. 2020Continual Prototype Evolution: Learning Online from Non-Stationary Data Streamscited by 143
  4. 2019Scalable and Order-robust Continual Learning with Additive Parameter Decompositioncited by 125
  5. 2019Overcoming Forgetting in Federated Learning on Non-IID Datacited by 120
  6. 2018Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilizationcited by 104
  7. 2018Generative replay with feedback connections as a general strategy for continual learningcited by 103
  8. 2020Optimal Continual Learning has Perfect Memory and is NP-hardcited by 86
  9. 2019Toward Understanding Catastrophic Forgetting in Continual Learningcited by 67
  10. 2018Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologiescited by 43
  11. 2024Learning Continually by Spectral Regularizationcited by 34
  12. 2022The Effect of Task Ordering in Continual Learningcited by 30
  13. 2020Dropout as an Implicit Gating Mechanism For Continual Learningcited by 24
  14. 2021Formalizing the Generalization-Forgetting Trade-off in Continual Learningcited by 24
  15. 2023Continual Learning with Strong Experience Replaycited by 7
  16. 2024RECAST: Reparameterized, Compact weight Adaptation for Sequential Taskscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Ready2Blend: From Natural-Language Instructions to Composable Alignment Promptscited by 0