mlsanityData to 14 Oct 2026

Structural Abstraction as an Inductive Bias for Non-Stationary Language Model Training

arXiv 2603.17198 ↗First posted 17 Mar 2026cs.LG cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

A foundational principle in cognitive science holds that intelligent agents do not learn by storing experiences as isolated instances, but by forming abstract schemas that capture relational structure shared across situations. Even though this claim is well supported by behavioral and neuroimaging studies, its role as a computational training signal in language models remains underexplored. We target this gap in the setting of non-stationary language model training, asking does biasing learning toward structural abstraction reduce catastrophic interference and improve relational generalization as predicted by human results? To study this question, we introduce Abstraction-Augmented Training (AAT), a lightweight loss-level modification that jointly optimizes over concrete instances and their structural abstractions, and two benchmarks, the Relational Cycle Benchmark (RCB) and the Narrative Abstraction Benchmark (NAB). These resources operationalize core cognitive constructs: entity masking as a computational analog of relational alignment, and proverbs as vehicles for implicit abstract meaning that must be inferred across surface-dissimilar situations. Our empirical results demonstrate that AAT consistently reduces forgetting and improves generalization in a pattern that aligns with cognitive predictions for schema-based learning. Beyond the practical implications for continual learning, these results offer preliminary computational evidence that structural abstraction is a signal for stable learning in non-stationary environments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 57 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

33 of 57 references.

References · 33

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  3. 2018Experience Replay for Continual Learningcited by 1,095
  4. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  5. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  6. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  7. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  8. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  9. 2023Continual Pre-training of Language Modelscited by 198
  10. 2021Towards Continual Knowledge Learning of Language Modelscited by 174
  11. 2023Understanding plasticity in neural networkscited by 174
  12. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
21 more
  1. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
  2. 2019Toward Understanding Catastrophic Forgetting in Continual Learningcited by 68
  3. 2023TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Modelscited by 66
  4. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  5. 2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61
  6. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Modelcited by 61
  7. 2021Curriculum-Meta Learning for Order-Robust Continual Relation Extractioncited by 34
  8. 2023CITB: A Benchmark for Continual Instruction Tuningcited by 33
  9. 2022Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learningcited by 29
  10. 2023Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learningcited by 27
  11. 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 23
  12. 2023Exploring Continual Learning for Code Generation Modelscited by 22
  13. 2024Adaptive Memory Replay for Continual Learningcited by 22
  14. 2025TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretrainingcited by 15
  15. 2021ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understandingcited by 14
  16. 2024Plastic Learning with Deep Fourier Featurescited by 14
  17. 2024Self-Normalized Resets for Plasticity in Continual Learningcited by 11
  18. 2022Cross-lingual Lifelong Learningcited by 9
  19. 2023Continual Learning with Strong Experience Replaycited by 7
  20. 2023Online Continual Knowledge Learning for Language Modelscited by 6
  21. 2025Experience Replay Addresses Loss of Plasticity in Continual Learningcited by 4

Built on by

Not cited yet.