mlsanityData to 14 Oct 2026

Working Memory Constraints Scaffold Learning in Transformers under Data Scarcity

arXiv 2604.20789 ↗First posted 22 Apr 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We investigate the integration of human-like working memory constraints into the Transformer architecture and implement several cognitively inspired attention variants, including fixed-width windows based and temporal decay based attention mechanisms. Our modified GPT-2 models are trained from scratch on developmentally plausible datasets (10M and 100M words). Performance is evaluated on grammatical judgment tasks (BLiMP) and alignment with human reading time data. Our results indicate that these cognitively-inspired constraints, particularly fixed-width attention, can significantly improve grammatical accuracy especially when training data is scarce. These constrained models also tend to show a stronger alignment with human processing metrics. The findings suggest that such constraints may serve as a beneficial inductive bias, guiding models towards more robust linguistic representations, especially in data-limited settings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 50 references that are in the map. First posted on arXiv 22 Apr 2026.

Builds on

19 of 50 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  5. 2020Rethinking Attention with Performerscited by 1,601
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  7. 2018Self-Attention with Relative Position Representationscited by 1,120
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  9. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  10. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  11. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  12. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 113
7 more
  1. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 88
  2. 2022Context Limitations Make Neural Language Models More Human-Likecited by 34
  3. 2021Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attentioncited by 27
  4. 2023A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processingcited by 22
  5. 2024Linear Recency Bias During Training Improves Transformers' Fit to Reading Timescited by 14
  6. 2020Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Modelscited by 8
  7. 2023Are words equally surprising in audio and audio-visual comprehension?cited by 1

Built on by

Not cited yet.