mlsanityData to 14 Oct 2026

Sample-Efficient Language Modeling with Linear Attention and Lightweight Enhancements

arXiv 2511.05560 ↗First posted 4 Nov 2025cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

We study architectural and optimization techniques for sample-efficient language modeling under the constraints of the BabyLM 2025 shared task. Our model, BLaLM, replaces self-attention with a linear-time mLSTM token mixer and explores lightweight enhancements, including short convolutions, sliding window attention with dynamic modulation, and Hedgehog feature maps. To support training in low-resource settings, we curate a high-quality corpus emphasizing readability and pedagogical structure. Experiments across both STRICT and STRICT-SMALL tracks show that (1) linear attention combined with sliding window attention consistently improves zero-shot performance, and (2) the Muon optimizer stabilizes convergence and reduces perplexity over AdamW. These results highlight effective strategies for efficient language modeling without relying on scale.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 19 of its 27 references that are in the map. First posted on arXiv 4 Nov 2025.

Builds on

19 of 27 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2020Longformer: The Long-Document Transformercited by 3,413
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  6. 2019Root Mean Square Layer Normalizationcited by 1,480
  7. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  8. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  9. 2025Muon is Scalable for LLM Trainingcited by 428
  10. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  11. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  12. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
7 more
  1. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
  2. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 120
  3. 2025Practical Efficiency of Muon for Pretrainingcited by 80
  4. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Modelscited by 67
  5. 2018A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguisticscited by 64
  6. 2025Lizard: An Efficient Linearization Framework for Large Language Modelscited by 6
  7. 2025What Matters in Linearizing Language Models? A Comparative Study of Architecture, Scale, and Task Adaptationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026TriPLU: Bypassing the Gate with Direct Trilinear Product FFNs in Tiny Language Modelscited by 0