mlsanityData to 13 Oct 2026

LPC-SM: Local Predictive Coding and Sparse Memory for Long-Context Language Modeling

arXiv 2604.03263 ↗First posted 12 Mar 2026cs.CL cs.AI cs.NEcited by 0

Not on the map. Why?

Abstract (arXiv)

Most current long-context language models still rely on attention to handle both local interaction and long-range state, which leaves relatively little room to test alternative decompositions of sequence modeling. We propose LPC-SM, a hybrid autoregressive architecture that separates local attention, persistent memory, predictive correction, and run-time control within the same block, and we use Orthogonal Novelty Transport (ONT) to govern slow-memory writes. We evaluate a 158M-parameter model in three stages spanning base language modeling, mathematical continuation, and 4096-token continuation. Removing mHC raises the Stage-A final LM loss from 12.630 to 15.127, while adaptive sparse control improves the Stage-B final LM loss from 12.137 to 10.787 relative to a matched fixed-ratio continuation. The full route remains stable at sequence length 4096, where Stage C ends with final LM loss 11.582 and improves the delayed-identifier diagnostic from 14.396 to 12.031 in key cross-entropy. Taken together, these results show that long-context autoregressive modeling can be organized around a broader division of labor than attention alone.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 27 of its 35 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

27 of 35 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,106
  2. 2020Scaling Laws for Neural Language Modelscited by 6,494
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,658
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,463
  5. 2020Longformer: The Long-Document Transformercited by 3,393
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,835
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,681
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,538
  10. 2019Root Mean Square Layer Normalizationcited by 1,449
  11. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,307
  12. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 820
15 more
  1. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 707
  2. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 603
  3. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 491
  4. 2024Titans: Learning to Memorize at Test Timecited by 312
  5. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 307
  6. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 233
  7. 2019Depth-Adaptive Transformercited by 230
  8. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 204
  9. 2022Recurrent Memory Transformercited by 199
  10. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 193
  11. 2019Adaptive Attention Span in Transformerscited by 148
  12. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 96
  13. 2020Predictive Coding Approximates Backprop along Arbitrary Computation Graphscited by 89
  14. 2013Projection Methods: Swiss Army Knives for Solving Feasibility and Best Approximation Problems with Halfspacescited by 4
  15. 2020Best approximation mappings in Hilbert spacescited by 1

Built on by

Not cited yet.