mlsanityData to 7 Oct 2026

Locas: Your Models are Principled Initializers of Locally-Supported Parametric Memories

arXiv 2602.05085 ↗First posted 4 Feb 2026cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

In this paper, we aim to bridge test-time-training with a new type of parametric memory that can be flexibly offloaded from or merged into model parameters. We present Locas, a Locally-Supported parametric memory that shares the design of FFN blocks in modern transformers, allowing it to be flexibly permanentized into the model parameters while supporting efficient continual learning. We discuss two major variants of Locas: one with a conventional two-layer MLP design that has a clearer theoretical guarantee; the other one shares the same GLU-FFN structure with SOTA LLMs, and can be easily attached to existing models for both parameter-efficient and computation-efficient continual learning. Crucially, we show that proper initialization of such low-rank sideway-FFN-style memories -- performed in a principled way by reusing model parameters, activations and/or gradients -- is essential for fast convergence, improved generalization, and catastrophic forgetting prevention. We validate the proposed memory mechanism on the PG-19 whole-book language modeling and LoCoMo long-context dialogue question answering tasks. With only 0.02% additional parameters in the lowest case, Locas-GLU is capable of storing the information from past context while maintaining a much smaller context window. In addition, we also test the model's general capability loss after memorizing the whole book with Locas, through comparative MMLU evaluation. Results show the promising ability of Locas to permanentize past context into parametric knowledge with minimized catastrophic forgetting of the model's existing internal knowledge.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 35 of its 39 references that are in the map. First posted on arXiv 4 Feb 2026.

Builds on

35 of 39 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  12. 2020GLU Variants Improve Transformercited by 1,724
23 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  2. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  3. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  4. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  5. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  6. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  7. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  8. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 722
  9. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  10. 2014Neural Turing Machinescited by 630
  11. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  12. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  13. 2014Memory Networkscited by 354
  14. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  15. 2024WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Modelscited by 90
  16. 2025Test-Time Learning for Large Language Modelscited by 46
  17. 2024With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generationcited by 36
  18. 2025Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selectioncited by 13
  19. 2025SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularizationcited by 11
  20. 2025GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuningcited by 8
  21. 2025CTR-LoRA: Curvature-Aware and Trust-Region Guided Low-Rank Adaptation for Large Language Modelscited by 4
  22. 2025LoKI: Low-damage Knowledge Implanting of Large Language Modelscited by 2
  23. 2025DiffoRA: Enabling Parameter-Efficient Fine-Tuning via Differential Module Selectioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agentscited by 9
  2. 2026SmartSearch: How Ranking Beats Structure for Conversational Memory Retrievalcited by 5
  3. 2026Metis: Memory Foundation Modelcited by 5
  4. 2026Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Explorationcited by 3
  5. 2026MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversationscited by 3
  6. 2026Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Trainingcited by 0