mlsanityData to 7 Oct 2026

Co-LMLM: Continuous-Query Limited Memory Language Models

arXiv 2607.07707 ↗First posted 8 Jul 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Externalizing knowledge in LLM pre-training is a promising avenue to achieve higher performance at smaller scales, control knowledge use, and overall increase model transparency. We propose continuous-query limited memory language models (Co-LMLM), an LLM that interleaves flexible vector retrieval queries with next-token predictions, and is pre-trained to copy knowledge returned from the KB, rather than memorize it. Co-LMLM is pre-trained with a scalable approach that jointly trains a knowledge-externalizing LLM, induces its knowledge base, and learns an expressive continuous retrieval mechanism. Across pre-training at multiple model scales, Co-LMLM outperforms prior knowledge-externalizing and vanilla LLMs in both perplexity and factual precision. At 360M scale, this includes lower perplexity than models pre-trained on 40× more data, and SimpleQA-verified performance that is in line with gpt-4o-mini and higher than Claude Sonnet 4.5.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 46 of its 110 references that are in the map. First posted on arXiv 8 Jul 2026.

Builds on

46 of 110 references.

References · 46

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
34 more
  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  2. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  3. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  5. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  6. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  7. 2022Toy Models of Superpositioncited by 928
  8. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  9. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  10. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  11. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  12. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  13. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  14. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  15. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  16. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  17. 2024The Faiss librarycited by 370
  18. 2024Measuring short-form factuality in large language modelscited by 345
  19. 2024Generative Representational Instruction Tuningcited by 227
  20. 2022Memorizing Transformerscited by 219
  21. 2019Large Memory Layers with Product Keyscited by 120
  22. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  23. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  24. 2024DRAGIN: Dynamic Retrieval Augmented Generation based on the Information Needs of Large Language Modelscited by 63
  25. 2025SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledgecited by 44
  26. 2024Memory Layers at Scalecited by 39
  27. 2024MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memorycited by 35
  28. 2024KBLaM: Knowledge Base augmented Language Modelcited by 16
  29. 2024OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMscited by 14
  30. 2024Advancing Large Language Model Attribution through Self-Improvingcited by 14
  31. 2025Pretraining with hierarchical memories: separating long-tail and common knowledgecited by 13
  32. 2025Pre-training Limited Memory Language Models with Internal and External Knowledgecited by 11
  33. 2025Memorization Sinks: Isolating Memorization during LLM Trainingcited by 7
  34. 2025ImpRAG: Retrieval-Augmented Generation with Implicit Queriescited by 5

Built on by

Not cited yet.