mlsanityData to 7 Oct 2026

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

arXiv 2601.07372 ↗First posted 12 Jan 2026cs.CL cs.AIcited by 78

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

While Mixture-of-Experts (MoE) scales capacity via conditional computation, Transformers lack a native primitive for knowledge lookup, forcing them to inefficiently simulate retrieval through computation. To address this, we introduce conditional memory as a complementary sparsity axis, instantiated via Engram, a module that modernizes classic N-gram embedding for O(1) lookup. By formulating the Sparsity Allocation problem, we uncover a U-shaped scaling law that optimizes the trade-off between neural computation (MoE) and static memory (Engram). Guided by this law, we scale Engram to 27B parameters, achieving superior performance over a strictly iso-parameter and iso-FLOPs MoE baseline. Most notably, while the memory module is expected to aid knowledge retrieval (e.g., MMLU +3.4; CMMLU +4.0), we observe even larger gains in general reasoning (e.g., BBH +5.0; ARC-Challenge +3.7) and code/math domains~(HumanEval +3.0; MATH +2.4). Mechanistic analyses reveal that Engram relieves the backbone's early layers from static reconstruction, effectively deepening the network for complex reasoning. Furthermore, by delegating local dependencies to lookups, it frees up attention capacity for global context, substantially boosting long-context retrieval (e.g., Multi-Query NIAH: 84.2 to 97.0). Finally, Engram establishes infrastructure-aware efficiency: its deterministic addressing enables runtime prefetching from host memory, incurring negligible overhead. We envision conditional memory as an indispensable modeling primitive for next-generation sparse models.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 106 of its 140 references that are in the map. First posted on arXiv 12 Jan 2026.

Builds on

106 of 140 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning26 Aug
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  3. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 100

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  12. 2014Going Deeper with Convolutionscited by 4,231
38 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  8. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  13. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  14. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  15. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  16. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  17. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  18. 2019Root Mean Square Layer Normalizationcited by 1,425
  19. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  20. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  21. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  22. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  23. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  24. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  25. 2022Mass-Editing Memory in a Transformercited by 976
  26. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  27. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  28. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  29. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  30. 2023StarCoder: may the source be with you!cited by 790
  31. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  32. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  33. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  34. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  35. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  36. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  37. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  38. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518

50 most cited of 100 All 100 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 78.

78 papers.

Sort:

By area:

  1. 2026Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineeringcited by 68
  2. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 20
  3. 2026Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioningcited by 15
  4. 2026Scaling Embeddings Outperforms Scaling Experts in Language Modelscited by 13
  5. 2026Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baselinecited by 8
  6. 2026UMEM: Unified Memory Extraction and Management Framework for Generalizable Memorycited by 7
  7. 2026HRM-Text: Efficient Pretraining Beyond Scalingcited by 7
  8. 2026ThinkRouter: Efficient Reasoning via Routing Thinking between Latent and Discrete Spacescited by 6
  9. 2026PM-Bench: Evaluating Prospective Memory in LLM Agentscited by 6
  10. 2026MeKi: Memory-based Expert Knowledge Injection for Efficient LLM Scalingcited by 5
  11. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  12. 2026MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokenscited by 5
38 more
  1. 2026Kernel-Smith: A Unified Recipe for Evolutionary Kernel Optimizationcited by 5
  2. 2026Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memorycited by 4
  3. 2026LiteToken: Removing Intermediate Merge Residues From BPE Tokenizerscited by 3
  4. 2026go-mHC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matricescited by 3
  5. 2026Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMscited by 3
  6. 2026Mem-π: Adaptive Memory through Learning When and What to Generatecited by 3
  7. 2026Lngram: N-gram Conditional Memory in Latent Spacecited by 3
  8. 2026Frozen Memory Is Not Enough: Rethinking External Memory as Extractioncited by 3
  9. 2026A Collision-Free Hot-Tier Extension for Engram-Style Conditional Memory: A Controlled Study of Training Dynamicscited by 2
  10. 2026L3: Large Lookup Layerscited by 2
  11. 2026Simple is Better: Multiplication May Be All You Need for LLM Request Schedulingcited by 2
  12. 2026Cram Less to Fit More: Training Data Pruning Improves Memorization of Factscited by 2
  13. 2026Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulationcited by 2
  14. 2026When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1-norm Lower Boundscited by 2
  15. 2026Simulating Human Memory with Language Modelscited by 2
  16. 2026User as Engram: Internalizing Per-User Memory as Local Parametric Editscited by 2
  17. 2026Memory for Large Language Modelscited by 2
  18. 2026MemSFT: Mitigating Alignment Tax with an External Parametric Memorycited by 2
  19. 2026Beyond Conditional Computation: Retrieval-Augmented Genomic Foundation Models with Gengramcited by 1
  20. 2026AllMem: A Memory-centric Recipe for Efficient Long-context Modelingcited by 1
  21. 2026Adaptive Engram Memory System for Indonesian Language Model: Generative AI Based on TOBA LM for Batak and Minang Languagecited by 1
  22. 2026Decidable By Construction: Design-Time Verification for Truly Fearless Systemscited by 1
  23. 2026Beyond N-gram: Data-Aware X-GRAM Extraction for Efficient Embedding Parameter Scalingcited by 1
  24. 2026Efficient Pre-Training with Token Superpositioncited by 1
  25. 2026Geometric Factual Recall in Transformerscited by 1
  26. 2026Phasor Memory Networks: Stable Backpropagation Through Time for Scalable Explicit Memorycited by 1
  27. 2026Tiny-Engram: Trigger-Indexed Concept Tables for Generative Visioncited by 1
  28. 2026Tensor Memory: Fixed-Size Recurrent State for Long-Horizon Transformerscited by 1
  29. 2026Do Value Vectors in Deep Layers Need Context from the Residual Stream?cited by 1
  30. 2026Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMscited by 1
  31. 2026Augmenting Molecular Language Models with Local n-gram Memorycited by 1
  32. 2026Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instabilitycited by 1
  33. 2026CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limitscited by 1
  34. 2026When to Adapt: Conditional Memory Adapters for Retention-Preserving Domain Specializationcited by 1
  35. 2026Digital Metabolism: Decoupling Logic from Facts via Regenerative Unlearning -- Towards a Pure Neural Logic Corecited by 0
  36. 2026ROSA-Tuning: Enhancing Long-Context Modeling via Suffix Matchingcited by 0
  37. 2026Mixture of Chapters: Scaling Learnt Memory in Transformerscited by 0
  38. 2026NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipeliningcited by 0

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MLP Memory: A Retriever-Pretrained Memory for Large Language Models
  2. 2026Less is MoE: Trimming Experts in Domain-Specialist Language Models
  3. 2025Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks
  4. 2026EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory
  5. 2022An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks
  6. 2025ExplicitLM: Decoupling Knowledge from Parameters via Explicit Memory Banks
  7. 2024Memory3: Language Modeling with Explicit Memory
  8. 2024Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks