mlsanityData to 14 Oct 2026

PERK: Long-Context Reasoning as Test-Time Learning

arXiv 2507.06415 ↗First posted 8 Jul 2025cs.CL cs.LGcited by 4

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Long-context reasoning requires accurately identifying relevant information in extensive, noisy input contexts. In this work, we propose PERK (Parameter Efficient Reasoning over Knowledge), a scalable approach for learning to encode long contexts using gradient updates at test time. Specifically, PERK employs two nested optimization loops in a meta-training phase. The inner loop rapidly encodes contexts into a low-rank adapter (LoRA) that serves as a parameter-efficient memory module for the base model. Concurrently, the outer loop learns to use the updated adapter to accurately recall and reason over relevant information from the encoded long context. Our evaluations on several long-context reasoning tasks show that PERK significantly outperforms the standard long-context finetuning, achieving average absolute performance gains of up to 20% for Qwen-2.5 (0.5B & 7B) on synthetic and real-world long-context reasoning. PERK also maintains its advantages across model scales and families. Compared to specialized long-context LLMs, PERK matches or surpasses their performance. Finally, our analyses show PERK is more robust to reasoning complexity, length extrapolation, and the positions of relevant information in contexts. https://perk-long-context.web.app

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 50 of its 83 references that are in the map. First posted on arXiv 8 Jul 2025.

Builds on

50 of 83 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Titans: Learning to Memorize at Test Time31 Dec

Other references · 49

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  7. 2017Decoupled Weight Decay Regularizationcited by 1,831
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  10. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  12. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
37 more
  1. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  2. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  3. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  4. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  5. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  6. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  7. 2024xLSTM: Extended Long Short-Term Memorycited by 327
  8. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  9. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  10. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  11. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  12. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  13. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 216
  14. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 208
  15. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  16. 2025Qwen2.5-1M Technical Reportcited by 164
  17. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 152
  18. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  19. 2022Learning by Distilling Contextcited by 138
  20. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 136
  21. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  22. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  23. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  24. 2023HyperAttention: Long-context Attention in Near-Linear Timecited by 85
  25. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 66
  26. 2024FABLES: Evaluating faithfulness and content selection in book-length summarizationcited by 65
  27. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 62
  28. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 56
  29. 2023A Study on ReLU and Softmax in Transformercited by 53
  30. 2023Meta-Learning Online Adaptation of Language Modelscited by 43
  31. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 29
  32. 2024A Controlled Study on Long Context Extension and Generalization in LLMscited by 26
  33. 2025Training Plug-n-Play Knowledge Modules with Deep Context Distillationcited by 24
  34. 2024Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystackcited by 19
  35. 2024From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Datacited by 18
  36. 2025Eliciting In-context Retrieval and Reasoning for Long-context Large Language Modelscited by 11
  37. 2025Minerva: A Programmable Memory Test Benchmark for Language Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Accumulating Context Changes the Beliefs of Language Modelscited by 28
  2. 2026DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Modelscited by 2
  3. 2026Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Trainingcited by 1
  4. 2026Self-Guided Test-Time Training for Long-Context LLMscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026A Decomposition Perspective to Long-context Reasoning for LLMs
  2. 2026LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning
  3. 2025Let's (not) just put things in Context: Test-Time Training for Long-Context LLMs
  4. 2024Large Language Models Can Self-Improve in Long-context Reasoning
  5. 2026MILES: Modular Instruction Memory with Learnable Selection for Self-Improving LLM Reasoning
  6. 2025Less is More: Improving LLM Reasoning with Minimal Test-Time Intervention
  7. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion
  8. 2025Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning