mlsanityData to 14 Oct 2026

Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models

arXiv 2609.27373 ↗First posted 23 Sep 2026cs.CL cs.LGcited by 1

On the map: LLM reasoning → Test-time compute and efficient thinking also fits: Diffusion reasoning and test-time scaling

Abstract (arXiv)

Recurrent-depth language models, such as looped Transformers, repeatedly apply shared network blocks to refine latent representations without generating explicit intermediate reasoning tokens. However, each step recomputes full attention over the entire context, repeating costly global routing. We study how attention routing evolves across recurrent depth and find a consistent separation in convergence timescales: attention support and distributions stabilize substantially earlier than hidden states and attention outputs. This suggests two stages of recurrent inference: early discovery of a sparse working set, followed by representation refinement over largely stable routing support. Motivated by this finding, we introduce WISE (Working-set Inference with Support Exploitation), a training-free method that uses unrestricted attention during early recurrent steps to discover a block-structured working set, then reuses its support in later steps while keeping attention weights and recurrent refinement dynamic. Controlled interventions show that multi-step discovery yields more effective working sets than first-step selection, and that support reuse better preserves model behavior than more restrictive forms of attention reuse. Across multi-hop QA benchmarks, WISE largely preserves full-attention performance. Matched context-scaling experiments reveal an increasingly favorable quality-efficiency tradeoff as routing support becomes sparser with longer contexts. A sparse-attention implementation achieves up to a 1.76x late-step attention speedup over native FlashAttention at 4K context. Code: https://github.com/tbn5pj/WISE_code.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 30 of its 32 references that are in the map. First posted on arXiv 23 Sep 2026.

Builds on

30 of 32 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference18 Dec
  2. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence10 Nov
  3. 2025Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scaling22 Aug
  4. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformers24 Feb
  5. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach7 Feb

Other references · 25

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  10. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  11. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  12. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
13 more
  1. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  2. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  3. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  4. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  5. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 273
  6. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  7. 2023Looped Transformers as Programmable Computerscited by 237
  8. 2019Depth-Adaptive Transformercited by 236
  9. 2021PonderNet: Learning to Pondercited by 168
  10. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 149
  11. 2026Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformerscited by 37
  12. 2019Sharing Attention Weights for Fast Transformercited by 34
  13. 2024RefreshKV: Updating Small KV Cache During Long-form Generationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026RAMP: Recurrent Adaptive Mixed Precision for Efficient Prefill in Recurrent-Depth Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Mechanistic Analysis of Looped Reasoning Language Models
  2. 2026RAT+: Train Dense, Infer Sparse -- Recurrence Augmented Attention for Dilated Inference
  3. 2026Not All Thinking is Created Equal: Latent Reasoning Discovers a Recurrent Search Algorithm for Depth Generalization
  4. 2026Think Deep, Speak Once: Relit, A Recursive Latent Implicit Transformer Framework
  5. 2026DeepThink: A Conceptual Model for Knowledge-Addressable Recurrent Language Modeling
  6. 2026On-Demand Attention: Language Models Know When to Recall
  7. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
  8. 2026RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory