mlsanityData to 14 Oct 2026

Grounding latent algorithm routing in transformer reasoning

arXiv 2607.24471 ↗First posted 27 Jul 2026cs.CLcited by 0

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

A central question in the in-context learning literature is whether transformers can organize episode-level adaptation around different inductive-bias families. We study this question in a controlled setting through latent algorithm routing: route-like behavior in which the solver-family preference changes with the latent data-generating regime while prompt form is held fixed, remains stable under nuisance perturbations, and is selectively influenced by targeted activation interventions without large losses in answer quality. We introduce ROUTEBENCH, a diagnostic benchmark whose regimes differentially favor global shrinkage, sparsity, robustness, and locality, operationalized by ridge-like, lasso-like, Huber-like, and kNN-like family representatives. Across dense decoder-only transformers trained from scratch at 44M-612M parameters, a 306M model closes 80.9 percent of the oracle-routing gap and achieves route F1 of 84.1. The effect remains substantial under natural-language renderings, shuffled supports, lexical paraphrases, and a unified four-way routing setting. Stronger adaptive alternatives, including an input-conditioned soft mixture and an unsupervised Gumbel router, narrow the gap but remain below the 306M and 612M models on route F1 and OOD performance. Probe controls and matched activation-patching controls further show that route-relevant internal directions are decodable and functionally involved in solver-family-consistent output behavior. These results provide controlled evidence that dense transformers trained on ROUTEBENCH can develop route-like internal variables, but they do not establish universal routing in pretrained language models or unrestricted natural-language reasoning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 41 references that are in the map. First posted on arXiv 27 Jul 2026.

Builds on

36 of 41 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?31 Oct
  2. 2024How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning28 Feb

Other references · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  6. 2022In-context Learning and Induction Headscited by 964
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  8. 2023Steering Language Models With Activation Engineeringcited by 922
  9. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  10. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  11. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  12. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
22 more
  1. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  2. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  3. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  4. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  5. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  6. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  7. 2024How to use and interpret activation patchingcited by 214
  8. 2023Localizing Model Behavior with Path Patchingcited by 195
  9. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 110
  10. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 105
  11. 2023Uncovering mesa-optimization algorithms in Transformerscited by 100
  12. 2024What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formationcited by 89
  13. 2025Understanding Reasoning in Thinking Language Models via Steering Vectorscited by 72
  14. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 63
  15. 2024POSIX: A Prompt Sensitivity Index For Large Language Modelscited by 59
  16. 2024A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Taskcited by 49
  17. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineeringcited by 35
  18. 2026STABLE: Efficient Hybrid Nearest Neighbor Search via Magnitude-Uniformity and Cardinality-Robustnesscited by 26
  19. 2024On the Noise Robustness of In-Context Learning for Text Generationcited by 16
  20. 2024A Implies B: Circuit Analysis in LLMs for Propositional Logical Reasoningcited by 15
  21. 2026Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptationcited by 14
  22. 2026A Statistical Framework for Auditing Behavioral Dependence and Induced Bias in LLM Judgescited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference
  2. 2026Directional Routing in Transformers
  3. 2026Adaptive Loops and Memory in Transformers: Think Harder or Know More?
  4. 2026T^2MLR: Transformer with Temporal Middle-Layer Recurrence
  5. 2026ThinkRouter: Efficient Reasoning via Routing Thinking between Latent and Discrete Spaces
  6. 2025From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamics
  7. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Models
  8. 2024Understanding Transformer Reasoning Capabilities via Graph Algorithms