mlsanityData to 7 Oct 2026

From Concentration to Differentiation and Back: Routing Effective Rank in MoE Reasoning Cohorts

arXiv 2609.06403 ↗First posted 6 Sep 2026cs.AIcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Test-time scaling produces cohorts of reasoning rollouts, yet there is no standard label-free account of how their internal computation reorganizes as inference unfolds. We introduce routing effective rank deff, the entropy-effective dimensionality of a cross-rollout graph built from MoE expert-routing similarity. Across ten MoE configurations and five math/science benchmarks, deff exhibits a reproducible low-high-low trajectory, with a prominent interior maximum in 98.5% of 3,105 model-question cohorts: routing similarity is concentrated early, maximally differentiated at intermediate budgets, and reconcentrated later, and the timing of this maximum varies systematically with architecture and reasoning effort. An exact decomposition separates cohort-wide common-mode mass from residual spectral dimensionality: common-mode reallocation accounts for about two thirds of the trajectory, while the residual spectrum contributes about one quarter and retains substantial variation beyond the common mode. The decomposition further localizes behavior: among non-unanimous cohorts, increases in common-mode concentration strongly predict same-answer recoverability, and higher reasoning effort delays the maximum by 2.59 octaves (doublings of the token budget) and consistently expands the high-rank period across all four tested architectures, locating the effort effect in timing and duration rather than peak amplitude. Correctness comparisons separate structural monitoring from answer selection, positioning routing effective rank as a decomposable, label-free diagnostic of cohort organization - a principled spectral lens on how MoE reasoning cohorts differentiate and reconcentrate over inference time.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 43 of its 52 references that are in the map. First posted on arXiv 6 Sep 2026.

Builds on

43 of 52 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026Does the Same Token Mean the Same State? MoE Routing as Signal for Reasoning Control22 Jun
  2. 2026LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals7 Apr
  3. 2026NEX: Neuron Explore-Exploit Scoring for Label-Free Chain-of-Thought Selection and Model Ranking5 Feb
  4. 2025ReProbe: Efficient Test-Time Scaling of Multi-Step Reasoning by Probing Internal States of Large Language Models9 Nov
  5. 2025Do LLMs Signal When They're Right? Evidence from Neuron Agreement30 Oct
  6. 2025Tracing the Traces: Latent Temporal Signals for Efficient and Accurate Reasoning12 Oct
  7. 2025gpt-oss-120b & gpt-oss-20b Model Card8 Aug
  8. 2025s1: Simple test-time scaling31 Jan
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  10. 2024Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning19 Jan
  11. 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs19 May
  12. 2022Language Models (Mostly) Know What They Know11 Jul

Other references · 31

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  11. 2024Mixtral of Expertscited by 1,785
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
19 more
  1. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  2. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  3. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  4. 2017Selective Classification for Deep Neural Networkscited by 799
  5. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  6. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  7. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 299
  8. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
  9. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 88
  10. 2025Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Modelscited by 25
  11. 2025Mapping the Minds of LLMs: A Graph-Based Analysis of Reasoning LLMcited by 21
  12. 2025The Shape of Reasoning: Topological Analysis of Reasoning Traces in Large Language Modelscited by 9
  13. 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertisecited by 9
  14. 2026When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Modelscited by 8
  15. 2026The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Levelcited by 7
  16. 2025Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanismscited by 6
  17. 2026ReasoningFlow: Discourse Structures for Understanding LLM Reasoning Tracescited by 5
  18. 2026SliceGraph: Mapping Process Isomers in Multi-Run Chain-of-Thought Reasoningcited by 2
  19. 2026The Tell-Tale Norm: ℓ2 Magnitude as a Signal for Reasoning Dynamics in Large Language Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond the Trace: Coupling an Interpretable Reasoning-State Readout to Native MoE Routing
  2. 2025Dynamic Experts Search: Enhancing Reasoning in Mixture-of-Experts LLMs at Test Time
  3. 2026Certain Head, Uncertain Tail: Expert-Sample for Test-Time Scaling in Fine-Grained MoE
  4. 2026Is MoE Routing a Huffman Code? Discovering the Frequency-Diversity Law in Chain-of-Thought
  5. 2025The relationship between reasoning and performance in large language models--o3 (mini) thinks harder, not longer
  6. 2026ThinkRouter: Efficient Reasoning via Routing Thinking between Latent and Discrete Spaces
  7. 2026Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts
  8. 2026You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs