# Dynamic Reasoning Chains through Depth-Specialized Mixture-of-Experts in Transformer Architectures

arXiv: https://arxiv.org/abs/2509.20577 · HTML: https://mlsanity.com/paper/2509.20577/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 6 of its 28 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 6 of 6)

- [Harder Tasks Need More Experts: Dynamic Routing in MoE Models](https://mlsanity.com/paper/2403.07652/) (2024)
- [A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks](https://mlsanity.com/paper/2306.07303/) (2023)
- [What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Tasks](https://mlsanity.com/paper/2404.01601/) (2024)
- [DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models](https://mlsanity.com/paper/2409.06669/) (2024)
- [From Explainable to Interpretable Deep Learning for Natural Language Processing in Healthcare: How Far from Reality?](https://mlsanity.com/paper/2403.11894/) (2024)
- [Source-Free Domain Adaptation for RGB-D Semantic Segmentation with Vision Transformers](https://mlsanity.com/paper/2305.14269/) (2023)

## Built on by (most cited first, 1 of 1)

- [Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping](https://mlsanity.com/paper/2603.23998/) (2026)

## Abstract (arXiv)

Contemporary transformer architectures apply identical processing depth to all inputs, creating inefficiencies and limiting reasoning quality. Simple factual queries are subjected to the same multilayered computation as complex logical problems, wasting resources while constraining deep inference. To overcome this, we came up with a concept of Dynamic Reasoning Chains through Depth Specialised Mixture of Experts (DS-MoE), a modular framework that extends the Mixture of Experts paradigm from width-based to depth specialised computation. DS-MoE introduces expert modules optimised for distinct reasoning depths, shallow pattern recognition, compositional reasoning, logical inference, memory integration, and meta-cognitive supervision. A learned routing network dynamically assembles custom reasoning chains, activating only the necessary experts to match input complexity. The dataset on which we trained and evaluated DS-MoE is on The Pile, an 800GB corpus covering diverse domains such as scientific papers, legal texts, programming code, and web content, enabling systematic assessment across reasoning depths. Experimental results demonstrate that DS-MoE achieves up to 16 per cent computational savings and 35 per cent faster inference compared to uniform-depth transformers, while delivering 2.8 per cent higher accuracy on complex multi-step reasoning benchmarks. Furthermore, routing decisions yield interpretable reasoning chains, enhancing transparency and scalability. These findings establish DS-MoE as a significant advancement in adaptive neural architectures, demonstrating that depth-specialised modular processing can simultaneously improve efficiency, reasoning quality, and interpretability in large-scale language models.