mlsanityData to 14 Oct 2026

DND: Boosting Large Language Models with Dynamic Nested Depth

arXiv 2510.11001 ↗First posted 13 Oct 2025cs.CL cs.AIcited by 1

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

We introduce Dynamic Nested Depth (DND), a novel method that improves performance for off-the-shelf LLMs by selecting critical tokens to reprocess in a nested depth manner. Specifically, at the end of the given transformer layer, DND identifies more critical tokens with a router and feeds them back for an extra round of processing, effectively ``reviewing" difficult tokens while avoiding redundant computation for easier ones. The dynamic selection mechanism is tailored for precise control via two novel strategies: a router controlling loss to enhance token selection distinguishability, and a threshold control scheme to ensure selection stability. We demonstrate the effectiveness of DND by directly integrating it into pre-trained dense and MoE models during a post-training phase. On diverse benchmarks, this approach boosts the performances of the dense Qwen3-1.7B by 1.88% and the MoE Qwen3-30B-A3B by 0.87%, all with a minimal parameter and computing increase.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 34 of its 38 references that are in the map. First posted on arXiv 13 Oct 2025.

Builds on

34 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Qwen3 Technical Report14 May
  2. 2024Mixtral of Experts8 Jan
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
19 more
  1. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  2. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  3. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  4. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  5. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  6. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  7. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  8. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  9. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  10. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 240
  11. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
  12. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 136
  13. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  14. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinkingcited by 57
  15. 2025Estimating LLM Uncertainty with Evidencecited by 26
  16. 2025The Efficiency vs. Accuracy Trade-off: Optimizing RAG-Enhanced LLM Recommender Systems Using Multi-Head Early Exitcited by 10
  17. 2024Attention Is All You Need For Mixture-of-Depths Routingcited by 7
  18. 2025ThinkingViT: Matryoshka Thinking Vision Transformer for Elastic Inferencecited by 7
  19. 2025TR-PTS: Task-Relevant Parameter and Token Selection for Efficient Tuningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Improving Test-Time Scaling with Adaptive Looped Transformerscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMs
  2. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
  3. 2024Transformer Layer Injection: A Novel Approach for Efficient Upscaling of Large Language Models
  4. 2026Sparse or Dense? A Mechanistic Estimation of Computation Density in Transformer-based LLMs
  5. 2025Leveraging the true depth of LLMs
  6. 2026Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs
  7. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving
  8. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA