mlsanityData to 14 Oct 2026

Mixture-of-Depths: Dynamically allocating compute in transformer-based language models

arXiv 2404.02258 ↗First posted 2 Apr 2024cs.LG cs.CLcited by 258

Not on the map. Why?

Cited by 48 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformer-based language models spread FLOPs uniformly across input sequences. In this work we demonstrate that transformers can instead learn to dynamically allocate FLOPs (or compute) to specific positions in a sequence, optimising the allocation along the sequence for different layers across the model depth. Our method enforces a total compute budget by capping the number of tokens (k) that can participate in the self-attention and MLP computations at a given layer. The tokens to be processed are determined by the network using a top-k routing mechanism. Since k is defined a priori, this simple procedure uses a static computation graph with known tensor sizes, unlike other conditional computation techniques. Nevertheless, since the identities of the k tokens are fluid, this method can expend FLOPs non-uniformly across the time and model depth dimensions. Thus, compute expenditure is entirely predictable in sum total, but dynamic and context-sensitive at the token-level. Not only do models trained in this way learn to dynamically allocate compute, they do so efficiently. These models match baseline performance for equivalent FLOPS and wall-clock times to train, but require a fraction of the FLOPs per forward pass, and can be upwards of 50% faster to step during post-training sampling.

Key facts. Cited by 258 papers in the map (data through 14 Oct 2026). Builds on 15 of its 19 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

15 of 19 references.

References · 15

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  5. 2020Efficient Transformers: A Surveycited by 790
  6. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  7. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 332
  8. 2019Depth-Adaptive Transformercited by 236
  9. 2015Conditional Computation in Neural Networks for faster modelscited by 167
  10. 2013Deep Learning of Representations: Looking Forwardcited by 84
  11. 2020Compression of Deep Learning Models for Text: A Surveycited by 64
  12. 2020Controlling Computation versus Quality for Neural Sequence Modelscited by 27
3 more
  1. 2014Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learningcited by 24
  2. 2021Anytime Dense Prediction with Confidence Adaptivitycited by 18
  3. 2016Variable Computation in Recurrent Neural Networkscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 50; 2025: 80; 2026: 128.

258 papers.

Sort:

By area:

  1. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  2. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  3. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  4. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
  5. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 163
  6. 2024Learning-to-Cache: Accelerating Diffusion Transformer via Layer Cachingcited by 142
  7. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  8. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  9. 2025Why do LLMs attend to the first token?cited by 127
  10. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 125
  11. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 117
  12. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
38 more
  1. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 78
  2. 2024Complexity Experts are Task-Discriminative Learners for Any Image Restorationcited by 77
  3. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 74
  4. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  5. 2025MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulationcited by 70
  6. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 62
  7. 2024Online Video Understanding: OVBench and VideoChat-Onlinecited by 61
  8. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 61
  9. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 58
  10. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinkingcited by 57
  11. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  12. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 55
  13. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 54
  14. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 47
  15. 2024TinyFusion: Diffusion Transformers Learned Shallowcited by 45
  16. 2024AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Modelscited by 44
  17. 2024Memory3: Language Modeling with Explicit Memorycited by 40
  18. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 38
  19. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 34
  20. 2024Data curation via joint example selection further accelerates multimodal learningcited by 33
  21. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  22. 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
  23. 2024Calibrating Reasoning in Language Models with Internal Consistencycited by 28
  24. 2025Adaptive Layer-skipping in Pre-trained LLMscited by 27
  25. 2025System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcutscited by 27
  26. 2024Mixture of Nested Experts: Adaptive Processing of Visual Tokenscited by 25
  27. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniquescited by 24
  28. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughtscited by 24
  29. 2024A deeper look at depth pruning of LLMscited by 23
  30. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 23
  31. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exitingcited by 23
  32. 2024Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformerscited by 22
  33. 2025TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Trainingcited by 21
  34. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizationscited by 21
  35. 2024p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decaycited by 16
  36. 2025Adaptive Self-improvement LLM Agentic System for ML Library Developmentcited by 16
  37. 2025Beyond Linear Probes: Dynamic Safety Monitoring for Language Modelscited by 16
  38. 2025Think-at-Hard: Dynamic Looped Transformers for Improved Reasoningcited by 16

50 most cited of 258 All 258 (JSON)