mlsanityData to 7 Oct 2026

Depth-Adaptive Transformer

arXiv 1910.10073 ↗First posted 22 Oct 2019cs.CL cs.LGcited by 227

Not on the map. Why?

Cited by 66 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

State of the art sequence-to-sequence models for large scale tasks perform a fixed number of computations for each input sequence regardless of whether it is easy or hard to process. In this paper, we train Transformer models which can make output predictions at different stages of the network and we investigate different ways to predict how much computation is required for a particular sequence. Unlike dynamic computation in Universal Transformers, which applies the same set of layers iteratively, we apply different layers at every step to adjust both the amount of computation as well as the model capacity. On IWSLT German-English translation our approach matches the accuracy of a well tuned baseline Transformer while using less than a quarter of the decoder layers.

Key facts. Cited by 227 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 24; 2022: 11; 2023: 25; 2024: 55; 2025: 46; 2026: 65.

227 papers.

Sort:

By area:

  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  3. 2023LightGlue: Local Feature Matching at Light Speedcited by 583
  4. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  5. 2021Dynamic Neural Networks: A Surveycited by 405
  6. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  7. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  8. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  9. 2022Confident Adaptive Language Modelingcited by 297
  10. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  11. 2023Speculative Decoding with Big Little Decodercited by 163
  12. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
38 more
  1. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  2. 2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challengescited by 114
  3. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  4. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 95
  5. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  6. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  7. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  8. 2021Adaptive Inference through Early-Exit Networks: Design, Challenges and Directionscited by 74
  9. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  10. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  11. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  12. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  13. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 61
  14. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 58
  15. 2021A Survey on Green Deep Learningcited by 52
  16. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 48
  17. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 45
  18. 2021Towards Efficient NLP: A Standard Evaluation and A Strong Baselinecited by 44
  19. 2024A Survey on Transformer Compressioncited by 44
  20. 2024TinyFusion: Diffusion Transformers Learned Shallowcited by 44
  21. 2022Token Dropping for Efficient BERT Pretrainingcited by 42
  22. 2022You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Modelcited by 42
  23. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 41
  24. 2021Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervisioncited by 40
  25. 2021Paradigm Shift in Natural Language Processingcited by 39
  26. 2024Memory3: Language Modeling with Explicit Memorycited by 39
  27. 2024Faster Cascades via Speculative Decodingcited by 38
  28. 2024Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questionscited by 35
  29. 2022Path Independent Equilibrium Models Can Better Exploit Test-Time Computationcited by 33
  30. 2022A Simple Hash-Based Early Exiting Approach For Language Understanding and Generationcited by 32
  31. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanismcited by 31
  32. 2023Can Public Large Language Models Help Private Cross-device Federated Learning?cited by 30
  33. 2021Accelerating BERT Inference for Sequence Labeling via Early-Exitcited by 29
  34. 2025Why Prompt Design Matters and Works: A Complexity Analysis of Prompt Search Space in LLMscited by 29
  35. 2023The Benefits of Bad Advice: Autocontrastive Decoding across Model Layerscited by 28
  36. 2023Sparse Universal Transformercited by 28
  37. 2021Dual-view Molecule Pre-trainingcited by 27
  38. 2023CoTFormer: A Chain-of-Thought Driven Architecture with Budget-Adaptive Computation Cost at Inferencecited by 27

50 most cited of 227 All 227 (JSON)