mlsanityData to 7 Oct 2026

Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search

arXiv 2010.07003 ↗First posted 14 Oct 2020cs.CL cs.LGcited by 76

Not on the map. Why?

Cited by 26 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite transformers' impressive accuracy, their computational cost is often prohibitive to use with limited computational resources. Most previous approaches to improve inference efficiency require a separate model for each possible computational budget. In this paper, we extend PoWER-BERT (Goyal et al., 2020) and propose Length-Adaptive Transformer that can be used for various inference scenarios after one-shot training. We train a transformer with LengthDrop, a structural variant of dropout, which stochastically determines a sequence length at each layer. We then conduct a multi-objective evolutionary search to find a length configuration that maximizes the accuracy and minimizes the efficiency metric under any given computational budget. Additionally, we significantly extend the applicability of PoWER-BERT beyond sequence-level classification into token-level classification with Drop-and-Restore process that drops word-vectors temporarily in intermediate layers and restores at the last layer if necessary. We empirically verify the utility of the proposed approach by demonstrating the superior accuracy-efficiency trade-off under various setups, including span-based question answering and text classification. Code is available at https://github.com/clovaai/length-adaptive-transformer.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 17; 2023: 21; 2024: 18; 2025: 12; 2026: 2.

76 papers.

Sort:

By area:

  1. 2022Token Merging: Your ViT But Fastercited by 995
  2. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  3. 2022Confident Adaptive Language Modelingcited by 297
  4. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  5. 2021Learned Token Pruning for Transformerscited by 138
  6. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 123
  7. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  8. 2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 89
  9. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  10. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 61
  11. 2021A Survey on Green Deep Learningcited by 52
  12. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
38 more
  1. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  2. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  3. 2024ElasticTok: Adaptive Tokenization for Image and Videocited by 44
  4. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 43
  5. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  6. 2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inferencecited by 40
  7. 2024MELTing point: Mobile Evaluation of Language Transformerscited by 35
  8. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  9. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
  10. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  11. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25
  12. 2025Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representationcited by 25
  13. 2022A Survey on Dynamic Neural Networks for Natural Language Processingcited by 24
  14. 2024Token-level Correlation-guided Compression for Efficient Multimodal Document Understandingcited by 22
  15. 2022Transkimmer: Transformer Learns to Layer-wise Skimcited by 21
  16. 2022AdapLeR: Speeding up Inference by Adaptive Length Reductioncited by 19
  17. 2024Characterizing Prompt Compression Methods for Long Context Inferencecited by 18
  18. 2023OrchestraLLM: Efficient Orchestration of Language Models for Dialogue State Trackingcited by 17
  19. 2021Block-Skim: Efficient Question Answering for Transformercited by 16
  20. 2021A Study on Token Pruning for ColBERTcited by 14
  21. 2023Accelerating Transducers through Adjacent Token Mergingcited by 14
  22. 2022Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selectioncited by 13
  23. 2022DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routingcited by 13
  24. 2023ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inferencecited by 13
  25. 2022Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformerscited by 11
  26. 2024iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Modelscited by 11
  27. 2022Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?cited by 10
  28. 2023Optimizing Retrieval-augmented Reader Models via Token Eliminationcited by 10
  29. 2023Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decodingcited by 10
  30. 2023Do We Really Need a Large Number of Visual Prompts?cited by 9
  31. 2024Selection-p: Self-Supervised Task-Agnostic Prompt Compression for Faithfulness and Transferabilitycited by 9
  32. 2025CipherPrune: Efficient and Scalable Private Transformer Inferencecited by 9
  33. 2022E-LANG: Energy-Based Joint Inferencing of Super and Swift Language Modelscited by 8
  34. 2025Put Teacher in Student's Shoes: Cross-Distillation for Ultra-compact Model Compression Frameworkcited by 8
  35. 2025Rethinking Chain-of-Thought Reasoning for Videoscited by 8
  36. 2023DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Modelscited by 7
  37. 2024Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classificationcited by 7
  38. 2025LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compressioncited by 7

50 most cited of 76 All 76 (JSON)