# AdapLeR: Speeding up Inference by Adaptive Length Reduction

arXiv: https://arxiv.org/abs/2203.08991 · HTML: https://mlsanity.com/paper/2203.08991/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 69 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT](https://mlsanity.com/paper/1909.05840/) (2019)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning](https://mlsanity.com/paper/2012.09852/) (2020)
- [FastBERT: a Self-distilling BERT with Adaptive Inference Time](https://mlsanity.com/paper/2004.02178/) (2020)

## Built on by (most cited first, 19 of 19)

- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://mlsanity.com/paper/2310.05736/) (2023)
- [C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness](https://mlsanity.com/paper/2412.11664/) (2024)
- [Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems](https://mlsanity.com/paper/2312.15234/) (2023)
- [Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems](https://mlsanity.com/paper/2210.05528/) (2022)
- [The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Survey](https://mlsanity.com/paper/2401.07872/) (2024)
- [AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression](https://mlsanity.com/paper/2305.10010/) (2023)
- [Efficiently Controlling Multiple Risks with Pareto Testing](https://mlsanity.com/paper/2210.07913/) (2022)
- [Efficient Prompting via Dynamic In-Context Learning](https://mlsanity.com/paper/2305.11170/) (2023)
- [Rethinking Token Reduction for State Space Models](https://mlsanity.com/paper/2410.14725/) (2024)
- [Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?](https://mlsanity.com/paper/2211.12707/) (2022)
- [Rethinking Multimodal Content Moderation from an Asymmetric Angle with Mixed-modality](https://mlsanity.com/paper/2305.10547/) (2023)
- [Selection-p: Self-Supervised Task-Agnostic Prompt Compression for Faithfulness and Transferability](https://mlsanity.com/paper/2410.11786/) (2024)
- [Language Model Guided Interpretable Video Action Reasoning](https://mlsanity.com/paper/2404.01591/) (2024)
- [SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language Models](https://mlsanity.com/paper/2305.15033/) (2023)
- [LibraGrad: Balancing Gradient Flow for Universally Better Vision Transformer Attributions](https://mlsanity.com/paper/2411.16760/) (2024)
- [Deploying Foundation Model Powered Agent Services: A Survey](https://mlsanity.com/paper/2412.13437/) (2024)
- [Elastic On-Device LLM Service](https://mlsanity.com/paper/2409.09071/) (2024)
- [Finch: Prompt-guided Key-Value Cache Compression](https://mlsanity.com/paper/2408.00167/) (2024)

## Abstract (arXiv)

Pre-trained language models have shown stellar performance in various downstream tasks. But, this usually comes at the cost of high latency and computation, hindering their usage in resource-limited settings. In this work, we propose a novel approach for reducing the computational cost of BERT with minimal loss in downstream performance. Our method dynamically eliminates less contributing tokens through layers, resulting in shorter lengths and consequently lower computational cost. To determine the importance of each token representation, we train a Contribution Predictor for each layer using a gradient-based saliency method. Our experiments on several diverse classification tasks show speedups up to 22x during inference time without much sacrifice in performance. We also validate the quality of the selected tokens in our method using human annotations in the ERASER benchmark. In comparison to other widely used strategies for selecting important tokens, such as saliency and attention, our proposed method has a significantly lower false positive rate in generating rationales. Our code is freely available at https://github.com/amodaresi/AdapLeR .