mlsanityData to 14 Oct 2026

Infor-Coef: Information Bottleneck-based Dynamic Token Downsampling for Compact and Efficient language model

arXiv 2305.12458 ↗First posted 21 May 2023cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

The prevalence of Transformer-based pre-trained language models (PLMs) has led to their wide adoption for various natural language processing tasks. However, their excessive overhead leads to large latency and computational costs. The statically compression methods allocate fixed computation to different samples, resulting in redundant computation. The dynamic token pruning method selectively shortens the sequences but are unable to change the model size and hardly achieve the speedups as static pruning. In this paper, we propose a model accelaration approaches for large language models that incorporates dynamic token downsampling and static pruning, optimized by the information bottleneck loss. Our model, Infor-Coef, achieves an 18x FLOPs speedup with an accuracy degradation of less than 8% compared to BERT. This work provides a promising approach to compress and accelerate transformer-based models for NLP tasks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 31 of its 38 references that are in the map. First posted on arXiv 21 May 2023.

Builds on

31 of 38 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  9. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  11. 2016Deep Variational Information Bottleneckcited by 930
  12. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
19 more
  1. 2019Are Sixteen Heads Really Better than One?cited by 816
  2. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  3. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  4. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  5. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  6. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  7. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  8. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  9. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  10. 2019Structured Pruning of Large Language Modelscited by 203
  11. 2021Block Pruning For Faster Transformerscited by 177
  12. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  13. 2021Learned Token Pruning for Transformerscited by 139
  14. 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
  15. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  16. 2018Compressing Neural Networks using the Variational Information Bottleneckcited by 59
  17. 2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inferencecited by 40
  18. 2019Explaining a black-box using Deep Variational Information Bottleneck Approachcited by 36
  19. 2022Transkimmer: Transformer Learns to Layer-wise Skimcited by 24

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024A Survey of Transformer-based Language Models with Focus on Efficiencycited by 6
  2. 2025Learning What to Remember: Adaptive Probabilistic Memory Retention for Memory-Efficient Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2022Token Dropping for Efficient BERT Pretraining
  2. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models
  3. 2023A Survey on Model Compression for Large Language Models
  4. 2024Model Compression and Efficient Inference for Large Language Models: A Survey
  5. 2024Efficiency optimization of large-scale language models based on deep learning in natural language processing tasks
  6. 2023How to Prune Your Language Model: Recovering Accuracy on the "Sparsity May Cry'' Benchmark
  7. 2023Divergent Token Metrics: Measuring degradation to prune away LLM components -- and optimize quantization
  8. 2025Pruning Laws for Large Language Models