mlsanityData to 7 Oct 2026

Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selection

arXiv 2203.14380 ↗First posted 27 Mar 2022cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Transformer-based language models such as BERT have achieved the state-of-the-art performance on various NLP tasks, but are computationally prohibitive. A recent line of works use various heuristics to successively shorten sequence length while transforming tokens through encoders, in tasks such as classification and ranking that require a single token embedding for prediction. We present a novel solution to this problem, called Pyramid-BERT where we replace previously used heuristics with a em core-set based token selection method justified by theoretical results. The core-set based token selection technique allows us to avoid expensive pre-training, gives a space-efficient fine tuning, and thus makes it suitable to handle longer sequence lengths. We provide extensive experiments establishing advantages of pyramid BERT over several baselines and existing works on the GLUE benchmarks and Long Range Arena datasets.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 43 of its 52 references that are in the map. First posted on arXiv 27 Mar 2022.

Builds on

43 of 52 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2020Reformer: The Efficient Transformercited by 1,658
  9. 2020Rethinking Attention with Performerscited by 1,584
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
31 more
  1. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  2. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  3. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  4. 2019Are Sixteen Heads Really Better than One?cited by 803
  5. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  6. 2018Universal Transformerscited by 613
  7. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  8. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  9. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  10. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  11. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  12. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  13. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  14. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  15. 2019Structured Pruning of Large Language Modelscited by 201
  16. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  17. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  18. 2019Blockwise Self-Attention for Long Document Understandingcited by 158
  19. 2019End-to-End Open-Domain Question Answering with BERTserinicited by 152
  20. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  21. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  22. 2018Linguistically-Informed Self-Attention for Semantic Role Labelingcited by 71
  23. 2019Visualizing and Understanding the Effectiveness of BERTcited by 67
  24. 2019Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Modelcited by 58
  25. 2018Generalization Error in Deep Learningcited by 46
  26. 2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inferencecited by 40
  27. 2019BP-Transformer: Modelling Long-Range Context via Binary Partitioningcited by 38
  28. 2020ETC: Encoding Long and Structured Inputs in Transformerscited by 34
  29. 2021Centroid Transformers: Learning to Abstract with Attentioncited by 20
  30. 2019Structured Pruning of a BERT-based Question Answering Modelcited by 16
  31. 2021Not All Attention Is All You Needcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 6; 2025: 2; 2026: 2.

13 papers.

Sort:

By area:

  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  2. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
  3. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  4. 2024Efficient Large Multi-modal Models via Visual Context Compressioncited by 46
  5. 2024Rethinking Token Reduction for State Space Modelscited by 14
  6. 2025Compressor-VLA: Instruction-Guided Visual Token Compression for Efficient Robotic Manipulationcited by 7
  7. 2023Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokenscited by 6
  8. 2023DiffusionNER: Boundary Diffusion for Named Entity Recognitioncited by 5
  9. 2024Finch: Prompt-guided Key-Value Cache Compressioncited by 2
  10. 2026When LLaVA Meets Objects: Token Composition for Vision-Language-Modelscited by 2
  11. 2024Self-Satisfied: An end-to-end framework for SAT generation and predictioncited by 1
  12. 2024S2-Attention: Hardware-Aware Context Sharding Among Attention Headscited by 0
1 more
  1. 2026A Diversity Diet for a Healthier Model: A Case Study of French ModernBERTcited by 0