mlsanityData to 7 Oct 2026

Efficient Transformers with Dynamic Token Pooling

arXiv 2211.09761 ↗First posted 17 Nov 2022cs.CLcited by 75

Not on the map. Why?

Abstract (arXiv)

Transformers achieve unrivalled performance in modelling language, but remain inefficient in terms of memory and time complexity. A possible remedy is to reduce the sequence length in the intermediate layers by pooling fixed-length segments of tokens. Nevertheless, natural units of meaning, such as words or phrases, display varying sizes. To address this mismatch, we equip language models with a dynamic-pooling mechanism, which predicts segment boundaries in an autoregressive fashion. We compare several methods to infer boundaries, including end-to-end learning through stochastic re-parameterisation, supervised learning (based on segmentations from subword tokenizers or spikes in conditional entropy), as well as linguistically motivated boundaries. We perform character-level evaluation on texts from multiple datasets and morphologically diverse languages. The results demonstrate that dynamic pooling, which jointly segments and models language, is both faster and more accurate than vanilla Transformers and fixed-length pooling within the same computational budget.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Builds on 45 of its 55 references that are in the map. First posted on arXiv 17 Nov 2022.

Builds on

45 of 55 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  11. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
33 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2020Rethinking Attention with Performerscited by 1,584
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  5. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  6. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  7. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  8. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  9. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  10. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  11. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  12. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  13. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  14. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  15. 2022Memorizing Transformerscited by 219
  16. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  17. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  18. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  19. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  20. 2015Gradient Estimation Using Stochastic Computation Graphscited by 120
  21. 2016Hierarchical Multiscale Recurrent Neural Networkscited by 102
  22. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 99
  23. 2014A Clockwork RNNcited by 70
  24. 2021Combiner: Full Attention Transformer with Sparse Computation Costcited by 64
  25. 2021Revisiting the Uniform Information Density Hypothesiscited by 63
  26. 2016Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activationscited by 54
  27. 2017Skip RNN: Learning to Skip State Updates in Recurrent Neural Networkscited by 49
  28. 2020Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentationcited by 43
  29. 2021Segmental Contrastive Predictive Coding for Unsupervised Word Segmentationcited by 32
  30. 2017Neural Speed Reading via Skim-RNNcited by 14
  31. 2022Variable-rate hierarchical CPC leads to acoustic unit discovery in speechcited by 13
  32. 2017Fast-Slow Recurrent Neural Networkscited by 12
  33. 2016Surprisal-Driven Zoneoutcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 18; 2025: 28; 2026: 20.

75 papers.

Sort:

By area:

  1. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
  2. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  3. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  4. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  5. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  6. 2025SuperBPE: Space Travel for Language Modelscited by 58
  7. 2023PuMer: Pruning and Merging Tokens for Efficient Vision Language Modelscited by 53
  8. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  9. 2024Efficient Large Multi-modal Models via Visual Context Compressioncited by 46
  10. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  11. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  12. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 40
38 more
  1. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 34
  2. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  3. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 31
  4. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  5. 2024Zero-Shot Tokenizer Transfercited by 30
  6. 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 27
  7. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  8. 2024The Foundations of Tokenization: Statistical and Computational Concernscited by 20
  9. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 19
  10. 2024Understanding and Mitigating Tokenization Bias in Language Modelscited by 18
  11. 2025Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizationscited by 16
  12. 2025LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Mergingcited by 16
  13. 2024Retrofitting Large Language Models with Dynamic Tokenizationcited by 15
  14. 2025From Bytes to Ideas: Language Modeling with Autoregressive U-Netscited by 15
  15. 2024Rethinking Token Reduction for State Space Modelscited by 14
  16. 2025Fast and Expressive Multi-Byte Prediction with Probabilistic Circuitscited by 9
  17. 2025FLEXITOKENS: Flexible Tokenization for Evolving Language Modelscited by 8
  18. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
  19. 2023nanoT5: A PyTorch Framework for Pre-training and Fine-tuning T5-style Models with Limited Resourcescited by 7
  20. 2023Toucan: Token-Aware Character Level Language Modelingcited by 7
  21. 2025Sampling from Your Language Model One Byte at a Timecited by 7
  22. 2025Delta-LLaVA: Base-then-Specialize Alignment for Token-Efficient Vision-Language Modelscited by 7
  23. 2023Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokenscited by 6
  24. 2025StochasTok: Improving Fine-Grained Subword Understanding in LLMscited by 6
  25. 2026Cross-Tokenizer LLM Distillation through a Byte-Level Interfacecited by 6
  26. 2026Breaking the Tokenizer Barrier: On-Policy Distillation across Model Familiescited by 5
  27. 2025TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behaviorcited by 4
  28. 2023Hierarchical Attention Encoder Decodercited by 3
  29. 2025zip2zip: Inference-Time Adaptive Tokenization via Online Compressioncited by 3
  30. 2025Learning Compact Vision Tokens for Efficient Large Multimodal Modelscited by 3
  31. 2026Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenizationcited by 3
  32. 2026Proxy Compression for Language Modelingcited by 3
  33. 2023StrAE: Autoencoding for Pre-Trained Embeddings using Explicit Structurecited by 2
  34. 2025MedPatch: Confidence-Guided Multi-Stage Fusion for Multimodal Clinical Datacited by 2
  35. 2025Deconstructing Attention: Investigating Design Principles for Effective Language Modelingcited by 2
  36. 2025Controllably Efficient Language Modelscited by 2
  37. 2025MergeDNA: Context-aware Genome Modeling with Dynamic Tokenization through Token Mergingcited by 2
  38. 2025DPAR: Dynamic Patchification for Efficient Autoregressive Visual Generationcited by 2

50 most cited of 75 All 75 (JSON)