Data to 7 Oct 2026

Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing

arXiv 2006.03236 ↗First posted 5 Jun 2020cs.LG cs.CL stat.MLcited by 161

Not on the map. Why?

Abstract (arXiv)

With the success of language pretraining, it is highly desirable to develop more efficient architectures of good scalability that can exploit the abundant unlabeled data at a lower cost. To improve the efficiency, we examine the much-overlooked redundancy in maintaining a full-length token-level presentation, especially for tasks that only require a single-vector presentation of the sequence. With this intuition, we propose Funnel-Transformer which gradually compresses the sequence of hidden states to a shorter one and hence reduces the computation cost. More importantly, by re-investing the saved FLOPs from length reduction in constructing a deeper or wider model, we further improve the model capacity. In addition, to perform token-level predictions as required by common pretraining objectives, Funnel-Transformer is able to recover a deep representation for each token from the reduced hidden sequence via a decoder. Empirically, with comparable or fewer FLOPs, Funnel-Transformer outperforms the standard Transformer on a wide variety of sequence-level prediction tasks, including text classification, language understanding, and reading comprehension. The code and pretrained checkpoints are available at https://github.com/laiguokun/Funnel-Transformer.

Key facts. Cited by 161 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 30; 2022: 31; 2023: 34; 2024: 36; 2025: 20; 2026: 10.

161 papers.

Sort:

By area:

2021Multiscale Vision Transformerscited by 896
2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
2021A Survey of Transformerscited by 399
2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
2022Long Range Language Modeling via Gated State Spacescited by 265
2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
2024A Survey on Efficient Inference for Large Language Modelscited by 197
2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
2023Efficient Large Language Models: A Surveycited by 183
2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158

38 more

2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
2022Block-Recurrent Transformerscited by 107
2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
2021Hierarchical Transformers Are More Efficient Language Modelscited by 99
2024Efficient Multimodal Large Language Models: A Surveycited by 95
2021Finetuning Pretrained Transformers into RNNscited by 94
2021Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimationcited by 94
2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
2021Scalable Vision Transformers with Hierarchical Poolingcited by 84
2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
2022Parameter-efficient Model Adaptation for Vision Transformerscited by 81
2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
2022Efficient Transformers with Dynamic Token Poolingcited by 75
2021A Practical Survey on Faster and Lighter Transformerscited by 73
2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
2022Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting--Full Versioncited by 59
2023How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Taskscited by 58
2024MambaByte: Token-free Selective State Space Modelcited by 58
2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
2021Packed Levitated Marker for Entity and Relation Extractioncited by 49
2024Efficient Large Multi-modal Models via Visual Context Compressioncited by 46
2022Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Videocited by 43
2022Token Dropping for Efficient BERT Pretrainingcited by 42
2024SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Foldingcited by 42
2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inferencecited by 40
2021From block-Toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked Transformerscited by 38
2021Your fairness may vary: Pretrained language model fairness in toxic text classificationcited by 38
2021A Simple and Effective Positional Encoding for Transformerscited by 37
2021Efficient Transformers in Reinforcement Learning using Actor-Learner Distillationcited by 35
2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 32
2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
2022Linear Complexity Randomized Self-attention Mechanismcited by 31
2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 31
2021The Surprising Performance of Simple Baselines for Misinformation Detectioncited by 27
2021How could Neural Networks understand Programs?cited by 26

50 most cited of 161 All 161 (JSON)