mlsanityData to 7 Oct 2026

Poolingformer: Long Document Modeling with Pooling Attention

arXiv 2105.04371 ↗First posted 10 May 2021cs.CLcited by 60

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce a two-level attention schema, Poolingformer, for long document modeling. Its first level uses a smaller sliding window pattern to aggregate information from neighbors. Its second level employs a larger window to increase receptive fields with pooling attention to reduce both computational cost and memory consumption. We first evaluate Poolingformer on two long sequence QA tasks: the monolingual NQ and the multilingual TyDi QA. Experimental results show that Poolingformer sits atop three official leaderboards measured by F1, outperforming previous state-of-the-art models by 1.9 points (79.8 vs. 77.9) on NQ long answer, 1.9 points (79.5 vs. 77.6) on TyDi QA passage answer, and 1.6 points (67.6 vs. 66.0) on TyDi QA minimal answer. We further evaluate Poolingformer on a long sequence summarization task. Experimental results on the arXiv benchmark continue to demonstrate its superior performance.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 39 of its 44 references that are in the map. First posted on arXiv 10 May 2021.

Builds on

39 of 44 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2020Reformer: The Efficient Transformercited by 1,658
  12. 2020Rethinking Attention with Performerscited by 1,584
27 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  4. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  5. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  6. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  7. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  8. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  9. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  10. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  11. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  12. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 221
  13. 2016A Decomposable Attention Model for Natural Language Inferencecited by 214
  14. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  15. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  16. 2019Blockwise Self-Attention for Long Document Understandingcited by 158
  17. 2019Hierarchical Transformers for Multi-Document Summarizationcited by 132
  18. 2017Simple and Effective Multi-Paragraph Reading Comprehensioncited by 120
  19. 2019Star-Transformercited by 94
  20. 2018Document-Level Neural Machine Translation with Hierarchical Attention Networkscited by 85
  21. 2019On Extractive and Abstractive Neural Document Summarization with Transformer Language Modelscited by 83
  22. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  23. 2019BP-Transformer: Modelling Long-Range Context via Binary Partitioningcited by 38
  24. 2019A BERT Baseline for the Natural Questionscited by 33
  25. 2020RikiNet: Reading Wikipedia Pages for Natural Question Answeringcited by 21
  26. 2020Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encodingcited by 19
  27. 2020No Answer is Better Than Wrong Answer: A Reflection Model for Document Level Machine Reading Comprehensioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 12; 2023: 19; 2024: 14; 2025: 5; 2026: 2.

60 papers.

Sort:

By area:

  1. 2021A Survey of Transformerscited by 399
  2. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  3. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 235
  4. 2023Efficient Large Language Models: A Surveycited by 183
  5. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  6. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Modelscited by 78
  7. 2021DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarizationcited by 72
  8. 2021Fastformer: Additive Attention Can Be All You Needcited by 61
  9. 2022Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generationcited by 48
  10. 2021Long-Range Transformers for Dynamic Spatiotemporal Forecastingcited by 46
  11. 2024Paths-over-Graph: Knowledge Graph Empowered Large Language Model Reasoningcited by 44
  12. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
38 more
  1. 2022Adapting Pretrained Text-to-Text Models for Long Text Sequencescited by 21
  2. 2024HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processingcited by 20
  3. 2022Trial2Vec: Zero-Shot Clinical Trial Document Similarity Search using Self-Supervisioncited by 18
  4. 2023TextGenSHAP: Scalable Post-hoc Explanations in Text Generation with Long Documentscited by 17
  5. 2023Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Modelscited by 16
  6. 2024TrafficGPT: Breaking the Token Barrier for Efficient Long Traffic Analysis and Generationcited by 15
  7. 2021On Learning the Transformer Kernelcited by 14
  8. 2023SummaryMixing: A Linear-Complexity Alternative to Self-Attention for Speech Recognition and Understandingcited by 11
  9. 2022CAB: Comprehensive Attention Benchmarking on Long Sequence Modelingcited by 10
  10. 2022APOLLO: An Optimized Training Approach for Long-form Numerical Reasoningcited by 8
  11. 2023Sparse Binary Transformers for Multivariate Time Series Modelingcited by 8
  12. 2023Plug-and-Play Document Modules for Pre-trained Modelscited by 6
  13. 2024Orchid: Flexible and Data-Dependent Convolution for Sequence Modelingcited by 6
  14. 2021PoNet: Pooling Network for Efficient Token Mixing in Long Sequencescited by 5
  15. 2021Do Answers to Boolean Questions Need Explanations? Yescited by 5
  16. 2022Rethinking Attention Mechanism in Time Series Classificationcited by 5
  17. 2023Noisy Pair Corrector for Dense Retrievalcited by 5
  18. 2024Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequencescited by 5
  19. 2023Dynamic Multi-Scale Context Aggregation for Conversational Aspect-Based Sentiment Quadruple Analysiscited by 4
  20. 2023Linear Attention via Orthogonal Memorycited by 4
  21. 2024LongVQ: Long Sequence Modeling with Vector Quantization on Structured Memorycited by 4
  22. 2024Target conversation extraction: Source separation using turn-taking dynamicscited by 4
  23. 2021Smart Bird: Learnable Sparse Attention for Efficient and Effective Transformercited by 3
  24. 2022DBA: Efficient Transformer with Dynamic Bilinear Low-Rank Attentioncited by 3
  25. 2023Modeling Sequential Sentence Relation to Improve Cross-lingual Dense Retrievalcited by 3
  26. 2023A Multilingual Translator to SQL with Database Schema Pruning to Improve Self-Attentioncited by 3
  27. 2023ConvFormer: Revisiting Transformer for Sequential User Modelingcited by 3
  28. 2024UniMem: Towards a Unified View of Long-Context Large Language Modelscited by 3
  29. 2022DialoGen: Generalized Long-Range Context Representation for Dialogue Systemscited by 2
  30. 2022Convolution-enhanced Evolving Attention Networkscited by 2
  31. 2023Global Structure Knowledge-Guided Relation Extraction Method for Visually-Rich Documentcited by 2
  32. 2024NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documentscited by 2
  33. 2022GAAMA 2.0: An Integrated System that Answers Boolean and Extractive Questionscited by 1
  34. 2024GOVERN: Gradient Orientation Vote Ensemble for Multi-Teacher Reinforced Distillationcited by 1
  35. 2025Long document summarization using page specific target text alignment and distilling page importancecited by 1
  36. 2022Linear Self-Attention Approximation via Trainable Feedforward Kernelcited by 0
  37. 2023Attention Link: An Efficient Attention-Based Low Resource Machine Translation Architecturecited by 0
  38. 2023A Quantitative Review on Language Model Efficiency Researchcited by 0

50 most cited of 60 All 60 (JSON)