mlsanityData to 14 Oct 2026

PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization

arXiv 2110.08499 ↗First posted 16 Oct 2021cs.CLcited by 62

Not on the map. Why?

Abstract (arXiv)

We introduce PRIMERA, a pre-trained model for multi-document representation with a focus on summarization that reduces the need for dataset-specific architectures and large amounts of fine-tuning labeled data. PRIMERA uses our newly proposed pre-training objective designed to teach the model to connect and aggregate information across documents. It also uses efficient encoder-decoder transformers to simplify the processing of concatenated input documents. With extensive experiments on 6 multi-document summarization datasets from 3 different domains on zero-shot, few-shot and full-supervised settings, PRIMERA outperforms current state-of-the-art dataset-specific and pre-trained models on most of these settings with large margins. The code and pre-trained models can be found at urlhttps://github.com/allenai/PRIMER.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Builds on 24 of its 46 references that are in the map. First posted on arXiv 16 Oct 2021.

Builds on

24 of 46 references.

References · 24

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  5. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  6. 2019Text Summarization with Pretrained Encoderscited by 556
  7. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  8. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  9. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  10. 2018Bottom-Up Abstractive Summarizationcited by 177
  11. 2019Hierarchical Transformers for Multi-Document Summarizationcited by 132
  12. 2020Multi-XScience: A Large-scale Dataset for Extreme Multi-document Summarization of Scientific Articlescited by 97
12 more
  1. 2020A Large-Scale Multi-Document Summarization Dataset from the Wikipedia Current Events Portalcited by 72
  2. 2020Leveraging Graph to Improve Abstractive Multi-Document Summarizationcited by 67
  3. 2021FLEX: Unifying Evaluation for Few-Shot NLPcited by 67
  4. 2021HTLM: Hyper-Text Pre-Training and Prompting of Language Modelscited by 58
  5. 2017Graph-based Neural Multi-Document Summarizationcited by 54
  6. 2020PMI-Masking: Principled masking of correlated spanscited by 54
  7. 2020Generating Representative Headlines for News Storiescited by 39
  8. 2018Abstract Meaning Representation for Multi-Document Summarizationcited by 35
  9. 2020Multi-document Summarization with Maximal Marginal Relevance-guided Reinforcement Learningcited by 28
  10. 2021Controllable Abstractive Dialogue Summarization with Sketch Supervisioncited by 27
  11. 2020DynE: Dynamic Ensemble Decoding for Multi-Document Summarizationcited by 8
  12. 2021PoBRL: Optimizing Multi-Document Summarization by Blending Reinforcement Learning Policiescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 12; 2023: 29; 2024: 11; 2025: 6; 2026: 4.

62 papers.

Sort:

By area:

  1. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  2. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 128
  3. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  4. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 102
  5. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  6. 2022Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularitiescited by 47
  7. 2023A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Taskscited by 36
  8. 2022MOPRD: A multidisciplinary open peer review datasetcited by 32
  9. 2023Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answeringcited by 28
  10. 2023Summarizing Multiple Documents with Conversational Structure for Meta-Review Generationcited by 27
  11. 2024Closing the gap between open-source and commercial large language models for medical evidence summarizationcited by 21
  12. 2023Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluationscited by 18
38 more
  1. 2022Socratic Pretraining: Question-Driven Pretraining for Controllable Summarizationcited by 17
  2. 2024A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industrycited by 17
  3. 2022LSG Attention: Extrapolation of pretrained Transformers to long sequencescited by 16
  4. 2023Towards Unifying Multi-Lingual and Cross-Lingual Summarizationcited by 15
  5. 2022Summary-Oriented Vision Modeling for Multimodal Abstractive Summarizationcited by 14
  6. 2024LitLLMs, LLMs for Literature Review: Are we there yet?cited by 14
  7. 2023Bringing order into the realm of Transformer-based language models for artificial intelligence and lawcited by 13
  8. 2022Entity-based SpanCopy for Abstractive Summarization to Improve the Factual Consistencycited by 11
  9. 2023Do Multi-Document Summarization Models Synthesize?cited by 11
  10. 2022Leveraging Locality in Abstractive Text Summarizationcited by 9
  11. 2023Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformerscited by 9
  12. 2022How "Multi" is Multi-Document Summarization?cited by 8
  13. 2023What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarizationcited by 8
  14. 2024SEAM: A Stochastic Benchmark for Multi-Document Taskscited by 8
  15. 2023PDSum: Prototype-driven Continuous Summarization of Evolving Multi-document Sets Streamcited by 7
  16. 2023LLM Based Multi-Document Summarization Exploiting Main-Event Biased Monotone Submodular Content Extractioncited by 7
  17. 2023Multi-Stage Pre-training Enhanced by ChatGPT for Multi-Scenario Multi-Domain Dialogue Summarizationcited by 7
  18. 2025Can LLMs Automate Fact-Checking Article Writing?cited by 7
  19. 2025An Evaluation of Large Language Models on Text Summarization Tasks Using Prompt Engineering Techniquescited by 7
  20. 2023The Current State of Summarizationcited by 6
  21. 2023Revisiting Sentence Union Generation as a Testbed for Text Consolidationcited by 6
  22. 2023Overview of the BioLaySumm 2023 Shared Task on Lay Summarization of Biomedical Research Articlescited by 6
  23. 2022Unsupervised Multi-Granularity Summarizationcited by 5
  24. 2022Multi-Document Summarization with Centroid-Based Pretrainingcited by 5
  25. 2024Understanding Position Bias Effects on Fairness in Social Multi-Document Summarizationcited by 4
  26. 2023Compressed Heterogeneous Graph for Abstractive Multi-Document Summarizationcited by 3
  27. 2023Named Entity Inclusion in Abstractive Text Summarizationcited by 3
  28. 2023Multi-document Summarization: A Comparative Evaluationcited by 3
  29. 2023ODSum: New Benchmarks for Open Domain Multi-Document Summarizationcited by 3
  30. 2023Ascle: A Python Natural Language Processing Toolkit for Medical Text Generationcited by 3
  31. 2024M2DS: Multilingual Dataset for Multi-document Summarisationcited by 3
  32. 2025MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidancecited by 3
  33. 2022LED down the rabbit hole: exploring the potential of global attention for biomedical multi-document summarisationcited by 2
  34. 2023Summarizing Indian Languages using Multilingual Transformers based Modelscited by 2
  35. 2023A Hierarchical Encoding-Decoding Scheme for Abstractive Multi-document Summarizationcited by 2
  36. 2023Unsupervised Multi-document Summarization with Holistic Inferencecited by 2
  37. 2023Fovea Transformer: Efficient Long-Context Modeling with Structured Fine-to-Coarse Attentioncited by 2
  38. 2023Controllable Multi-document Summarization: Coverage & Coherence Intuitive Policy with Large Language Model Based Rewardscited by 1

50 most cited of 62 All 62 (JSON)