mlsanityData to 14 Oct 2026

SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization

arXiv 2203.06569 ↗First posted 13 Mar 2022cs.CLcited by 61

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Sequence-to-sequence neural networks have recently achieved great success in abstractive summarization, especially through fine-tuning large pre-trained language models on the downstream dataset. These models are typically decoded with beam search to generate a unique summary. However, the search space is very large, and with the exposure bias, such decoding is not optimal. In this paper, we show that it is possible to directly train a second-stage model performing re-ranking on a set of summary candidates. Our mixture-of-experts SummaReranker learns to select a better candidate and consistently improves the performance of the base model. With a base PEGASUS, we push ROUGE scores by 5.44% on CNN-DailyMail (47.16 ROUGE-1), 1.31% on XSum (48.12 ROUGE-1) and 9.34% on Reddit TIFU (29.83 ROUGE-1), reaching a new state-of-the-art. Our code and checkpoints will be available at https://github.com/ntunlp/SummaReranker.

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 36 of its 66 references that are in the map. First posted on arXiv 13 Mar 2022.

Builds on

36 of 66 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  8. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  9. 2018Hierarchical Neural Story Generationcited by 1,193
  10. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  11. 2015Teaching Machines to Read and Comprehendcited by 1,049
  12. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
23 more
  1. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  2. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  3. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  4. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  5. 2019Passage Re-ranking with BERTcited by 645
  6. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  7. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  8. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  9. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 301
  10. 2019Pareto Multi-Task Learningcited by 224
  11. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  12. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
  13. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 132
  14. 2020GSum: A General Framework for Guided Neural Abstractive Summarizationcited by 129
  15. 2018Abstractive Summarization of Reddit Posts with Multi-level Memory Networkscited by 102
  16. 2020Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Modelscited by 44
  17. 2021ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generationcited by 31
  18. 2021RefSum: Refactoring Neural Summarizationcited by 25
  19. 2021Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generationcited by 19
  20. 2018Adaptive Document Retrieval for Deep Question Answeringcited by 16
  21. 2019RankQA: Neural Question Answering with Answer Re-Rankingcited by 15
  22. 2021Evaluating the Efficacy of Summarization Evaluation across Languagescited by 14
  23. 2021Alleviating Exposure Bias via Contrastive Learning for Abstractive Text Summarizationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 12; 2023: 22; 2024: 14; 2025: 11; 2026: 2.

61 papers.

Sort:

By area:

  1. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  2. 2023Large Language Model Routing with Benchmark Datasetscited by 172
  3. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  4. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  5. 2023Lift Yourself Up: Retrieval-augmented Text Generation with Self Memorycited by 103
  6. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 102
  7. 2025Universal Model Routing for Efficient LLM Inferencecited by 86
  8. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 61
  9. 2023Extractive Summarization via ChatGPT for Faithful Summary Generationcited by 59
  10. 2023Detecting and Mitigating Hallucinations in Multilingual Summarisationcited by 48
  11. 2023NoteChat: A Dataset of Synthetic Doctor-Patient Conversations Conditioned on Clinical Notescited by 48
  12. 2023Fusing Models with Complementary Expertisecited by 46
38 more
  1. 2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 46
  2. 2022Prompted Opinion Summarization with GPT-3.5cited by 44
  3. 2024Routoo: Learning to Route to Large Language Models Effectivelycited by 34
  4. 2023Faithfulness-Aware Decoding Strategies for Abstractive Summarizationcited by 31
  5. 2024Smoothie: Label Free Language Model Routingcited by 31
  6. 2024Metasql: A Generate-then-Rank Framework for Natural Language to SQL Translationcited by 17
  7. 2022MVP: Multi-task Supervised Pre-training for Natural Language Generationcited by 16
  8. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  9. 2024PickLLM: Context-Aware RL-Assisted Large Language Model Routingcited by 13
  10. 2024Cool-Fusion: Fuse Large Language Models without Trainingcited by 12
  11. 2023Lightweight reranking for language model generationscited by 10
  12. 2023Generating EDU Extracts for Plan-Guided Summary Re-Rankingcited by 9
  13. 2023Towards Argument-Aware Abstractive Summarization of Long Legal Opinions with Summary Rerankingcited by 9
  14. 2025Ensemble Learning for Large Language Models in Text and Code Generation: A Surveycited by 9
  15. 2022Towards Summary Candidates Fusioncited by 8
  16. 2023Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyondcited by 8
  17. 2023Cross-lingual Cross-temporal Summarization: Dataset, Models, Evaluationcited by 8
  18. 2024SelectLLM: Query-Aware Efficient Selection Algorithm for Large Language Modelscited by 8
  19. 2024Flexible and Adaptable Summarization via Expertise Separationcited by 7
  20. 2022Best-k Search Algorithm for Neural Text Generationcited by 6
  21. 2022DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarizationcited by 6
  22. 2023FactReranker: Fact-guided Reranker for Faithful Radiology Report Summarizationcited by 6
  23. 2023Cappy: Outperforming and Boosting Large Multi-Task LMs with a Small Scorercited by 5
  24. 2022Joint Generator-Ranker Learning for Natural Language Generationcited by 4
  25. 2022Revision for Concision: A Constrained Paraphrase Generation Taskcited by 4
  26. 2023EEL: Efficiently Encoding Lattices for Rerankingcited by 4
  27. 2023Calibrating Likelihoods towards Consistency in Summarization Modelscited by 4
  28. 2022Improving abstractive summarization with energy-based re-rankingcited by 3
  29. 2024Prompt-tuning for Clickbait Detection via Text Summarizationcited by 3
  30. 2025Poetry in Pixels: Prompt Tuning for Poem Image Generation via Diffusion Modelscited by 3
  31. 2025Wisdom and Delusion of LLM Ensembles for Code Generation and Repaircited by 3
  32. 2022Lexical Complexity Controlled Sentence Generationcited by 2
  33. 2022PairReranker: Pairwise Reranking for Natural Language Generationcited by 2
  34. 2023Domain-Agnostic Neural Architecture for Class Incremental Continual Learning in Document Processing Platformcited by 2
  35. 2024Disentangling Instructive Information from Ranked Multiple Candidates for Multi-Document Scientific Summarizationcited by 2
  36. 2024A Bayesian Optimization Approach to Machine Translation Rerankingcited by 2
  37. 2025Routesplain: Towards Faithful and Intervenable Routing for Software-related Taskscited by 2
  38. 2023Balancing Lexical and Semantic Quality in Abstractive Summarizationcited by 1

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021HydraSum: Disentangling Stylistic Features in Text Summarization using Multi-Decoder Models
  2. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization
  3. 2023SPEC: Summary Preference Decomposition for Low-Resource Abstractive Summarization
  4. 2020Learning to summarize from human feedback
  5. 2026FlexMoRE: A Flexible Mixture of Rank-heterogeneous Experts for Efficient Federatedly-trained Large Language Models
  6. 2019Mixture Content Selection for Diverse Sequence Generation
  7. 2022Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation
  8. 2024Model-based Preference Optimization in Abstractive Summarization without Human Feedback