Reranking-based Generation for Unbiased Perspective Summarization
Not on the map. Why?
Abstract (arXiv)
Generating unbiased summaries in real-world settings such as political perspective summarization remains a crucial application of Large Language Models (LLMs). Yet, existing evaluation frameworks rely on traditional metrics for measuring key attributes such as coverage and faithfulness without verifying their applicability, and efforts to develop improved summarizers are still nascent. We address these gaps by (1) identifying reliable metrics for measuring perspective summary quality, and (2) investigating the efficacy of LLM-based methods beyond zero-shot inference. Namely, we build a test set for benchmarking metric reliability using human annotations and show that traditional metrics underperform compared to language model-based metrics, which prove to be strong evaluators. Using these metrics, we show that reranking-based methods yield strong results, and preference tuning with synthetically generated and reranking-labeled data further boosts performance. Our findings aim to contribute to the reliable evaluation and development of perspective summarization methods.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 63 of its 78 references that are in the map. First posted on arXiv 19 Jun 2025.
Builds on
63 of 78 references.
References · 63
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
- 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
- 2020Learning to summarize from human feedbackcited by 2,588
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
38 more
- 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
- 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
- 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
- 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
- 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
- 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
- 2022Large Language Models Can Self-Improvecited by 664
- 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
- 2023Benchmarking Large Language Models for News Summarizationcited by 445
- 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
- 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
- 2022News Summarization and Evaluation in the Era of GPT-3cited by 327
- 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
- 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 294
- 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 293
- 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 288
- 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 257
- 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
- 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
- 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 113
- 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 106
- 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 101
- 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
- 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
- 2022Parallel Context Windows for Large Language Modelscited by 81
- 2023Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Methodcited by 76
- 2023On Measuring Faithfulness or Self-consistency of Natural Language Explanationscited by 73
- 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 72
- 2019Unsupervised Opinion Summarization as Copycat-Review Generationcited by 68
- 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 65
- 2021Improving Factual Consistency of Abstractive Summarization via Question Answeringcited by 55
- 2021Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarizationcited by 51
- 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarizationcited by 50
- 2022Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Modelscited by 49
- 2021Aspect-Controllable Opinion Summarizationcited by 47
- 2023Multi-Dimensional Evaluation of Text Summarization with In-Context Learningcited by 44
- 2021Finding a Balanced Degree of Automation for Summary Evaluationcited by 43
- 2023LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedbackcited by 39
50 most cited of 63 All 63 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.