# When Evidence Conflicts: Reliability-aware Meta-review Generation

arXiv: https://arxiv.org/abs/2609.24028 · HTML: https://mlsanity.com/paper/2609.24028/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 21 Sep 2026.

## Builds on (most cited first, 12 of 12)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [Extractive Opinion Summarization in Quantized Transformer Spaces](https://mlsanity.com/paper/2012.04443/) (2020)
- [OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews](https://mlsanity.com/paper/2412.11948/) (2024)
- [Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review](https://mlsanity.com/paper/2412.01708/) (2024)
- [SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)](https://mlsanity.com/paper/2304.14803/) (2023)
- [Learning from Crowds by Modeling Common Confusions](https://mlsanity.com/paper/2012.13052/) (2020)
- [Summarizing Multiple Documents with Conversational Structure for Meta-Review Generation](https://mlsanity.com/paper/2305.01498/) (2023)
- [Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP](https://mlsanity.com/paper/2601.09065/) (2026)
- [A Sentiment Consolidation Framework for Meta-Review Generation](https://mlsanity.com/paper/2402.18005/) (2024)
- [Decomposed Opinion Summarization with Verified Aspect-Aware Modules](https://mlsanity.com/paper/2501.17191/) (2025)
- [Aspect-Based Opinion Summarization with Argumentation Schemes](https://mlsanity.com/paper/2506.09917/) (2025)

## Abstract (arXiv)

Generating coherent meta-reviews from multiple peer reviews is challenging when reviewer evidence conflicts and varies in reliability. Existing approaches typically formulate meta-review generation as a multi-document summarization task and aggregate reviewer feedback uniformly, making it difficult to determine which opinions should be prioritized under disagreement. In this paper, we study meta-review generation through reliability-aware evidence aggregation. Our framework first extracts aspect-level opinions from peer reviews and identifies conflicting evidence within each aspect. It then estimates opinion-level support and review-level quality to measure evidence reliability. Based on these signals, the framework assigns reliability-aware weights to reviewer feedback, enabling the generator to prioritize better-supported arguments while preserving diverse perspectives. Experiments demonstrate that our method consistently improves meta-review generation over strong baselines on both automatic and human evaluations, with clear gains in conflict recognition and resolution under high-conflict review scenarios. The code and implementation details are publicly available at https://github.com/Wangxz729/reliability-aware-meta-review.