Input Order Shapes LLM Semantic Alignment in Multi-Document Summarization
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) are now used in settings such as Google's AI Overviews, where it summarizes multiple long documents. However, it remains unclear whether they weight all inputs equally. Focusing on abortion-related news, we construct 40 pro-neutral-con article triplets, permute each triplet into six input orders, and prompt Gemini 2.5 Flash to generate a neutral overview. We evaluate each summary against its source articles using ROUGE-L (lexical overlap), BERTScore (semantic similarity), and SummaC (factual consistency). One-way ANOVA reveals a significant primacy effect for BERTScore across all stances, indicating that summaries are more semantically aligned with the first-seen article. Pairwise comparisons further show that Position 1 differs significantly from Positions 2 and 3, while the latter two do not differ from each other, confirming a selective preference for the first document. The findings present risks for applications that rely on LLM-generated overviews and for agentic AI systems, where the steps involving LLMs can disproportionately influence downstream actions.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 15 of its 21 references that are in the map. First posted on arXiv 2 Dec 2025.
Builds on
15 of 21 references.
References · 15
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2023Whose Opinions Do Language Models Reflect?cited by 759
- 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
- 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
- 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
- 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 167
- 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
- 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 80
- 2023Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Methodcited by 76
- 2019Earlier Isn't Always Better: Sub-aspect Analysis on Corpus and System Biases in Summarizationcited by 28
3 more
- 2024Revisiting Zero-Shot Abstractive Summarization in the Era of Large Language Models from the Perspective of Position Biascited by 28
- 2024When Neutral Summaries are not that Neutral: Quantifying Political Neutrality in LLM-Generated News Summariescited by 7
- 2024Understanding Position Bias Effects on Fairness in Social Multi-Document Summarizationcited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2022Methodological reflections for AI alignment research using human feedback
- 2025Multi-round, Chain-of-thought Post-editing for Unfaithful Summaries
- 2023Evaluating Factual Consistency of Summaries with Large Language Models
- 2026A Stepwise Questioning Expert-Editor Multi-Agent Framework for Long-Document Summarization
- 2023Aligning Large Language Models with Human: A Survey
- 2026A Training-Free Mixture-of-Agents Framework for Multi-Document Summarization using LLMs and Knowledge Graphs
- 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization
- 2025Multi2: Multi-Agent Test-Time Scalable Framework for Multi-Document Processing