AMEL: Accumulated Message Effects on LLM Judgments
Not on the map. Why?
Abstract (arXiv)
Large language models are routinely used as automated evaluators: to review code, moderate content, or score outputs, often with many items passing through one conversation. We ask whether the polarity of prior conversation history biases subsequent judgments, an effect we call the accumulated message effect on LLM judgments (AMEL). Across 84,088 API calls to 12 models from 5 providers (OpenAI, Anthropic, Google, DeepSeek, and four open-source models), we present identical test items in isolation or following histories saturated with predominantly positive or negative evaluations. Models shift toward the conversation's prevailing polarity (d = -0.17, p < 10^-53). The effect concentrates on items where the model is genuinely uncertain at baseline (d = -0.36 for high-entropy items, vs d = -0.15 when the baseline is deterministic). Bias does not grow with context length: 5 prior turns and 50 produce the same shift (Spearman |r| < 0.01; OLS slope p = 0.80). And there is a negativity asymmetry: paired per item, negative histories induce 1.52x more bias than positive (t = 13.03, p < 10^-36, n = 2,733). Scaling helps but does not solve it (Anthropic: Haiku -0.22 to Opus -0.17; OpenAI: Nano -0.34 to GPT-5.2 -0.17). Three follow-ups narrow the mechanism. The token probability distribution shifts continuously, not at a threshold. The negativity asymmetry has both token-level and semantic components, though attributing the balance is exploratory at our sample sizes. Position does not matter: five biased turns anywhere in a 50-turn history produce the same shift. The simplest fix for evaluation pipelines is a fresh context per item; when batching is unavoidable, balancing the history helps.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 21 May 2026.
Builds on
31 of 42 references.
References · 31
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
- 2023Large Language Models are not Fair Evaluatorscited by 1,085
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
- 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
- 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
- 2024Many-Shot In-Context Learningcited by 206
- 2024Cognitive Bias in Decision-Making with LLMscited by 106
19 more
- 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 104
- 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasonerscited by 73
- 2024Anchoring Bias in Large Language Models: An Experimental Studycited by 37
- 2025Interaction Context Often Increases Sycophancy in LLMscited by 26
- 2026How RLHF Amplifies Sycophancycited by 23
- 2025Drift No More? Context Equilibria in Multi-Turn LLM Interactionscited by 22
- 2024AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance Assessmentcited by 20
- 2024Cognitive Biases in Large Language Models: A Survey and Mitigation Experimentscited by 20
- 2023How Robust are LLMs to In-Context Majority Label Bias?cited by 14
- 2024Correcting Negative Bias in Large Language Models through Negative Attention Score Alignmentcited by 12
- 2024CBEval: A framework for evaluating and interpreting cognitive biases in LLMscited by 12
- 2025Evaluating the Sensitivity of LLMs to Prior Contextcited by 9
- 2026Mitigating Conversational Inertia in Multi-Turn Agentscited by 7
- 2026Old Habits Die Hard: How Conversational History Geometrically Traps LLMscited by 6
- 2025Systematic Bias in Large Language Models: Discrepant Response Patterns in Binary vs. Continuous Judgment Taskscited by 5
- 2025Acquiescence Bias in Large Language Modelscited by 5
- 2026Lost in the Middle at Birth: An Exact Theory of Transformer Position Biascited by 4
- 2025A Multifaceted Analysis of Negative Bias in Large Language Models through the Lens of Parametric Knowledgecited by 1
- 2026RBCorr: Response Bias Correction in Language Modelscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.