mlsanityData to 7 Oct 2026

The Evaluator Is Part of the Experiment: Measuring Open-Ended LLM Conformity

arXiv 2608.04463 ↗First posted 5 Aug 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Prior work on LLM conformity largely measures discrete answer flips under verifiable labels. Open-ended revisions require a different measurement strategy because answer quality is graded, latent, and judged imperfectly. We introduce an experimental protocol implemented across a pooled main peer-condition corpus and separately constructed decomposition corpora, allowing us to separate ordinary re-answering, candidate-content exposure, a bundled peer-presentation residual, and directional judge sensitivity to visible peer context. Across four open-weight generators and three benchmarks, all-wrong peer input produces the lowest-quality revisions in every generator-dataset cell. Blind and informed ratings of identical answers also differ by evaluator: one judge shifts toward the peer-endorsed position, two shift away, one is approximately neutral, and GPT-4o and GPT-5.4-mini audits are likewise non-neutral. Finally, an anchor audit shows that terse correct anchors can be misread often enough to destabilize the latent scale unless calibration is checked explicitly. These results support four conclusions: flip rates are insufficient as a complete measure of open-ended conformity, wrong peers harm open-ended revision, evaluators are not neutral, and anchor calibration is necessary.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 5 Aug 2026.

Builds on

21 of 33 references.

References · 21

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  8. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  9. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  10. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  11. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
  12. 2019Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyrocited by 187
9 more
  1. 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 91
  2. 2019Rank-normalization, folding, and localization: An improved widehatR for assessing convergence of MCMCcited by 90
  3. 2023When Large Language Models contradict humans? Large Language Models' Sycophantic Behaviourcited by 61
  4. 2024MultiAgent Collaboration Attack: Investigating Adversarial Attacks in Large Language Model Collaborations via Debatecited by 54
  5. 2025The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judgecited by 13
  6. 2025Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplifications and Resistance in Multi-Agent Based LLM-as-Judgecited by 10
  7. 2026Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformitycited by 6
  8. 2026It's Not Always Sycophancy: Measuring LLM Conformity as a Function of Epistemic Uncertaintycited by 4
  9. 2026Most LLM Conformity Needs No Speaker: Measuring the Speaker-Free Floor in Peer-Pressure Benchmarkscited by 4

Built on by

Not cited yet.