mlsanityData to 7 Oct 2026

OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment

arXiv 2607.26981 ↗First posted 29 Jul 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models are increasingly used as decision aids whose probability judgments shape downstream choices. Whether those judgments carry a systematic directional tilt has been hard to detect: calibration metrics aggregate unsigned errors, and naturalistic uncertainty offers no ground-truth probability. When an LLM rates a startup's success at 70% but its failure at 15%, the missing 15 points expose a distortion no aggregate score flags. We introduce OptimismBench, which detects directional bias with inverted pairs: each scenario elicits both P(success) and P(failure), and asymmetry between the two framings yields a signed bias score without ground truth. Across 16 models from 8 providers, fourteen are optimistic; pessimism appears only in Anthropic's frontier tier. Eleven matched base-versus-chat pairs across four families show post-training sets the sign of the bias, with opposite shifts in different families. The pattern survives prompt, temperature, perspective, and self-debiasing ablations. A seventeen-model six-language comparison further shows model identity dominates language, with inter-model variance at 4.7x inter-language variance. We release 3,870 items across 10 languages for per-model directional-bias auditing. When alignment makes a model more helpful, it also tilts its probabilities; downstream pipelines inherit the tilt by default.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 34 references that are in the map. First posted on arXiv 29 Jul 2026.

Builds on

24 of 34 references.

References · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  5. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  6. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  7. 2023Whose Opinions Do Language Models Reflect?cited by 759
  8. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  9. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  10. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  11. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  12. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
12 more
  1. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  2. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  3. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  4. 2024Cognitive Bias in Decision-Making with LLMscited by 104
  5. 2024Approaching Human-Level Forecasting with Language Modelscited by 102
  6. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 97
  7. 2024Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracycited by 66
  8. 2024CogBench: a large language model walks into a psychology labcited by 60
  9. 2024A Comprehensive Evaluation of Cognitive Biases in LLMscited by 32
  10. 2024Consistency Checks for Language Model Forecasterscited by 27
  11. 2014Human language reveals a universal positivity biascited by 23
  12. 2025Exploring the Potential for Large Language Models to Demonstrate Rational Probabilistic Beliefscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Fragile Preferences: A Deep Dive Into Order Effects in Large Language Models
  2. 2024Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models
  3. 2026GAIN: A Benchmark for Goal-Aligned Decision-Making of Large Language Models under Imperfect Norms
  4. 2026When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models
  5. 2025Actions Speak Louder than Words: Agent Decisions Reveal Implicit Biases in Language Models
  6. 2025Knowing What You Know Is Not Enough: Large Language Model Confidences Don't Align With Their Actions
  7. 2025Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models
  8. 2025Rethinking Prospect Theory for LLMs: Revealing the Instability of Decision-Making under Epistemic Uncertainty