mlsanityData to 14 Oct 2026

Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges

arXiv 2610.18122 ↗First posted 13 Oct 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Typed decision models (TDMs) return a probability for each allowed answer instead of generating text, which makes them candidates for cheaply replacing LLM judges that compare two answers. Whether they share the biases of LLM judges is untested. We run the CALM benchmark's bias tests on eight TDMs and seven LLM judges of 2024 and 2026. Every TDM is biased, in three to eight of twelve tests. Emotional tone affects them most, and rewriting the better answer in such a tone lowers the accuracy of a typical TDM from 99% to 42%. TDMs are more biased than 2026 LLM judges. All eight change verdicts when the answers swap places, which the 2026 LLM judges hardly do, and several side with an answer that most people allegedly prefer or that a famous model allegedly wrote. Yet both kinds of judge are tripped up by the same prompts. The TDMs' probabilities show that most tests mainly flip verdicts the TDM was unsure of, and that good calibration does not protect against bias. As TDMs are more consistent and 37 times cheaper than 2026 LLM judges, our results suggest they can be attractive judges if their biases are tested for the task.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 58 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

34 of 58 references.

References · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  8. 2024A Survey on LLM-as-a-Judgecited by 1,423
  9. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  10. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  12. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
22 more
  1. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  4. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  5. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  6. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  7. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  8. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  9. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  10. 2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsurecited by 30
  11. 2026Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to Itcited by 25
  12. 2026Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factualitycited by 11
  13. 2026JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Modelscited by 10
  14. 2026Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfacescited by 3
  15. 2026More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Modelscited by 2
  16. 2026Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnessescited by 2
  17. 2026The Argument and the Letterhead: Source-Position Coherence in AI Evaluationcited by 1
  18. 2026Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistencycited by 1
  19. 2026Benchmarking System One decision models against trained classifiers and language models for automated decision gatescited by 1
  20. 2026Judgement in the Age of Jev: From Evaluation Scarcity to Evaluation Abundancecited by 1
  21. 2026Labels Override Definitions in Jev-Style Typed Decision Modelscited by 1
  22. 2026JEV versus LLMs: Accuracy, Cost and Calibration on Seven Political Science Replicationscited by 1

Built on by

Not cited yet.