mlsanityData to 14 Oct 2026

More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models

arXiv 2609.38827 ↗First posted 30 Sep 2026cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Direct-decision models enable efficient classification and evaluation by selecting from predefined candidates, but their reliability depends on more than prediction accuracy. Across 40 datasets and nine models from the JEV, KEV, and StartLux Decision families, we identify a systematic tendency to underuse ordinal rating scales. We call this phenomenon ordinal scale-utilization bias: final decisions concentrate on fewer effective labels than the gold distribution. The bias is substantially weaker on tasks with unordered semantic categories. Evaluating the same items on finer ordinal scales strengthens compression despite controls for gold imbalance and candidate position. Randomizing candidate order reduces but does not eliminate compression, while widening ordinal gaps at fixed candidate count can improve accuracy without restoring utilization. For most models, concentrated decisions coexist with broad average candidate probabilities. Greater model size improves accuracy without consistently restoring scale utilization. Together, these findings characterize compression as strongly associated with ordinal structure and intensified by scale refinement, rather than fully explained by presentation position or local spacing. This bias can be effectively mitigated through targeted training, making scale utilization an actionable reliability target alongside accuracy. The code and data are available at https://anonymous.4open.science/r/jev_ordinal_scale_bia/.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 34 of its 63 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

34 of 63 references.

References · 34

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  4. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  6. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  9. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  10. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  11. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  12. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
22 more
  1. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 417
  2. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  3. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 404
  4. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 377
  5. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 310
  6. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  7. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
  8. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  9. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 203
  10. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  11. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  12. 2013From Amateurs to Connoisseurs: Modeling the Evolution of User Expertise through Online Reviewscited by 108
  13. 2013A Computational Approach to Politeness with Application to Social Factorscited by 106
  14. 2023Mitigating Label Biases for In-context Learningcited by 89
  15. 2022EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysiscited by 76
  16. 2019A Large-scale Dataset for Argument Quality Ranking: Construction and Analysiscited by 67
  17. 2020DynaSent: A Dynamic Benchmark for Sentiment Analysiscited by 63
  18. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 43
  19. 2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsurecited by 30
  20. 2019"President Vows to Cut <Taxes> Hair": Dataset and Analysis of Creative Text Editing for Humorous Headlinescited by 27
  21. 2014LABR: A Large Scale Arabic Sentiment Analysis Benchmarkcited by 22
  22. 2026Decision Models vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Placescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.