# Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understanding

arXiv: https://arxiv.org/abs/2609.27678 · HTML: https://mlsanity.com/paper/2609.27678/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 24 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Builds on (most cited first, 21 of 21)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)
- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [MathArena: Evaluating LLMs on Uncontaminated Math Competitions](https://mlsanity.com/paper/2505.23281/) (2025)
- [CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review](https://mlsanity.com/paper/2103.06268/) (2021)
- [AI Agents That Matter](https://mlsanity.com/paper/2407.01502/) (2024)
- [Show Your Work: Improved Reporting of Experimental Results](https://mlsanity.com/paper/1909.03004/) (2019)
- [ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts](https://mlsanity.com/paper/2110.01799/) (2021)
- [Batch Prompting: Efficient Inference with Large Language Model APIs](https://mlsanity.com/paper/2301.08721/) (2023)
- [Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging](https://mlsanity.com/paper/1707.09861/) (2017)
- [MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understanding](https://mlsanity.com/paper/2301.00876/) (2023)
- [BatchPrompt: Accomplish more with less](https://mlsanity.com/paper/2309.00384/) (2023)
- [General Synthetic-Powered Inference](https://mlsanity.com/paper/2509.20345/) (2025)

## Built on by (most cited first, 11 of 11)

- [Typed Decision Models: An Early Evidence Audit and Evaluation Checklist](https://mlsanity.com/paper/2609.32160/) (2026)
- [Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces](https://mlsanity.com/paper/2609.37470/) (2026)
- [When the Right Answer Is Missing: An Arithmetic-Dependent Rejection Bottleneck in Jev](https://mlsanity.com/paper/2609.39496/) (2026)
- [Benchmarking System One decision models against trained classifiers and language models for automated decision gates](https://mlsanity.com/paper/2610.00346/) (2026)
- [Hidden Risks of Jev: An Empirical Study of Security, Privacy, and Dual Use](https://mlsanity.com/paper/2610.04985/) (2026)
- [Evaluating System One Models for Agent Security Decisions: Reliability, Calibration, and Selective Automation](https://mlsanity.com/paper/2609.33401/) (2026)
- [Can Jev be Your Q or Policy in Reinforcement Learning?](https://mlsanity.com/paper/2610.11692/) (2026)
- [Can Decision Models Understand Stance? Evaluating Jev Against General-Purpose LLMs](https://mlsanity.com/paper/2610.11901/) (2026)
- [Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks](https://mlsanity.com/paper/2610.11978/) (2026)
- [The Company a Question Keeps: Auditing Open Jev-Style Models for Retrieval](https://mlsanity.com/paper/2610.14631/) (2026)
- [Correct Choices, Failed Sequences: Choice2Seq Benchmarks Correctness Retention Across Output Interfaces](https://mlsanity.com/paper/2610.16191/) (2026)

## Abstract (arXiv)

Aggregate accuracy summarizes how often a model is correct, but not which judgments remain correct when requests change. We evaluate Jev and nine language models on ContractNLI, with repeated requests that preserve the contract and target while varying visible hypotheses, requested answers, and output order. A four-model follow-up adds an equal-call repetition control and coverage of the remaining test contracts. Corrections and regressions can cancel, leaving unchanged accuracy despite changed decisions; consistent answers can also preserve errors. Majority voting over recorded responses yields small, uncertain gains, with repeated wrong answers accounting for much of the observed error. Gold-class diagnostics further reveal different error profiles behind aggregate model rankings. Resource comparisons add distinct trade-offs: Jev is cheapest and fastest for the joint baseline, while hosted language models achieve higher accuracy. Together, these findings connect decision-level evaluation to whether additional calls improve correctness and what the resulting decisions cost.