# JEV versus LLMs: Accuracy, Cost and Calibration on Seven Political Science Replications

arXiv: https://arxiv.org/abs/2610.06625 · HTML: https://mlsanity.com/paper/2610.06625/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 43 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Builds on (most cited first, 3 of 3)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Measuring Scalar Constructs in Social Science with LLMs](https://mlsanity.com/paper/2509.03116/) (2025)

## Built on by (most cited first, 1 of 1)

- [Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges](https://mlsanity.com/paper/2610.18122/) (2026)

## Abstract (arXiv)

Large language models (LLMs) annotate and scale political text or constructs by generating text tokens. A new class of models, which TypeSafe markets as "System One" models, instead returns decisions and probability distributions across a user-supplied fixed answer set. A commercial model, JEV, is advertised as having a dramatic cost and speed advantage over traditional LLMs along with better calibrated decisions. As such, it might be useful for social scientists looking to quickly and cost-effectively annotate or scale large corpora of text and have a reliable indicator of a classifier's uncertainty. Yet, the accuracy of these claims and the broader model accuracy in social science text-based tasks are not yet established. In this paper, we do just that and hope to establish the suitability of JEV for social science tasks. We compare JEV with LLMs and human coders from published research, and with a current mid-tier commercial LLM (GPT-6 Luna) and an open-weight alternative (Qwen3.8-27B). We find that JEV matches, or comes close to, the capabilities of both LLMs in a variety of tasks. However, we find no cost advantage over GPT-6 Luna at OpenAI's batch prices. Further, we find that, when each question is asked once, JEV's probabilities are better calibrated than GPT-6 Luna's token probabilities, but not consistently better than Qwen3.8-27B's. We conclude that unless researchers have a need for speed, JEV's only obvious advantage is ease of parsing the underlying choice probabilities.