Typed Decision Models: An Early Evidence Audit and Evaluation Checklist
Not on the map. Why?
Abstract (arXiv)
Typed decision models (TDMs) return probability distributions over caller-defined options without generating text. TypeSafe released Jev, a commercial typed decision model, on 15 September 2026, and a small body of evaluation and replication work appeared within days. We review 28 papers posted between 19 and 24 September and relate their findings to earlier work on label-probability classification, constrained decoding, reranking, calibration, and model cascades. In this early literature, the typed readout itself has not shown an independent accuracy advantage over comparable label-probability readouts. Jev's clearest gains are in latency and cost, while accuracy gaps remain on harder tasks. In practical deployments, confidence is often used to decide when to defer to a stronger model or a human. We use recurring weaknesses in these studies to derive a 14-item evaluation checklist for future TDM work. Because the evidence covers only the first nine days after the release of one hosted model, the review should be read as an early evidence map rather than a settled assessment of the model class.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 76 of its 94 references that are in the map. First posted on arXiv 26 Sep 2026.
Builds on
76 of 94 references.
References · 76
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2022Language Models (Mostly) Know What They Knowcited by 1,856
- 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
- 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
- 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
- 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
- 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
- 2017Selective Classification for Deep Neural Networkscited by 852
38 more
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
- 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
- 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
- 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
- 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
- 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 411
- 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 376
- 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
- 2019Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approachcited by 308
- 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
- 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
- 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
- 2020Selective Question Answering under Domain Shiftcited by 238
- 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 233
- 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 203
- 2023Efficient Guided Generation for Large Language Modelscited by 200
- 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 197
- 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
- 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 165
- 2023AutoMix: Automatically Mixing Language Modelscited by 125
- 2024Language Model Cascades: Token-level uncertainty and beyondcited by 121
- 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 117
- 2024Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generationcited by 101
- 2024Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Modelscited by 96
- 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 94
- 2023GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformercited by 88
- 2024XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Modelscited by 75
- 2024"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Modelscited by 69
- 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 60
- 2024Grammar-Aligned Decodingcited by 54
- 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 52
- 2025Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotationcited by 50
- 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 44
- 2025CRANE: Reasoning with constrained LLM generationcited by 38
- 2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsurecited by 30
- 2026Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to Itcited by 25
- 2023Building Efficient Universal Classifiers with Natural Language Inferencecited by 22
- 2025GLiClass: Generalist Lightweight Model for Sequence Classification Taskscited by 15
50 most cited of 76 All 76 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.