Benchmarking candidate coverage and rejection policy transfer in typed decision models
Not on the map. Why?
Abstract (arXiv)
Rejection policies must remain useful as candidate sets and tasks change. We compare Laya, Jev and Qwen2.5-7B-Instruct using public reference labels, testing Laya/Jev policy transfer at equal calibration budgets and all three models on artificial omission, natural retrieval misses and public out-of-scope queries. Source calibration often fails to preserve the target operating point. A Jev policy calibrated on DBpedia rejects 69.3% of covered Emotion test inputs, while an Emotion policy loses detection entirely. Retrieval exposes a different tradeoff: with ten intent candidates, Laya detects 99.0% of out-of-scope queries but rejects 48.8% of covered queries. Separating missing-answer sources reveals these costs alongside retrieval coverage. The benchmark provides shared inputs, explicit decision and failure categories, and reproducible scoring to assess rejection policies under the conditions in which they are reused. Code and benchmark artifacts are available at https://github.com/luckykevvv/Decision_Model_Benchmark.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 9 of its 20 references that are in the map. First posted on arXiv 2 Oct 2026.
Builds on
9 of 20 references.
References · 9
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
- 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 386
- 2021Open-Set Recognition: a Good Closed-Set Classifier is All You Need?cited by 353
- 2026Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to Itcited by 25
- 2020"None of the Above":Measure Uncertainty in Dialog Response Retrievalcited by 8
- 2025LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systemscited by 7
- 2026Do System One Decisions Add Up? A Study of Probabilistic Coherencecited by 3
- 2026When No Answer Is Correct: Diagnosing Absent Answer Detection for MLLMs in Video Understandingcited by 2
- 2026When the Right Answer Is Missing: An Arithmetic-Dependent Rejection Bottleneck in Jevcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.