mlsanityData to 14 Oct 2026

Lost in the Fallback: How Reliably Do Typed Decision Models Use Fallback Options?

arXiv 2610.19856 ↗First posted 14 Oct 2026cs.CR cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Typed decision models (TDMs) are increasingly used in applications such as request routing and model selection, making reliable decisions critical to system performance. Fallback options accommodate categories outside regular choices, enabling deferral to human experts or more capable models. While recent works have examined adversarial vulnerabilities in TDMs, their ability to follow fallback rules in non-adversarial settings remains underexplored. We introduce the Paired Fallback Robustness (PFR) Test to evaluate two aspects of fallback reliability: whether models correctly switch between regular and fallback options, and whether they make consistent decisions under logically equivalent definitions of the same rule. We evaluate open-source decision models on three benchmarks, with controlled comparisons across architectures and parameter counts. We find that correct recognition does not reliably transfer to correct fallback options, and errors persist even when the correct category is explicitly supplied.Moreover, logically equivalent formulations of the same fallback rule can also produce substantial disagreement in model decisions. These findings reveal a crucial reliability gap in current TDMs: fallback mechanisms designed to improve reliability can instead introduce errors. It highlights faithful execution of fallback rules and consistency under equivalent definitions as essential requirements for reliable typed decision-making.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 41 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

22 of 41 references.

References · 22

  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  4. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  5. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  6. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  7. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  8. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 386
  9. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 350
  10. 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
  11. 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 197
  12. 2024The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performancecited by 100
10 more
  1. 2018Interpretation of Natural Language Rules in Conversational Machine Readingcited by 75
  2. 2020Selective Classification via One-Sided Predictioncited by 38
  3. 2026Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to Itcited by 25
  4. 2026Decision Hijacking: Prompt Injection Attacks on Jev's Typed Probabilistic Decisionscited by 10
  5. 2026JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Modelscited by 10
  6. 2024Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognitioncited by 5
  7. 2026Do System One Decisions Add Up? A Study of Probabilistic Coherencecited by 3
  8. 2026Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfacescited by 3
  9. 2026SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agentscited by 1
  10. 2026Benchmarking candidate coverage and rejection policy transfer in typed decision modelscited by 1

Built on by

Not cited yet.