Signed Lexical Confidence for Risk-Calibrated Intent Routing
Not on the map. Why?
Abstract (arXiv)
Selective intent routing allows an assistant to act on reliable predictions while deferring uncertain requests. Standard confidence scores primarily reflect the base model's representation, leaving an opportunity to incorporate complementary evidence without changing its decisions. We introduce a signed lexical gate that combines a sentence classifier's logit margin with a sparse lexical model's support for the classifier's predicted intent. By assigning positive evidence to lexical agreement and negative evidence to a lexically favored competing intent, the gate retains more information than either unsigned lexical confidence or a hard agreement rule. An independent binomial calibration stage selects an operating threshold for a specified risk target. Across ten runs on BANKING77, CLINC150, and HWU64, the proposed score reduces area under the risk-coverage curve by 15.8%, 15.1%, and 11.8% relative to a learned semantic-only gate. At a nominal 5% error target, it increases accepted coverage by 1.83 and 5.14 percentage points on BANKING77 and HWU64, while CLINC150 is already near full coverage. At a stricter 2% target, the simultaneous binomial procedure yields a nonempty policy in all 30 dataset-run combinations at the available calibration budgets. Matched controls show that the proposed feature improves average error ranking over the tested unsigned lexical-confidence feature, with dataset-dependent gains over binary agreement. The resulting two-feature gate provides a compact, interpretable confidence enhancement for risk-calibrated intent routing while preserving the base classifier's predictions.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 11 of its 15 references that are in the map. First posted on arXiv 24 Sep 2026.
Builds on
11 of 15 references.
References · 11
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2017Selective Classification for Deep Neural Networkscited by 799
- 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
- 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
- 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 256
- 2018To Trust Or Not To Trust A Classifiercited by 205
- 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
- 2024Selective Classification Under Distribution Shiftscited by 12
- 2025Selective Conformal Risk Controlcited by 10
- 2026Conformal Selective Prediction with General Risk Controlcited by 8
- 2025Know When to Abstain: Optimal Selective Classification with Likelihood Ratioscited by 5
Built on by
Not cited yet.