Confidently Wrong, Silently So: Auditing Undetectable Failures of a Deployed On-Device Language Model
On the map: LLM safety and security → Safety alignment, refusal and guardrails
Abstract (arXiv)
Aligning deployed language models requires knowing when their outputs can be trusted, yet on-device models now ship to hundreds of millions of devices with no server-side moderation, and the configuration developers can actually deploy is rarely audited independently. We present a reproducible reliability audit of the developer-accessible on-device foundation model, framed as an oversight question: can a user or a resource-constrained developer tell when the model is wrong? Red-teaming it on calibration, confident confabulation on false-premise questions, and over-refusal of benign prompts, we find a task-asymmetric miscalibration: its guardrails fail in opposite directions across tasks (confabulating on 69% of false premises while refusing 18% of entirely benign inputs), atop a self-reported confidence that is saturated and non-discriminative (AUROC 0.47; ECE 70, worst among comparable small models). Crucially, confident-correct and confident-wrong outputs are surface-indistinguishable: a classifier over 15 user-visible features separates them at AUROC only 0.55 (equivalence-confirmed), leaving no signal for oversight at inference time. No cheap single-generation signal flags these failures (≤0.68 AUROC), whereas a black-box consistency wrapper requiring no model access recovers reliability (confident confabulation 75%→3%; selective accuracy 43%→83%) at a tunable cost. We contribute a model-agnostic audit protocol, a surface-indistinguishability test, and released code and frozen evaluation items as reusable infrastructure for auditing deployed models.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 7 of its 11 references that are in the map. First posted on arXiv 24 Aug 2026.
Builds on
7 of 11 references.
References · 7
- 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
- 2022Language Models (Mostly) Know What They Knowcited by 1,856
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
- 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
- 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
- 2026SELFDOUBT: Uncertainty Quantification for Reasoning LLMs via the Hedge-to-Verify Ratiocited by 3
- 2026What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Modelscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026The Knowing-Saying Gap: When Probes See Errors that Confidence Misses
- 2026Skills as Verifiable Artifacts: A Trust Schema and a Biconditional Correctness Criterion for Human-in-the-Loop Agent Runtimes
- 2025Trusted Uncertainty in Large Language Models: A Unified Framework for Confidence Calibration and Risk-Controlled Refusal
- 2026When Small Models Are Right for Wrong Reasons: Process Verification for Trustworthy Agents
- 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMs
- 2025The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems
- 2026Layerwise Convergence Fingerprints for Runtime Misbehavior Detection in Large Language Models
- 2026Loud Failures, Quiet Failures: Fault Detection and Recovery in Tool-Using Language Model Agents