# LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogy

arXiv: https://arxiv.org/abs/2603.14893 · HTML: https://mlsanity.com/paper/2603.14893/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Builds on (most cited first, 15 of 15)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [The Effect of Sampling Temperature on Problem Solving in Large Language Models](https://mlsanity.com/paper/2402.05201/) (2024)
- [A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions](https://mlsanity.com/paper/2412.05563/) (2024)
- [What Large Language Models Know and What People Think They Know](https://mlsanity.com/paper/2401.13835/) (2024)
- [Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey](https://mlsanity.com/paper/2503.15850/) (2025)
- [Thermometer: Towards Universal Calibration for Large Language Models](https://mlsanity.com/paper/2403.08819/) (2024)
- [Metacognition and Uncertainty Communication in Humans and Large Language Models](https://mlsanity.com/paper/2504.14045/) (2025)
- [Contrast Sensitivity in Multimodal Large Language Models: A Psychophysics-Inspired Evaluation](https://mlsanity.com/paper/2508.10367/) (2025)

## Built on by (most cited first, 7 of 7)

- [Do LLMs Know What They Know? Measuring Metacognitive Efficiency with Signal Detection Theory](https://mlsanity.com/paper/2603.25112/) (2026)
- [LLM Judges Have Dark Current: Stratified Null and Utility Tests for LLM Judge Reliability](https://mlsanity.com/paper/2606.15610/) (2026)
- [Metacognition in LLMs: Foundations, Progress, and Opportunities](https://mlsanity.com/paper/2607.11881/) (2026)
- [Domain-level metacognitive monitoring in frontier LLMs: A 33-model atlas](https://mlsanity.com/paper/2605.06673/) (2026)
- [Prior Audit-Repair Context Shifts LLM Verifier Thresholds Toward Leniency](https://mlsanity.com/paper/2608.16003/) (2026)
- [Anatomy of LLM Sycophancy: What a Flip Rate Hides](https://mlsanity.com/paper/2610.06522/) (2026)
- [Sensitivity or Criterion? Measuring and Moving When Language Models Act](https://mlsanity.com/paper/2610.15415/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are evaluated for calibration using metrics such as Expected Calibration Error that conflate two distinct components: the model's ability to discriminate correct from incorrect answers (sensitivity) and its tendency toward confident or cautious responding (bias). Signal Detection Theory (SDT) decomposes these components. While SDT-derived metrics such as AUROC are increasingly used, the full parametric framework - unequal-variance model fitting, criterion estimation, z-ROC analysis - has not been applied to LLMs as signal detectors. In this pre-registered study, we treat three LLMs as observers performing factual discrimination across 168,000 trials and test whether temperature functions as a criterion shift analogous to payoff manipulations in human psychophysics. Critically, this analogy may break down because temperature changes the generated answer itself, not only the confidence assigned to it. Our results confirm the breakdown with temperature simultaneously increasing sensitivity (AUC) and shifting criterion. All models exhibited unequal-variance evidence distributions (z-ROC slopes 0.52-0.84), with instruct models showing more extreme asymmetry (0.52-0.63) than the base model (0.77-0.87) or human recognition memory (~0.80). The SDT decomposition revealed that models occupying distinct positions in sensitivity-bias space could not be distinguished by calibration metrics alone, demonstrating that the full parametric framework provides diagnostic information unavailable from existing metrics.