Do LLM Recommenders Know When They're Hallucinating? Auditing Confidence Calibration in Catalog Faithfulness
Not on the map. Why?
Abstract (arXiv)
LLM recommenders for top-K item suggestion regularly emit titles outside the target catalog. Prior audits report a binary out-of-domain rate; none ask whether the model knew. We jointly audit hallucination rate (OOD@10) and verbalized-confidence calibration (ECE, Brier, reliability) for four zero-shot LLM recommenders from four independent vendors (Mistral Large, Llama-3.3-70B, GPT-OSS-120B, Claude Sonnet 4.6), not grounded or fine-tuned systems, across three catalogs (MovieLens-25M, Amazon Reviews 2023 Toys, Yelp Open Dataset), stratified by item popularity. Measuring catalog membership is itself the hard part: on identical outputs the reported rate moves by an order of magnitude with the string matcher used, and F1 cannot separate the candidates. We validate the instrument against 201 human judgments and select on net bias, where the adopted one is off by -0.040 against +0.144 for the common fuzzy rule. Hallucination is then strongly catalog-dependent (0.6-2.7% on MovieLens, 11.6-38.7% on Yelp, 49.3-61.0% on Amazon Toys). Each model holds a near-constant confidence level barely responsive to the catalog, while the catalog-hit rate swings 60 points, so the sign of the error is set by where a model's constant lands against a catalog's accuracy: 7 of the twelve cells are under-confident and 5 over-confident, all four under-confident on MovieLens, all four over-confident on Amazon Toys. We read this as an elicitation mismatch: "Just Ask" elicits a generic quality rating, not a catalog-membership probability. A conformal abstention threshold over verbalized confidence changes hallucination by at most 1.65 pp across four alpha levels, because the channel cannot separate correct items from hallucinations. We recommend that audits report calibration alongside OOD, validate the matcher producing the OOD number, and use catalog-anchored elicitation.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 7 Aug 2026.
Builds on
17 of 31 references.
References · 17
- 2022Language Models (Mostly) Know What They Knowcited by 1,856
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
- 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
- 2020Selective Question Answering under Domain Shiftcited by 238
- 2024Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoderscited by 224
- 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
- 2023Large language models can accurately predict searcher preferencescited by 131
- 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
- 2023Perspectives on Large Language Models for Relevance Judgmentcited by 85
- 2023A Bi-Step Grounding Paradigm for Large Language Models in Recommendation Systemscited by 73
- 2024Mitigating LLM Hallucinations via Conformal Abstentioncited by 59
- 2024Generative Recommender with End-to-End Learnable Item Tokenizationcited by 34
5 more
- 2025Uncertainty Quantification and Decomposition for LLM-based Recommendationcited by 12
- 2025Toward Holistic Evaluation of Recommender Systems Powered by Generative Modelscited by 11
- 2025Eliminating Out-of-Domain Recommendations in LLM-based Recommender Systems: A Unified Viewcited by 6
- 2025Alleviating LLM-based Generative Retrieval Hallucination in Alipay Searchcited by 4
- 2026Evaluation on Entity Matching in Recommender Systemscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.