mlsanityData to 14 Oct 2026

Unmatched Does Not Mean False: Incomplete Reference Sets Can Reverse Calibration Rankings in Open-Ended Theory-of-Mind Tracking

arXiv 2608.25654 ↗First posted 26 Aug 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Open-ended Theory-of-Mind (ToM) trackers emit valid beliefs absent from finite references. A finite-reference-plus-matcher pipeline marks unmatched outputs false, creating proxy labels that can reverse proper-score model selection on fixed outputs. Holding 259 beliefs and paired scores fixed, reference recoding lowers weighted prevalence from 0.783 to 0.295 and reverses strictly proper Brier risk: a frozen source-prior rule leads native confidence by 0.227 under reference labels and trails by 0.152 under blinded adjudication, in all six authored scenarios. A reference-only Platt recalibrator reverses further. An ICE-specific reversal appears in a released 301-question NQ-open DPR-BERT pipeline: its average-confidence baseline improves instance-level calibration error by 0.045 under exact match but worsens it by 0.074 under human correctness, with both intervals excluding zero. On independently authored OpenToM narratives, 90-96% of audited unmatched beliefs are literally true and the paired direction again reverses. An exact decomposition attributes the distortion to omitted truths, and a closed-form criterion correctly classifies comparisons from twelve released systems. Frozen-audit retrospective replay shows 50 attempted annotations recover ranking direction with probability at least 0.996. TriSource-Restore anchors full-frame reference labels and frozen automatic judgments to a probability-sampled human pilot, maintains at least nominal coverage, narrows intervals, and repairs confidence subject to a base-rate deployment gate.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 43 references that are in the map. First posted on arXiv 26 Aug 2026.

Builds on

19 of 43 references.

References · 19

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  7. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  8. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  10. 2017Selective Classification for Deep Neural Networkscited by 852
  11. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  12. 2018Detecting and Correcting for Label Shift with Black Box Predictorscited by 366
7 more
  1. 2019Verified Uncertainty Calibrationcited by 309
  2. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
  3. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  4. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 171
  5. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructionscited by 34
  6. 2025ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mindcited by 21
  7. 2026When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMscited by 2

Built on by

Not cited yet.