mlsanityData to 14 Oct 2026

Scored vs. Generated Readouts in Behavioral Language Models: An Empirical Study of Elicitation Format

arXiv 2609.09882 ↗First posted 9 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Language models fine-tuned on customer behavior can predict outcomes and generate explanations, but these readouts are often treated as interchangeable. Holding model checkpoint and prompt content fixed, we compare probabilities obtained by scoring answer tokens with predictions generated after a written rationale. Across 13 model-domain cells covering four retail tasks in three markets, including two using fully public data and checkpoints, the scored readout ranks outcomes more accurately in 12 of 13 cells (two-sided sign test, p approximately 0.003), by 1.5 to 14.5 points in area under the receiver operating characteristic curve (AUC). Paired bootstrap confidence intervals exclude zero in every newly measured cell. The gap varies with task-specific supervision and mismatch between training and serving formats, ranging from -2.2 points for an untuned base model to +13.7 for rationale-format supervision. Analysis of approximately 9,000 rationales identifies two correlates: reduced reliance on the dominant predictive feature and convergence on stock formulations. Probability saturation does not track the gap. A third readout, eliciting a probability before any verdict, improves calibration (Brier score from 0.47 to 0.15) while ranking within noise of scoring, but only for outcome rates represented in training; it is worse than scoring when the scored head is already calibrated. We interpret these differences through the objectives matched by each readout, identify training choices that narrow the gap, and propose retaining generated rationales while sourcing ranking from the scored head.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 16 references that are in the map. First posted on arXiv 9 Sep 2026.

Builds on

16 of 16 references.

References · 16

  1. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  2. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  3. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  4. 2025When More is Less: Understanding Chain-of-Thought Length in LLMscited by 183
  5. 2025Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversitycited by 105
  6. 2024"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Modelscited by 69
  7. 2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviorscited by 52
  8. 2025OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulationcited by 28
  9. 2026Qwen-AgentWorld: Language World Models for General Agentscited by 28
  10. 2025Customer-R1: Personalized Simulation of Human Behaviors via RL-based LLM Agent in Online Shoppingcited by 19
  11. 2026Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewardscited by 14
  12. 2025LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratingscited by 12
4 more
  1. 2026Same Answer, Different Confidence: Protocol Sensitivity in LLM Confidence Calibrationcited by 6
  2. 2021DMBGN: Deep Multi-Behavior Graph Networks for Voucher Redemption Rate Predictioncited by 3
  3. 2026Calibration Drift Under Reasoning: How Chain-of-Thought Budgets Induce Overconfidence in Large Language Modelscited by 3
  4. 2026Large Behavior Model: A Promptable Digital Twin of the Retail Customercited by 1

Built on by

Not cited yet.