mlsanityData to 7 Oct 2026

Persuaded, Not Informed: Incentive-Misaligned Witnesses Defeat In-Context Grounding

arXiv 2609.28854 ↗First posted 23 Sep 2026cs.CL cs.AIcited by 0

On the map: LLM agents and tool use → Social simulation and multi-agent debate also fits: Financial trading and forecasting agents

Abstract (arXiv)

Language-model agents increasingly answer questions over customer-relationship management (CRM) records, such as whether to qualify a sales lead. We identify a failure mode not addressed by a stronger model: when the context contains an assertion by a party with an incentive toward optimism - here the sales representative, a witness recorded in the CRM - the model treats the assertion as evidence and clears deals the company's own records deem unacceptable. Across 100 lead-qualification tasks from CRMArena-Pro, the representative asserts an acceptable timeline in every call and an acceptable budget in 76; on the 31 tasks where such an assertion contradicts the price list and installation policy, a model reading only the transcript clears the deal in 29 of 31 cases. The signature is consistent across seven models from four providers (misled on 87-97%); scale and explicit reasoning confer no resistance. Only 3 of 35 genuine failures involve no assertion: the failure is persuasion, not missing information. We contribute a diagnostic method rather than an architecture: (i) a bucket analysis that separates persuasion from information gaps, (ii) a same-information control showing that supplying the records to the model lowers strict accuracy from 41 to 18 while raising recall - precision collapses - and (iii) a compute-step control that holds extraction fixed and varies only who computes Budget and Timeline. The margin ranges from 42 points on an inexpensive model to 2-5 points on models that already compute correctly; on the strongest models the arms are within confidence intervals, so the pattern is a consistent direction and a soundness property, not a proved performance floor. We pre-specify a generalization test that returns a negative result, characterize the precondition (a policy exactly specified in the inputs), and release all evaluation artifacts.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 28 of its 41 references that are in the map. First posted on arXiv 23 Sep 2026.

Builds on

28 of 41 references.

References · 28

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  9. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  11. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  12. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
16 more
  1. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  2. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  3. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  4. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  5. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 722
  6. 2022PAL: Program-aided Language Modelscited by 635
  7. 2025Kimi K2: Open Agentic Intelligencecited by 523
  8. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  9. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  10. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  11. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  12. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  13. 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
  14. 2023On the Risk of Misinformation Pollution with Large Language Modelscited by 124
  15. 2023Chain of Code: Reasoning with a Language Model-Augmented Code Emulatorcited by 120
  16. 2025CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactionscited by 46

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Knowledge-Verified Emergent Deception in LLM Agents Under Conflicting Incentives
  2. 2026Resist and Update: Counterfactual Report Coordinates for Incentive-Compatible LLMs
  3. 2026Used Car Salesbots? Honesty and Credulity of LLMs as Bargaining Agents under Partial Information
  4. 2026Manufactured Confidence: How Memory Consolidation Turns Hearsay into Confident Facts
  5. 2026Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail
  6. 2026When LLM Rationales Become User-Facing: Effects on Trust Perception, Decision-Making, and Gaze Behaviors
  7. 2026Language Models Are "Insecure" Reporters
  8. 2026Do Models Know Why They Changed Their Mind? Interpretability and Faithfulness of Chain-of-Thought Under Knowledge Conflict