mlsanityData to 14 Oct 2026

Disclosure-Gated User Simulation for Companion-Agent Evaluation

arXiv 2609.00982 ↗First posted 1 Sep 2026cs.CL cs.AI cs.HCcited by 1

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

Playing the user with a large language model is standard in scalable evaluation, and has a repeatedly diagnosed failure: the simulated user is excessively cooperative, so a system under test can score by the number of questions it asks rather than by making the user willing to speak. We answer with a disclosure gate conditioning information release on the companion agent's behaviour: its state is a ladder of five ordered gates, merged onto three observable depth layers. We specify, ablate, and audit it. The gate is a load-bearing component of the environment: on a published companion-agent benchmark's English corpus, withdrawing per-example gate statements from training displaces the ranking of the 12 systems beyond the noise band of a seed re-run, while per-system scores show no detectable change. We state two acceptance criteria: a ranking must be order-preserving, and absolute scores must be scale-stable. Only one candidate passes both -- the simulator we release -- correlating at 0.993 with the published leaderboard. Prompting a frontier model instead barely moves the ranking while shifting every score up, invisible to ranking-only checks; we supply the specification, ablations, and downstream sensitivity analysis the benchmark's environment lacked.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 17 of its 32 references that are in the map. First posted on arXiv 1 Sep 2026.

Builds on

17 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents13 May
  2. 2026Mind the Sim2Real Gap in User Simulation for Agentic Tasks11 Mar
  3. 2026Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations23 Jan

Other references · 14

  1. 2025SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?cited by 50
  2. 2020Evaluating Conversational Recommender Systems via User Simulationcited by 47
  3. 2023Reducing Privacy Risks in Online Self-Disclosures with Language Modelscited by 47
  4. 2026HumanLM: Simulating Users with State Alignment Beats Response Imitationcited by 37
  5. 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 36
  6. 2023User Simulation for Evaluating Information Access Systemscited by 31
  7. 2026Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistantscited by 15
  8. 2025Consistent Client Simulation for Motivational Interviewing-based Counselingcited by 14
  9. 2026CompanionBench: A Theory-Anchored, Real-World-Grounded Benchmark for AI Emotional Companionshipcited by 5
  10. 2025Detecting Emotional Dynamic Trajectories: An Evaluation Framework for Emotional Support in Language Modelscited by 4
  11. 2026PSI-Bench: Interpretable and Clinically Meaningful Evaluation of Depression Patient Simulatorscited by 2
  12. 2026SimEval-IR: A Unified Toolkit and Benchmark Suite for Evaluating User Simulators and Search Sessionscited by 1
2 more
  1. 2026The Empirically Grounded Adaptive Virtual Patient for Psychotherapy Training: Disclosure That Responds to Therapist Micro-Skillscited by 1
  2. 2026PatientAct: Theory-Grounded Mental Health Client Simulationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training
  2. 2026Learning User Simulators with Turing Rewards
  3. 2026Role Steering of Language Models for Social Simulations
  4. 2026RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation
  5. 2026CRAB-Bench: Evaluating LLM Agents under Complex Task Dependencies and Human-aligned User Simulation
  6. 2026MirrorBench: A Benchmark to Evaluate Conversational User-Proxy Agents for Human-Likeness
  7. 2026ACCORD: Action-Conditioned Contextual Grounding for Language Agents
  8. 2025Among Us: A Sandbox for Measuring and Detecting Agentic Deception