mlsanityData to 14 Oct 2026

AgentDiff: Meaning-Bearing Rewrites Trigger Deeper Divergence than Presentation Changes in LLM Agents

arXiv 2605.25981 ↗First posted 25 May 2026cs.CLcited by 0

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

LLM agents should respond to what an input means, not how it is presented. We show that they do not treat these two kinds of variation equally. AgentDiff measures the difference between answer changes caused by meaning-bearing rewrites (paraphrase and synonym substitution) and presentation changes (reordering, formatting, and distractors), while matching perturbation severity. Across 68 model--benchmark--scaffold cells spanning ten LLMs from seven architecture families, three benchmarks, 1,530 original questions, and approximately 11,150 variants, meaning-bearing rewrites produce a +19.69 percentage-point higher inconsistency rate (paired t=9.58, p<0.0001; 64/68 cells positive). The result is stable under four severity proxies (+18.9 to +20.9~pp, all p<0.0001) and remains +11.10~pp on the 48 cells outside the qwen family. A fully held-out qwen2.5-14B-Instruct evaluation then tests the structure of this effect on 1,800 new trajectories: the pre-registered capable-and-tractable partition is positive in 3/4 held-out cells and remains sharply separated after pooling (Welch t=3.81, p=9.6!×!10-4). Trace analysis explains how the difference propagates. Meaning-bearing rewrites preserve the first action but reduce thought similarity from step 2 onward by 5.6--10.5 points and extend the resulting cascade by 0.17 steps (paired t=7.69, p=2.5!×!10-14), a pattern we call stealth divergence. AgentDiff therefore establishes a reproducible directional robustness gap, identifies the regime in which it is most reliable, and connects final-answer inconsistency to a distinct trajectory-level signature. Code, perturbations, trajectories, and analysis scripts are released for review at https://anonymous.4open.science/r/agentdiff-emnlp-0BB4/

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 10 of its 25 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

10 of 25 references.

References · 10

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  7. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  8. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  9. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  10. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Probe-Rewrite-Evaluate: A Workflow for Reliable Benchmarks and Quantifying Evaluation Awareness
  2. 2026Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate
  3. 2025Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
  4. 2026Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity
  5. 2026From Delivery to Stateful Exploration: Rethinking the Index for Agentic Search
  6. 2025Unintended Misalignment from Agentic Fine-Tuning: Risks and Mitigation
  7. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Study
  8. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research