CoEvoke: Evoking Multi-Agent Collaboration for Trajectory Collection
Not on the map. Why?
Abstract (arXiv)
Training multi-agent collaboration requires trajectories of checking, feedback, and repair, yet task-solving rollouts may end in immediate approval or repeat unsuccessful attempts without useful correction. We introduce CoEvoke, a cross-domain framework that evokes these interactions during trajectory collection. Controlled perturbations create recoverable difficulties, while restricted reference-based guidance directs investigation. The resulting dataset contains 4,567 reconstructable trajectories over 4,130 software-engineering and reasoning tasks, linking public interactions to evidence, decisions, and task-native outcomes. Separate experiments over 2,287 same-task pairs show feedback increases of 47.72-100 percentage points across seven perturbation groups; guidance has smaller, model-dependent effects. With identical training tasks, decision-record counts, and updates, perturbation-enriched judge training improves ACPBench-Hard success from 35.38% to 42.12% (+6.73 points). SWE verifier training gains 7.60-9.40 points over interface-matched untrained references. Workflow and post-training evaluations preserve task-solving topology with unperturbed inputs and no private-reference guidance. These results support designing collection conditions to obtain corrective supervision that improves collaborative decisions with frozen solving agents.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 49 of its 49 references that are in the map. First posted on arXiv 29 Sep 2026.
Builds on
49 of 49 references.
References · 49
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2023Let's Verify Step by Stepcited by 3,789
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
37 more
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
- 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
- 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
- 2023ChatDev: Communicative Agents for Software Developmentcited by 919
- 2022Solving math word problems with process- and outcome-based feedbackcited by 901
- 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
- 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
- 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
- 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
- 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
- 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
- 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 324
- 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
- 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
- 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
- 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 280
- 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 241
- 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 201
- 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agentscited by 178
- 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 167
- 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Taskscited by 142
- 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 130
- 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?cited by 116
- 2025Multi-Agent Collaboration via Evolving Orchestrationcited by 108
- 2025Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RLcited by 80
- 2024MALT: Improving Reasoning with Multi-Agent LLM Trainingcited by 76
- 2025AgentRefine: Enhancing Agent Generalization through Refinement Tuningcited by 53
- 2025DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systemscited by 24
- 2025Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agentscited by 17
- 2026AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agentcited by 13
- 2025ACPBench Hard: Unrestrained Reasoning about Action, Change, and Planningcited by 8
Built on by
Not cited yet.