TRACER: Verifiable Generative Provenance for Multimodal Tool-Using Agents
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
also in 1 more
Abstract (arXiv)
Multimodal large language models increasingly solve vision-centric tasks by calling external tools for visual inspection, OCR, retrieval, calculation, and multi-step reasoning. Current tool-using agents usually expose the executed tool trajectory and the final answer, but they rarely specify which tool observation supports each generated claim. We call this missing claim-level dependency structure the provenance gap. The gap makes tool use hard to verify and hard to optimize, because useful evidence, redundant exploration, and unsupported reasoning are mixed in the same trajectory. We introduce TRACER, a framework for verifiable generative provenance in multimodal tool-using agents. Instead of adding citations after generation, TRACER generates each answer sentence together with a structured provenance record that identifies the supporting tool turn, evidence unit, and semantic support relation. Its relation space contains Quotation, Compression, and Inference, covering direct reuse, faithful condensation, and grounded derivation. TRACER verifies each record through schema checking, tool-turn alignment, source authenticity, and relation rationality, and then converts verified provenance into traceability constraints and provenance-derived local credit for reinforcement learning. We further construct TRACE-Bench, a benchmark for sentence-level provenance reconstruction from coarse multimodal tool trajectories. On TRACE-Bench, simply adding tools often introduces noise. With Qwen3-VL-8B, TRACER reaches 78.23% answer accuracy and 95.72% summary accuracy, outperforming the strongest closed-source tool-augmented baseline by 23.80 percentage points. Compared with tool-only supervised fine-tuning, it also reduces total test-set tool calls from 4949 to 3486. These results show that reliable multimodal tool reasoning depends on provenance-aware use of observations, not on more tool calls alone.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 41 references that are in the map. First posted on arXiv 11 May 2026.
Builds on
35 of 41 references.
References · 35
- 2024GPT-4o System Cardcited by 4,467
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
- 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
- 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
- 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 145
- 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
- 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 120
- 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
23 more
- 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 117
- 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 114
- 2025Process Reward Models for LLM Agents: Practical Framework and Directionscited by 81
- 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 69
- 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attributioncited by 68
- 2026Hindsight Credit Assignment for Long-Horizon LLM Agentscited by 51
- 2025ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understandingcited by 50
- 2025STeCa: Step-level Trajectory Calibration for LLM Agent Learningcited by 37
- 2026Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Modelscited by 33
- 2026Vision-DeepResearch Benchmark: Rethinking Visual and Textual Search for Multimodal Large Language Modelscited by 30
- 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 28
- 2026AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarioscited by 22
- 2024From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learningcited by 21
- 2025ToolVQA: A Dataset for Multi-step Reasoning VQA with External Toolscited by 13
- 2026Act Wisely: Cultivating Meta-Cognitive Tool Use in Agentic Multimodal Modelscited by 13
- 2026Towards Long-horizon Agentic Multimodal Searchcited by 9
- 2026MTA-Agent: An Open Recipe for Multimodal Deep Search Agentscited by 7
- 2026Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimizationcited by 7
- 2026E3-TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoningcited by 6
- 2026Learning from the Irrecoverable: Error-Localized Policy Optimization for Tool-Integrated LLM Reasoningcited by 5
- 2025TROVE: A Challenge for Fine-Grained Text Provenance via Source Sentence Tracing and Relationship Classificationcited by 4
- 2026GenProve: Learning to Generate Text with Fine-Grained Provenancecited by 2
- 2026EpiBench: Benchmarking Multi-turn Research Workflows for Multimodal Agentscited by 2
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026When May an Agent Stop? Evidence-Carrying Termination for Tool-Using LLMscited by 2
- 2026Cost-Utility Alignment in LLM Agent Trajectories:Profiling,Attribution,Diagnosis,Adaptation,and Evaluationcited by 1
- 2026Plans They Abandon, Reports They Author: The Narrative Layer of Autonomous Agentscited by 1
- 2026Plans They Abandon, Reports They Author: The Narrative Layer of Autonomous Agents10 Sep 2026
- 2026Cost-Utility Alignment in LLM Agent Trajectories:Profiling,Attribution,Diagnosis,Adaptation,and Evaluation25 Aug 2026
- 2026When May an Agent Stop? Evidence-Carrying Termination for Tool-Using LLMs22 Aug 2026
Similar, not cited
Close in wording, no citation either way
- 2026LEDGERMIND: Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger
- 2026ATP-Bench: Towards Agentic Tool Planning for MLLM Interleaved Generation
- 2026FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verification
- 2026InSight: A Benchmark for Agentic Claim Verification in Interactive Visualizations
- 2026Towards Verifiable Multimodal Deep Research: A Multi-Agent Harness for Interleaved Report Generation
- 2026From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agents
- 2025DART: Leveraging Multi-Agent Disagreement for Tool Recruitment in Multimodal Reasoning
- 2026Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gains