# PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on Structured Images

arXiv: https://arxiv.org/abs/2509.25185 · HTML: https://mlsanity.com/paper/2509.25185/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 46 of its 50 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 30 of 46)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate](https://mlsanity.com/paper/2305.19118/) (2023)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [AFlow: Automating Agentic Workflow Generation](https://mlsanity.com/paper/2410.10762/) (2024)
- [Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning](https://mlsanity.com/paper/2105.04165/) (2021)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [Character-LLM: A Trainable Agent for Role-Playing](https://mlsanity.com/paper/2310.10158/) (2023)
- [ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs](https://mlsanity.com/paper/2309.13007/) (2023)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models](https://mlsanity.com/paper/2406.09403/) (2024)
- [CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs](https://mlsanity.com/paper/2406.18521/) (2024)
- [Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2503.12605/) (2025)
- [ChartLlama: A Multimodal LLM for Chart Understanding and Generation](https://mlsanity.com/paper/2311.16483/) (2023)
- [MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making](https://mlsanity.com/paper/2404.15155/) (2024)
- [OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning](https://mlsanity.com/paper/2505.08617/) (2025)
- [MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering](https://mlsanity.com/paper/2212.09662/) (2022)
- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering](https://mlsanity.com/paper/2504.05506/) (2025)
- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding](https://mlsanity.com/paper/2501.05452/) (2025)

## Built on by (most cited first, 12 of 12)

- [Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?](https://mlsanity.com/paper/2607.09996/) (2026)
- [Dual Latent Memory for Visual Multi-agent System](https://mlsanity.com/paper/2602.00471/) (2026)
- [From Narrow to Panoramic Vision: Attention-Guided Cold-Start Reshapes Multimodal Reasoning](https://mlsanity.com/paper/2603.03825/) (2026)
- [SketchVL: Policy Optimization via Fine-Grained Credit Assignment for Chart Understanding and More](https://mlsanity.com/paper/2601.05688/) (2026)
- [SpotAgent: Grounding Visual Geo-localization in Large Vision-Language Models through Agentic Reasoning](https://mlsanity.com/paper/2602.09463/) (2026)
- [CogDoc: Towards Unified thinking in Documents](https://mlsanity.com/paper/2512.12658/) (2025)
- [See Less, See Right: Bi-directional Perceptual Shaping For Multimodal Reasoning](https://mlsanity.com/paper/2512.22120/) (2025)
- [Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reuse](https://mlsanity.com/paper/2606.23581/) (2026)
- [OmniHarness: Harnessing Generalizable Visual Generation via Symbolic Policy Learning](https://mlsanity.com/paper/2609.16057/) (2026)
- [Reasoning-Driven Amodal Completion: Collaborative Agents and Perceptual Evaluation](https://mlsanity.com/paper/2512.20936/) (2025)
- [InfoEdit: Probing Global Layout Reasoning in Infographic Editing](https://mlsanity.com/paper/2609.33286/) (2026)
- [RSure-Agent: Reliable Use of Tool Observations for Remote Sensing Agents](https://mlsanity.com/paper/2610.04836/) (2026)

## Abstract (arXiv)

Structured images (e.g., charts and geometric diagrams) remain challenging for multimodal large language models (MLLMs), as perceptual slips can cascade into erroneous conclusions. Intermediate visual cues can steer reasoning; however, existing cue-based methods are constrained with low-fidelity image processing and linear, rigid reasoning patterns, limiting their effectiveness on complex structured-image tasks. In this paper, we propose PixelCraft, a novel multi-agent system for high-fidelity image processing and flexible visual reasoning on structured images. The system comprises a dispatcher, a planner, a reasoner, critics, and a set of visual tool agents. To achieve high-fidelity processing, we construct a high-quality corpus and fine-tune an MLLM into a grounding model, whose pixel-level localizations are integrated with traditional computer vision (CV) algorithms in tool agents. Building on this foundation, PixelCraft facilitates flexible visual reasoning through a dynamic three-stage workflow of tool selection, agent discussion, and self-criticism. Moreover, unlike prior linear reasoning patterns that simply append historical images, PixelCraft maintains an image memory to allow the planner to adaptively revisit earlier visual steps, explore alternative reasoning branches, and dynamically adjust the reasoning trajectory during discussion. Extensive experiments on challenging chart and geometry benchmarks demonstrate that PixelCraft significantly improves visual reasoning performance for advanced MLLMs, setting a new standard for structured image reasoning. Our code will be available at https://github.com/microsoft/PixelCraft.