Sandboxed Coding Agents are Competitive Omni-modal Task Solvers
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
also in 1 more
Abstract (arXiv)
As multimodal LLMs increasingly target video and audio, it is often assumed that such tasks require native omnimodal models. We show that this is not always the case: coding agents with only text+image access and a sandboxed tool-use interface can match, and in several settings outperform, SOTA native omnimodal models and predefined multimodal agent scaffolds across multiple audio-video benchmarks. Our trajectory analysis suggests that their strength comes from writing code and orchestrating tools to extract relevant evidence from transcripts, frames, and other modality signals, thereby converting omnimodal tasks into retrieval and information-processing problems rather than ingesting entire media streams. We further characterize their limitations through a failure taxonomy and process-level trace analysis, and show that simple skill injection, including human-written and self-distilled skills, substantially improves performance. To explore open-source elicitation, we introduce Code-X, a training recipe with the OmniCoding trajectory dataset and verifiable reward, and provide baselines on Qwen-3.5-9B and Qwen-3.6-27B. Finally, we argue that the next frontier is many-modality processing, and introduce TerminalBench-O, a process-level benchmark for real-world omnimodal processing tasks. Code will be available at https://github.com/Dongping-Chen/OmniCoding.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 86 of its 107 references that are in the map. First posted on arXiv 30 May 2026.
Builds on
86 of 107 references.
References · 86
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
- 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
- 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
- 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
- 2025Group Sequence Policy Optimizationcited by 835
38 more
- 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
- 2025Qwen2.5-Omni Technical Reportcited by 736
- 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
- 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
- 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
- 2024Executable Code Actions Elicit Better LLM Agentscited by 564
- 2024Qwen2-Audio Technical Reportcited by 560
- 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
- 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 421
- 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
- 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
- 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 315
- 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 309
- 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 292
- 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
- 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
- 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 279
- 2024AutoCodeRover: Autonomous Program Improvementcited by 278
- 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
- 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
- 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
- 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 191
- 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
- 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
- 2025Thyme: Think Beyond Imagescited by 167
- 2025Measuring AI Ability to Complete Long Software Taskscited by 156
- 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
- 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
- 2026Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forwardcited by 113
- 2025MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Serverscited by 109
- 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 102
- 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 102
- 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 98
- 2025Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?cited by 95
- 2025PyVision: Agentic Vision with Dynamic Toolingcited by 79
- 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Taskscited by 75
- 2024MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learningcited by 69
- 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agentscited by 64
50 most cited of 86 All 86 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Qwen3.8-Omni: Towards Native Omni-Modal Agents
- 2026MMPostTrainBench: Benchmarking Autonomous Research for Multimodal Post-Training
- 2026GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents
- 2026VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation
- 2026MMSkills: Towards Multimodal Skills for General Visual Agents
- 2026A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications
- 2026Orchestra-o1: Omnimodal Agent Orchestration
- 2026Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?