mlsanityData to 14 Oct 2026

Sandboxed Coding Agents are Competitive Omni-modal Task Solvers

arXiv 2606.00579 ↗First posted 30 May 2026cs.CL cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

As multimodal LLMs increasingly target video and audio, it is often assumed that such tasks require native omnimodal models. We show that this is not always the case: coding agents with only text+image access and a sandboxed tool-use interface can match, and in several settings outperform, SOTA native omnimodal models and predefined multimodal agent scaffolds across multiple audio-video benchmarks. Our trajectory analysis suggests that their strength comes from writing code and orchestrating tools to extract relevant evidence from transcripts, frames, and other modality signals, thereby converting omnimodal tasks into retrieval and information-processing problems rather than ingesting entire media streams. We further characterize their limitations through a failure taxonomy and process-level trace analysis, and show that simple skill injection, including human-written and self-distilled skills, substantially improves performance. To explore open-source elicitation, we introduce Code-X, a training recipe with the OmniCoding trajectory dataset and verifiable reward, and provide baselines on Qwen-3.5-9B and Qwen-3.6-27B. Finally, we argue that the next frontier is many-modality processing, and introduce TerminalBench-O, a process-level benchmark for real-world omnimodal processing tasks. Code will be available at https://github.com/Dongping-Chen/OmniCoding.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 86 of its 107 references that are in the map. First posted on arXiv 30 May 2026.

Builds on

86 of 107 references.

References · 86

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  6. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  7. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  8. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  9. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  10. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  11. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  12. 2025Group Sequence Policy Optimizationcited by 835
38 more
  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  2. 2025Qwen2.5-Omni Technical Reportcited by 736
  3. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  4. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
  5. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  6. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
  7. 2024Qwen2-Audio Technical Reportcited by 560
  8. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  9. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 421
  10. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  11. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
  12. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 315
  13. 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 309
  14. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 292
  15. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  16. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
  17. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 279
  18. 2024AutoCodeRover: Autonomous Program Improvementcited by 278
  19. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
  20. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  21. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
  22. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 191
  23. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  24. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  25. 2025Thyme: Think Beyond Imagescited by 167
  26. 2025Measuring AI Ability to Complete Long Software Taskscited by 156
  27. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
  28. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  29. 2026Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forwardcited by 113
  30. 2025MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Serverscited by 109
  31. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 102
  32. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 102
  33. 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 98
  34. 2025Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?cited by 95
  35. 2025PyVision: Agentic Vision with Dynamic Toolingcited by 79
  36. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Taskscited by 75
  37. 2024MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learningcited by 69
  38. 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agentscited by 64

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Qwen3.8-Omni: Towards Native Omni-Modal Agents
  2. 2026MMPostTrainBench: Benchmarking Autonomous Research for Multimodal Post-Training
  3. 2026GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents
  4. 2026VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation
  5. 2026MMSkills: Towards Multimodal Skills for General Visual Agents
  6. 2026A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications
  7. 2026Orchestra-o1: Omnimodal Agent Orchestration
  8. 2026Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?