mlsanityData to 14 Oct 2026

SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning

arXiv 2603.23483 ↗First posted 24 Mar 2026cs.CV cs.CLcited by 1

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

Agentic multimodal large language models (MLLMs) (e.g., OpenAI o3 and Gemini Agentic Vision) achieve remarkable reasoning capabilities through iterative visual tool invocation. However, the cascaded perception, reasoning, and tool-calling loops introduce significant sequential overhead. This overhead, termed agentic depth, incurs prohibitive latency and seriously limits system-level concurrency. To this end, we propose SpecEyes, an agentic-level speculative acceleration framework that breaks this sequential bottleneck. Our key insight is that a lightweight, tool-free MLLM can serve as a speculative planner to predict the execution trajectory, enabling early termination of expensive tool chains without sacrificing accuracy. To regulate this speculative planning, we introduce a cognitive gating mechanism based on answer separability, which quantifies the model's confidence for self-verification without requiring oracle labels. Furthermore, we design a heterogeneous parallel funnel that exploits the stateless concurrency of the small model to mask the stateful serial execution of the large model, maximizing system throughput. Extensive experiments on V* Bench, HR-Bench, and POPE demonstrate that SpecEyes achieves 1.1-3.35x speedup over the agentic baseline while preserving or even improving accuracy (up to +6.7%), thereby boosting serving throughput under concurrent workloads.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 70 of its 78 references that are in the map. First posted on arXiv 24 Mar 2026.

Builds on

70 of 78 references.

References · 70

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2023Qwen Technical Reportcited by 3,344
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  10. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  11. 2022Token Merging: Your ViT But Fastercited by 1,021
  12. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
38 more
  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  2. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  3. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  4. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  5. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  6. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 410
  7. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  8. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  9. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  10. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  11. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 294
  12. 2025Deep Think with Confidencecited by 205
  13. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 171
  14. 2025Thyme: Think Beyond Imagescited by 167
  15. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 161
  16. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 138
  17. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 125
  18. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 125
  19. 2025Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Searchcited by 120
  20. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 96
  21. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 95
  22. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 72
  23. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  24. 2025Skywork R1V: Pioneering Multimodal Reasoning with Chain-of-Thoughtcited by 69
  25. 2025SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoningcited by 59
  26. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Modelscited by 54
  27. 2024Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Accelerationcited by 41
  28. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  29. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Groundingcited by 33
  30. 2025SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learningcited by 33
  31. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inferencecited by 31
  32. 2025Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearchcited by 27
  33. 2025CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimizationcited by 26
  34. 2024Efficient Inference of Vision Instruction-Following Models with Elastic Cachecited by 24
  35. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exitingcited by 23
  36. 2025QuoTA: Query-oriented Token Assignment via CoT Query Decouple for Long Video Comprehensioncited by 21
  37. 2025CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoningcited by 16
  38. 2026PyVision-RL: Forging Open Agentic Vision Models via RLcited by 16

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SWE-AGILE: A Software Agent Framework for Efficiently Managing Dynamic Reasoning Contextcited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2026SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference
  2. 2026IdleSpec: Exploiting Idle Time via Speculative Planning for LLM Agents
  3. 2025Towards Efficient Agents: A Co-Design of Inference Architecture and System
  4. 2025FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoning
  5. 2026Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?
  6. 2025ToolScope: An Agentic Framework for Vision-Guided and Long-Horizon Tool Use
  7. 2026ATP-Bench: Towards Agentic Tool Planning for MLLM Interleaved Generation
  8. 2026AgentSpec: Speculative Decoding for Batch Inference of LLM Agents