# V-ICAL Bench: Evaluating Video In-Context Learning for Multimodal Agents in Interactive Environments

arXiv: https://arxiv.org/abs/2609.15683 · HTML: https://mlsanity.com/paper/2609.15683/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 14 Sep 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 25 of 25)

- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [Many-Shot In-Context Learning](https://mlsanity.com/paper/2404.11018/) (2024)
- [Benchmarking the Spectrum of Agent Capabilities](https://mlsanity.com/paper/2109.06780/) (2021)
- [SkillOpt: Executive Strategy for Self-Evolving Agent Skills](https://mlsanity.com/paper/2605.23904/) (2026)
- [Many-Shot In-Context Learning in Multimodal Foundation Models](https://mlsanity.com/paper/2405.09798/) (2024)
- [Seed2.0 Model Card: Towards Intelligence Frontier for Real-World Complexity](https://mlsanity.com/paper/2607.00248/) (2026)
- [lmgame-Bench: How Good are LLMs at Playing Games?](https://mlsanity.com/paper/2505.15146/) (2025)
- [Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding](https://mlsanity.com/paper/2604.05015/) (2026)
- [VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks](https://mlsanity.com/paper/2410.19100/) (2024)
- [VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning](https://mlsanity.com/paper/2403.13164/) (2024)
- [LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations](https://mlsanity.com/paper/2412.01441/) (2024)
- [Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games](https://mlsanity.com/paper/2506.03610/) (2025)
- [GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents](https://mlsanity.com/paper/2604.07429/) (2026)
- [VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understanding](https://mlsanity.com/paper/2412.02186/) (2024)
- [Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Space](https://mlsanity.com/paper/2505.17011/) (2025)
- [RISE-Video: Can Video Generators Decode Implicit World Rules?](https://mlsanity.com/paper/2602.05986/) (2026)
- [Atari-GPT: Benchmarking Multimodal Large Language Models as Low-Level Policies in Atari Games](https://mlsanity.com/paper/2408.15950/) (2024)
- [Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisition](https://mlsanity.com/paper/2602.08439/) (2026)
- [GameVerse: Can Vision-Language Models Learn from Video-based Reflection?](https://mlsanity.com/paper/2603.06656/) (2026)
- [MM-WebAgent: A Hierarchical Multimodal Web Agent for Webpage Generation](https://mlsanity.com/paper/2604.15309/) (2026)
- [Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events](https://mlsanity.com/paper/2606.02522/) (2026)
- [MM-HELIX: Boosting Multimodal Long-Chain Reflective Reasoning with Holistic Platform and Adaptive Hybrid Policy Optimization](https://mlsanity.com/paper/2510.08540/) (2025)

## Abstract (arXiv)

While In-Context Learning (ICL) enables models to adapt from exemplars without parameter updates, multimodal ICL remains largely underexplored, particularly regarding video demonstrations in interactive environments. For multimodal agents, learning from videos presents unique challenges: they must translate in-context demonstrations into executable policies, ground these policies in novel visual states, and iteratively refine actions based on environmental feedback. We introduce V-ICAL, a novel benchmark designed to evaluate video-based ICL for multimodal agents. Comprising 342 interactive tasks across 37 environments, V-ICAL utilizes human-curated demonstration videos as task-specific behavioral exemplars, evaluating agents through sustained interaction from a target initialization. The benchmark seamlessly connects in-context knowledge induction with core agentic capabilities, including state grounding, temporal memory, planning, and adaptation in dynamic environments. Extensive evaluations across 19 state-of-the-art multimodal agents reveal significant limitations: the best-performing model, Seed-2.1-Pro, achieves a score of only 54.4/100, while other leading models (e.g., Gemini-3.1-Pro, GPT-5.6) fail to surpass 50, far below the human baseline of 83.6. Controlled comparisons further demonstrate that current agents struggle to reliably translate video exemplars into effective policies, failing to yield consistent performance gains. Ultimately, V-ICAL exposes a critical gap in the ICL capabilities of multimodal agents, underscoring an urgent need for future research.