# OPERA: A Unified Omnimodal Progressive Spatio-Temporal Reasoning Agent for Referring Video Segmentation

arXiv: https://arxiv.org/abs/2609.33338 · HTML: https://mlsanity.com/paper/2609.33338/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 22 of its 57 references that are in the map.
- First posted on arXiv 27 Sep 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 22 of 22)

- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment](https://mlsanity.com/paper/2310.01852/) (2023)
- [Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement](https://mlsanity.com/paper/2503.06520/) (2025)
- [Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos](https://mlsanity.com/paper/2501.04001/) (2025)
- [Medical SAM 2: Segment medical images as video via Segment Anything Model 2](https://mlsanity.com/paper/2408.00874/) (2024)
- [R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learning](https://mlsanity.com/paper/2503.05379/) (2025)
- [Natural-Language Agent Harnesses](https://mlsanity.com/paper/2603.25723/) (2026)
- [Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Training](https://mlsanity.com/paper/2508.00414/) (2025)
- [Tracking with Human-Intent Reasoning](https://mlsanity.com/paper/2312.17448/) (2023)
- [ViLLa: Video Reasoning Segmentation with Large Language Model](https://mlsanity.com/paper/2407.14500/) (2024)
- [Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation](https://mlsanity.com/paper/2510.24821/) (2025)
- [MC-CoT: A Modular Collaborative CoT Framework for Zero-shot Medical-VQA with LLM and MLLM Integration](https://mlsanity.com/paper/2410.04521/) (2024)
- [AdaMem: Adaptive User-Centric Memory for Long-Horizon Dialogue Agents](https://mlsanity.com/paper/2603.16496/) (2026)
- [What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Study](https://mlsanity.com/paper/2601.14888/) (2026)
- [VG-Refiner: Towards Tool-Refined Referring Grounded Reasoning via Agentic Reinforcement Learning](https://mlsanity.com/paper/2512.06373/) (2025)
- [DeskCraft: Benchmarking Desktop Agents on Professional Workflows and Human-in-the-Loop Collaboration](https://mlsanity.com/paper/2606.03103/) (2026)
- [Cascaded Self-Evaluation Augmented Training for Lightweight Multimodal LLMs](https://mlsanity.com/paper/2501.05662/) (2025)
- [SimSD: Simple Speculative Decoding in Diffusion Language Models](https://mlsanity.com/paper/2606.02544/) (2026)

## Abstract (arXiv)

Referring video segmentation with heterogeneous multimodal queries---spanning text, audio, and reference images---demands both robust cross-modal understanding and precise spatio-temporal reasoning. We propose OPERA (Omnimodal Progressive spatio-tEmporal Reasoning Agent), a unified reasoning agent built on a single MLLM that performs dual-axis progressive reasoning via three specialized stages. Along the temporal axis, a Temporal Reasoning Agent narrows the frame search space through coarse-to-fine filtering to identify the most informative key frame. Along the spatial axis, a Distillation Agent establishes what to locate via cross-modal semantic distillation, and a Grounding Agent enhanced with GRPO determines where the target appears, with dense mask propagation completing the pixel-level output. OPERA sets a new state of the art on OmniAVS and Ref-AVS and transfers zero-shot to standard referring video segmentation benchmarks.