# Seek to Segment: Active Perception for Panoramic Referring Segmentation

arXiv: https://arxiv.org/abs/2607.02497 · HTML: https://mlsanity.com/paper/2607.02497/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 55 of its 70 references that are in the map.
- First posted on arXiv 2 Jul 2026.

## Builds on (most cited first, 30 of 55)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [CRIS: CLIP-Driven Referring Image Segmentation](https://mlsanity.com/paper/2111.15174/) (2021)
- [REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments](https://mlsanity.com/paper/1904.10151/) (2019)
- [Segmentation from Natural Language Expressions](https://mlsanity.com/paper/1603.06180/) (2016)
- [GRES: Generalized Referring Expression Segmentation](https://mlsanity.com/paper/2306.00968/) (2023)
- [Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2108.05565/) (2021)
- [Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement](https://mlsanity.com/paper/2503.06520/) (2025)
- [MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions](https://mlsanity.com/paper/2308.08544/) (2023)
- [GSVA: Generalized Segmentation via Multimodal Large Language Models](https://mlsanity.com/paper/2312.10103/) (2023)
- [Memory OS of AI Agent](https://mlsanity.com/paper/2506.06326/) (2025)
- [Recurrent Multimodal Interaction for Referring Image Segmentation](https://mlsanity.com/paper/1703.07939/) (2017)
- [VLT: Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2210.15871/) (2022)
- [3D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunities](https://mlsanity.com/paper/2407.17418/) (2024)
- [SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation](https://mlsanity.com/paper/2409.10542/) (2024)
- [Kernel Transformer Networks for Compact Spherical Convolution](https://mlsanity.com/paper/1812.03115/) (2018)
- [A Survey of Multimodal-Guided Image Editing with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2406.14555/) (2024)
- [VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models](https://mlsanity.com/paper/2511.11007/) (2025)
- [SegPoint: Segment Any Point Cloud via Large Language Model](https://mlsanity.com/paper/2407.13761/) (2024)

## Abstract (arXiv)

Existing referring segmentation models passively process static images captured from fixed perspectives, limiting their applicability in Embodied AI, where agents must perform active perception in the continuous 360^circ environments. To bridge this gap, we introduce a novel task: Active Panoramic Referring Segmentation (APRS). In this setting, an agent is required to adjust its viewing direction (Deltaθ, Deltaφ) to explore the 360^circ environment, seeking the object specified by a user instruction for segmentation. To tackle this challenging task, we propose PanoSeeker, a memory-augmented agent for efficient APRS. Rather than relying on heuristic scanning, PanoSeeker integrates a Vision-Language Model (VLM) with EgoSphere, an explicit spatial visual memory. By progressively integrating sequential local observations into a unified 360^circ representation, EgoSphere enables the agent to plan efficient and non-redundant search trajectories. Once the target is found, the agent performs active viewpoint alignment and outputs the segmentation mask. Furthermore, we curate an expert-annotated search trajectory dataset with memory timelines for Supervised Fine-Tuning, followed by Reinforcement Learning post-training to explicitly optimize PanoSeeker's exploration efficiency. Extensive experiments on our newly established APRS benchmark demonstrate that PanoSeeker achieves superior search efficiency and segmentation accuracy, significantly outperforming adapted state-of-the-art baselines.