# STaR: Scalable Task-Conditioned Retrieval for Long-Horizon Multimodal Robot Memory

arXiv: https://arxiv.org/abs/2602.09255 · HTML: https://mlsanity.com/paper/2602.09255/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 19 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Builds on (most cited first, 16 of 16)

- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023)
- [Recognize Anything: A Strong Image Tagging Model](https://mlsanity.com/paper/2306.03514/) (2023)
- [NVILA: Efficient Frontier Visual Language Models](https://mlsanity.com/paper/2412.04468/) (2024)
- [Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation](https://mlsanity.com/paper/2403.17846/) (2024)
- [HourVideo: 1-Hour Video-Language Understanding](https://mlsanity.com/paper/2411.04998/) (2024)
- [Embodied Question Answering in Photorealistic Environments with Point Cloud Perception](https://mlsanity.com/paper/1904.03461/) (2019)
- [T*: Re-thinking Temporal Search for Long-Form Video Understanding](https://mlsanity.com/paper/2504.02259/) (2025)
- [Clio: Real-time Task-Driven Open-Set 3D Scene Graphs](https://mlsanity.com/paper/2404.13696/) (2024)
- [ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigation](https://mlsanity.com/paper/2409.13682/) (2024)
- [Khronos: A Unified Approach for Spatio-Temporal Metric-Semantic SLAM in Dynamic Environments](https://mlsanity.com/paper/2402.13817/) (2024)
- [Receding Moving Object Segmentation in 3D LiDAR Data Using Sparse 4D Convolutions](https://mlsanity.com/paper/2206.04129/) (2022)
- [Tokenize Anything via Prompting](https://mlsanity.com/paper/2312.09128/) (2023)
- [Towards Robust Robot 3D Perception in Urban Environments: The UT Campus Object Dataset](https://mlsanity.com/paper/2309.13549/) (2023)
- [OpenGraph: Open-Vocabulary Hierarchical 3D Graph Representation in Large-Scale Outdoor Environments](https://mlsanity.com/paper/2403.09412/) (2024)
- [ASHiTA: Automatic Scene-grounded HIerarchical Task Analysis](https://mlsanity.com/paper/2504.06553/) (2025)

## Abstract (arXiv)

Mobile robots are often deployed over long durations in diverse open, dynamic scenes, including indoor setting such as warehouses and manufacturing facilities, and outdoor settings such as agricultural and roadway operations. A core challenge is to build a scalable long-horizon memory that supports an agentic workflow for planning, retrieval, and reasoning over open-ended instructions at variable granularity, while producing precise, actionable answers for navigation. We present STaR, an agentic reasoning framework that (i) constructs a task-agnostic, multimodal long-term memory that generalizes to unseen queries while preserving fine-grained environmental semantics (object attributes, spatial relations, and dynamic events), and (ii) introduces a Scalable Task Conditioned Retrieval algorithm based on the Information Bottleneck principle to extract from long-term memory a compact, non-redundant, information-rich set of candidate memories for contextual reasoning. We evaluate STaR on NaVQA (mixed indoor/outdoor campus scenes) and WH-VQA, a customized warehouse benchmark with many visually similar objects built with Isaac Sim, emphasizing contextual reasoning. Across the two datasets, STaR consistently outperforms strong baselines, achieving higher success rates and markedly lower spatial error. We further deploy STaR on a real Husky wheeled robot in both indoor and outdoor environments, demonstrating robust long horizon reasoning, scalability, and practical utility. Project Website: https://trailab.github.io/STaR-website/