# RECAST: Recasting Vision-Language Semantics into an Actionable Cost Map for Robot Navigation

arXiv: https://arxiv.org/abs/2609.32595 · HTML: https://mlsanity.com/paper/2609.32595/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 44 of its 45 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 30 of 44)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [YOLO-World: Real-Time Open-Vocabulary Object Detection](https://mlsanity.com/paper/2401.17270/) (2024)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [GPT-Driver: Learning to Drive with GPT](https://mlsanity.com/paper/2310.01415/) (2023)
- [DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2411.15139/) (2024)
- [NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration](https://mlsanity.com/paper/2310.07896/) (2023)
- [NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation](https://mlsanity.com/paper/2402.15852/) (2024)
- [ViNT: A Foundation Model for Visual Navigation](https://mlsanity.com/paper/2306.14846/) (2023)
- [NaVILA: Legged Robot Vision-Language-Action Model for Navigation](https://mlsanity.com/paper/2412.04453/) (2024)
- [GNM: A General Navigation Model to Drive Any Robot](https://mlsanity.com/paper/2210.03370/) (2022)
- [Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks](https://mlsanity.com/paper/2412.06224/) (2024)
- [Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigation](https://mlsanity.com/paper/2203.15041/) (2022)
- [StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling](https://mlsanity.com/paper/2507.05240/) (2025)
- [Navigating to Objects in the Real World](https://mlsanity.com/paper/2212.00922/) (2022)
- [Gated-Attention Architectures for Task-Oriented Language Grounding](https://mlsanity.com/paper/1706.07230/) (2017)
- [Embodied Navigation Foundation Model](https://mlsanity.com/paper/2509.12129/) (2025)
- [Ground Slow, Move Fast: A Dual-System Foundation Model for Generalizable Vision-and-Language Navigation](https://mlsanity.com/paper/2512.08186/) (2025)
- [SACSoN: Scalable Autonomous Control for Social Navigation](https://mlsanity.com/paper/2306.01874/) (2023)
- [Pushing the Limits of Cross-Embodiment Learning for Manipulation and Navigation](https://mlsanity.com/paper/2402.19432/) (2024)
- [QUAR-VLA: Vision-Language-Action Model for Quadruped Robots](https://mlsanity.com/paper/2312.14457/) (2023)
- [DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving](https://mlsanity.com/paper/2512.07745/) (2025)
- [NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidance](https://mlsanity.com/paper/2505.08712/) (2025)
- [TrackVLA: Embodied Visual Tracking in the Wild](https://mlsanity.com/paper/2505.23189/) (2025)
- [VLM-Social-Nav: Socially Aware Robot Navigation through Scoring using Vision-Language Models](https://mlsanity.com/paper/2404.00210/) (2024)
- [Environment-agnostic Multitask Learning for Natural Language Grounded Navigation](https://mlsanity.com/paper/2003.00443/) (2020)
- [CoNVOI: Context-aware Navigation using Vision Language Models in Outdoor and Indoor Environments](https://mlsanity.com/paper/2403.15637/) (2024)
- [LeLaN: Learning A Language-Conditioned Navigation Policy from In-the-Wild Videos](https://mlsanity.com/paper/2410.03603/) (2024)

## Abstract (arXiv)

Safe and robust robot navigation across diverse environments requires a high-level understanding of complex scenes and the ability to carry it into stable motion. Recent works tackle this with learning-based models trained at scale and with approaches built on vision-language models (VLMs). However, learning-based models break down outside their training distribution, while VLM-based approaches bring that understanding but rarely ground it in the scene or align the action with it. To address these limitations, we present RECAST, a robot navigation framework that combines the reasoning of a VLM with the spatial grounding of vision foundation models to build an Actionable Cost map. Given the robot's front view and the user's instruction, we first decompose the scene with the VLM, judging which surfaces are traversable, which objects pose a risk, which heading to prefer, and which gaps are passable. Vision foundation models then ground these surfaces and objects in the image, and all four judgments are spatially recast into one compact cost map. This map both conditions the trajectory decoders and scores their proposals to select the one to execute. As the VLM's answers trail the live scene, both steps draw on cost maps from two points in time: the pivot frame the VLM judged, which carries all four judgments, and the current frame, whose terrain and collision costs are rebuilt from the current image. RECAST improves success over the strongest prior method by 13.3 points in simulation and 31.4 points on a real quadruped, and reduces the collision rate relative to it by 9.6 and 14.3 points, reaching the lowest collision rate among all methods. The project page is available at https://recast-nav.github.io/