# Language-in-the-Loop Culvert Inspection on the Erie Canal

arXiv: https://arxiv.org/abs/2509.21370 · HTML: https://mlsanity.com/paper/2509.21370/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 22 Sep 2025.

## Builds on (most cited first, 22 of 22)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [VCP-CLIP: A visual context prompting model for zero-shot anomaly segmentation](https://mlsanity.com/paper/2407.12276/) (2024)
- [Towards Training-free Anomaly Detection with Vision and Language Foundation Models](https://mlsanity.com/paper/2503.18325/) (2025)
- [Online Next-Best-View Planner for 3D-Exploration and Inspection With a Mobile Manipulator Robot](https://mlsanity.com/paper/2203.10113/) (2022)
- [Show or Tell? Effectively prompting Vision-Language Models for semantic segmentation](https://mlsanity.com/paper/2503.19647/) (2025)
- [Coverage Path Planning for Robotic Quality Inspection with Control on Measurement Uncertainty](https://mlsanity.com/paper/2201.04310/) (2022)
- [Large Language Model-Driven Closed-Loop UAV Operation with Semantic Observations](https://mlsanity.com/paper/2507.01930/) (2025)
- [Visibility-aware Trajectory Optimization with Application to Aerial Tracking](https://mlsanity.com/paper/2103.06742/) (2021)
- [Computationally-Efficient Roadmap-based Inspection Planning via Incremental Lazy Search](https://mlsanity.com/paper/2103.13573/) (2021)
- [Language-Guided Generation for Personalized Inspection Planning](https://mlsanity.com/paper/2506.02917/) (2025)
- [The Field-of-View Constraint of Markers for Mobile Robot with Pan-Tilt Camera](https://mlsanity.com/paper/1909.10682/) (2019)
- [Integrated Guidance and Gimbal Control for Coverage Planning With Visibility Constraints](https://mlsanity.com/paper/2305.12999/) (2023)
- [Take Your Best Shot: Sampling-Based Planning for Autonomous Photography](https://mlsanity.com/paper/2403.05477/) (2024)
- [Beyond Shortsighted Navigation: Merging Best View Trajectory Planning with Robot Navigation](https://mlsanity.com/paper/2408.12513/) (2024)

## Abstract (arXiv)

Culverts on canals such as the Erie Canal, built originally in 1825, require frequent inspections to ensure safe operation. Human inspection of culverts is challenging due to age, geometry, poor illumination, weather, and lack of easy access. We introduce VISION, an end-to-end, language-in-the-loop autonomy system that couples a web-scale vision-language model (VLM) with constrained viewpoint planning for autonomous inspection of culverts. Brief prompts to the VLM solicit open-vocabulary ROI proposals with rationales and confidences, stereo depth is fused to recover scale, and a planner -- aware of culvert constraints -- commands repositioning moves to capture targeted close-ups. Deployed on a quadruped in a culvert under the Erie Canal, VISION closes the see, decide, move, re-image loop on-board and produces high-resolution images for detailed reporting without domain-specific fine-tuning. In an external evaluation by New York Canal Corporation personnel, initial ROI proposals achieved 61.4% agreement with subject-matter experts, and final post-re-imaging assessments reached 80%, indicating that VISION converts tentative hypotheses into grounded, expert-aligned findings.