# Opportunistic Collaborative Planning with Large Vision Model Guided Control and Joint Query-Service Optimization

arXiv: https://arxiv.org/abs/2504.18057 · HTML: https://mlsanity.com/paper/2504.18057/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 25 Apr 2025.

## Builds on (most cited first, 17 of 17)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023)
- [LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving](https://mlsanity.com/paper/2310.03026/) (2023)
- [AgentsCoDriver: Large Language Model Empowered Collaborative Driving with Lifelong Learning](https://mlsanity.com/paper/2404.06345/) (2024)
- [Unknown Sniffer for Object Detection: Don't Turn a Blind Eye to Unknown Objects](https://mlsanity.com/paper/2303.13769/) (2023)
- [EPSILON: An Efficient Planning System for Automated Vehicles in Highly Interactive Environments](https://mlsanity.com/paper/2108.07993/) (2021)
- [BEVGPT: Generative Pre-trained Large Model for Autonomous Driving Prediction, Decision-Making, and Planning](https://mlsanity.com/paper/2310.10357/) (2023)
- [RDA: An Accelerated Collision Free Motion Planner for Autonomous Navigation in Cluttered Environments](https://mlsanity.com/paper/2210.00192/) (2022)
- [NeuPAN: Direct Point Robot Navigation with End-to-End Model-based Learning](https://mlsanity.com/paper/2403.06828/) (2024)
- [An Efficient Spatial-Temporal Trajectory Planner for Autonomous Vehicles in Unstructured Environments](https://mlsanity.com/paper/2208.13160/) (2022)
- [FogROS2: An Adaptive Platform for Cloud and Fog Robotics Using ROS 2](https://mlsanity.com/paper/2205.09778/) (2022)
- [Planning for robotic exploration based on forward simulation](https://mlsanity.com/paper/1502.02474/) (2015)
- [Edge Accelerated Robot Navigation With Collaborative Motion Planning](https://mlsanity.com/paper/2311.08983/) (2023)
- [Enhancing Large Vision Model in Street Scene Semantic Understanding through Leveraging Posterior Optimization Trajectory](https://mlsanity.com/paper/2501.01710/) (2025)
- [Multi-Uncertainty Aware Autonomous Cooperative Planning](https://mlsanity.com/paper/2411.00413/) (2024)

## Built on by (most cited first, 2 of 2)

- [Bridging Large-Model Reasoning and Real-Time Control via Agentic Fast-Slow Planning](https://mlsanity.com/paper/2604.01681/) (2026)
- [Event-Adaptive Motion Planning with Distilled Vision-Language Model in Safety-Critical Situations](https://mlsanity.com/paper/2606.25629/) (2026)

## Abstract (arXiv)

Navigating autonomous vehicles in open scenarios is a challenge due to the difficulties in handling unseen objects. Existing solutions either rely on small models that struggle with generalization or large models that are resource-intensive. While collaboration between the two offers a promising solution, the key challenge is deciding when and how to engage the large model. To address this issue, this paper proposes opportunistic collaborative planning (OCP), which seamlessly integrates efficient local models with powerful cloud models through two key innovations. First, we propose large vision model guided model predictive control (LVM-MPC), which leverages the cloud for LVM perception and decision making. The cloud output serves as a global guidance for a local MPC, thereby forming a closed-loop perception-to-control system. Second, to determine the best timing for large model query and service, we propose collaboration timing optimization (CTO), including object detection confidence thresholding (ODCT) and cloud forward simulation (CFS), to decide when to seek cloud assistance and when to offer cloud service. Extensive experiments show that the proposed OCP outperforms existing methods in terms of both navigation time and success rate.