mlsanityData to 14 Oct 2026

Automated Data Curation Using GPS & NLP to Generate Instruction-Action Pairs for Autonomous Vehicle Vision-Language Navigation Datasets

arXiv 2505.03174 ↗First posted 6 May 2025cs.RO cs.CV cs.LGcited by 2

On the map: End-to-end autonomous driving → Reasoning VLA and VLM driving policies

Abstract (arXiv)

Instruction-Action (IA) data pairs are valuable for training robotic systems, especially autonomous vehicles (AVs), but having humans manually annotate this data is costly and time-inefficient. This paper explores the potential of using mobile application Global Positioning System (GPS) references and Natural Language Processing (NLP) to automatically generate large volumes of IA commands and responses without having a human generate or retroactively tag the data. In our pilot data collection, by driving to various destinations and collecting voice instructions from GPS applications, we demonstrate a means to collect and categorize the diverse sets of instructions, further accompanied by video data to form complete vision-language-action triads. We provide details on our completely automated data collection prototype system, ADVLAT-Engine. We characterize collected GPS voice instructions into eight different classifications, highlighting the breadth of commands and referentialities available for curation from freely available mobile applications. Through research and exploration into the automation of IA data pairs using GPS references, the potential to increase the speed and volume at which high-quality IA datasets are created, while minimizing cost, can pave the way for robust vision-language-action (VLA) models to serve tasks in vision-language navigation (VLN) and human-interactive autonomous systems.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reasoning VLA and VLM driving policies, one of 9 directions in End-to-end autonomous driving. Builds on 20 of its 33 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

20 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigation8 Dec
  2. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving19 Aug

Other references · 18

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  3. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  4. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
  5. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
  6. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  7. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  8. 2022Active Learning on a Budget: Opposite Strategies Suit High and Low Budgetscited by 133
  9. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  10. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 125
  11. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 117
  12. 2024OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Drivingcited by 93
6 more
  1. 2020ViNG: Learning Open-World Navigation with Visual Goalscited by 80
  2. 2020Driver Anomaly Detection: A Dataset and Contrastive Learning Approachcited by 41
  3. 20193D BAT: A Semi-Automatic, Web-based 3D Annotation Toolbox for Full-Surround, Multi-Modal Data Streamscited by 36
  4. 2024The Why, When, and How to Use Active Learning in Large-Data-Driven 3D Object Detection for Safe Autonomous Driving: An Empirical Explorationcited by 17
  5. 2024ActiveAnno3D -- An Active Learning Framework for Multi-Modal 3D Object Detectioncited by 17
  6. 2020An Autonomous Driving Framework for Long-term Decision-making and Short-term Trajectory Planning on Frenet Spacecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Makingcited by 0
  2. 2026doPlan: A Variable-Horizon Dataset for Multi-Stage Language-Conditioned Planning in Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Data-Centric Recursive Self-Improvement for Navigation via Policy-Synchronized Language-Action Dual Bootstrapping
  2. 2026Less Language, More Latents: Annotation-Efficient VLAs for Driving
  3. 2026Vega: Learning to Drive with Natural Language Instructions
  4. 2022Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers
  5. 2026Natural Language Instructions for Scene-Responsive Human-in-the-Loop Motion Planning in Autonomous Driving using Vision-Language-Action Models
  6. 2023Language Prompt for Autonomous Driving
  7. 2025CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Models
  8. 2025A tutorial note on collecting simulated data for vision-language-action models