mlsanityData to 7 Oct 2026

Robusto-2: Benchmarking Humans & VLMs for Autonomous Driving in Lima & New York City

arXiv 2606.20980 ↗First posted 18 Jun 2026cs.CV cs.AI cs.ROcited by 0

On the map: End-to-end autonomous driving → VLMs for autonomous driving

Abstract (arXiv)

As Self-Driving Cars continue to expand internationally and use multi-modal systems such as VLMs as a cognitive backbone for their Action models; how well will these systems generalize in new settings, in particular out-of-distribution (OOD) edge-case scenarios in new geographies? In this paper, we study this open question by providing a full factorial analysis with human drivers of Lima, human drivers from New York City, and VLMs and showing them dashcam footage collected from Lima and New York City -- prompting them with a variety of questions under a Visual Question Answering (VQA) paradigm. In particular, we pick these two cities as they are highly challenging driving locations where no Self-Driving Car company currently operates in, and ask questions that span 4 categories: Factual, Ratings, Counterfactual and Reasoning. We find that Humans and VLMs diverge in their responses -- though this is modulated by the type of questions asked, and that Humans answer similarly independent of where they are from (Lima/NYC). To our surprise, we did not find a strong difference in terms of answers (Humans or VLMs) that was modulated by geography, likely due to their high out-of-distribution nature. Our dataset is available at: https://huggingface.co/datasets/Artificio/robusto-2

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving. Builds on 21 of its 52 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

21 of 52 references.

References · 21

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2025Qwen3-VL Technical Reportcited by 2,534
  3. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  4. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  5. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  6. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  7. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  8. 2024The Platonic Representation Hypothesiscited by 362
  9. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  10. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  11. 2026MIRAGE: The Illusion of Visual Understandingcited by 57
  12. 2025Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgettingcited by 52
9 more
  1. 2025Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrationscited by 15
  2. 2026Fail2Drive: Benchmarking Closed-Loop Driving Generalizationcited by 15
  3. 2026SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarioscited by 11
  4. 2025RealEngine: Simulating Autonomous Driving in Realistic Contextcited by 10
  5. 2022Joint rotational invariance and adversarial training of a dual-stream Transformer yields state of the art Brain-Score for Area V4cited by 7
  6. 2025Reinforcing VLMs to Use Tools for Detailed Visual Reasoning Under Resource Constraintscited by 6
  7. 2023Stress and Adaptation: Applying Anna Karenina Principle in Deep Learning for Image Classificationcited by 2
  8. 2025Uncovering Bias in Foundation Models: Impact, Testing, Harm, and Mitigationcited by 2
  9. 2025Are Vision Language Models Cross-Cultural Theory of Mind Reasoners?cited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Robusto-1 Dataset: Comparing Humans and VLMs on real out-of-distribution Autonomous Driving VQA from Peru
  2. 2025AutoDrive-QA: A Multiple-Choice Benchmark for Vision-Language Evaluation in Urban Autonomous Driving
  3. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives
  4. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving
  5. 2025VLMs Guided Interpretable Decision Making for Autonomous Driving
  6. 2023DriveLM: Driving with Graph Visual Question Answering
  7. 2025LAVQA: A Latency-Aware Visual Question Answering Framework for Shared Autonomy in Self-Driving Vehicles
  8. 2025STRIDE-QA: Visual Question Answering Dataset for Spatiotemporal Reasoning in Urban Driving Scenes