mlsanityData to 14 Oct 2026

UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spaces

arXiv 2503.06157 ↗First posted 8 Mar 2025cs.CV cs.AIcited by 36

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large multimodal models exhibit remarkable intelligence, yet their embodied cognitive abilities during motion in open-ended urban 3D space remain to be explored. We introduce a benchmark to evaluate whether video-large language models (Video-LLMs) can naturally process continuous first-person visual observations like humans, enabling recall, perception, reasoning, and navigation. We have manually control drones to collect 3D embodied motion video data from real-world cities and simulated environments, resulting in 1.5k video clips. Then we design a pipeline to generate 5.2k multiple-choice questions. Evaluations of 17 widely-used Video-LLMs reveal current limitations in urban embodied cognition. Correlation analysis provides insight into the relationships between different tasks, showing that causal reasoning has a strong correlation with recall, perception, and navigation, while the abilities for counterfactual and associative reasoning exhibit lower correlation with other tasks. We also validate the potential for Sim-to-Real transfer in urban embodiment through fine-tuning.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 83 references that are in the map. First posted on arXiv 8 Mar 2025.

Builds on

45 of 83 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometrics17 Feb
  2. 2025ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark9 Jan
  3. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  4. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI15 Oct
  5. 2024AeroVerse: UAV-Agent Benchmark Suite for Simulating, Pre-training, Finetuning, and Evaluating Aerospace Embodied World Models28 Aug

Other references · 40

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  7. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  8. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  9. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  10. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  11. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  12. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
28 more
  1. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
  2. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  3. 2020Detection and Tracking Meet Drones Challengecited by 263
  4. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  5. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  6. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 182
  7. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 179
  8. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  9. 2021Embodied Intelligence via Learning and Evolutioncited by 163
  10. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 145
  11. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 134
  12. 2023AerialVLN: Vision-and-Language Navigation for UAVscited by 121
  13. 2024HourVideo: 1-Hour Video-Language Understandingcited by 113
  14. 2020Human-centric Spatio-Temporal Video Grounding With Visual Transformerscited by 85
  15. 2024Does Spatial Cognition Emerge in Frontier Models?cited by 66
  16. 2024V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuningcited by 46
  17. 2024EmbodiedCity: A Benchmark Platform for Embodied Agent in Real-world City Environmentcited by 42
  18. 2023Urban Generative Intelligence (UGI): A Foundational Platform for Agents in Embodied City Environmentcited by 37
  19. 2021Detection, Tracking, and Counting Meets Drones in Crowds: A Benchmarkcited by 31
  20. 2024VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMscited by 29
  21. 2024Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructionscited by 25
  22. 2024AlanaVLM: A Multimodal Embodied AI Foundation Model for Egocentric Video Understandingcited by 23
  23. 2024MM-SAP: A Comprehensive Benchmark for Assessing Self-Awareness of Multimodal Large Language Models in Perceptioncited by 21
  24. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorialcited by 18
  25. 2024OPEx: A Component-Wise Analysis of LLM-Centric Agents in Embodied Instruction Followingcited by 14
  26. 2024MR-COGraphs: Communication-efficient Multi-Robot Open-vocabulary Mapping System via 3D Scene Graphscited by 5
  27. 2025Understanding and Evaluating Hallucinations in 3D Visual Language Modelscited by 5
  28. 2024Exploring Efficient Foundational Multi-modal Models for Video Summarizationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 19.

36 papers.

Sort:

By area:

  1. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 410
  2. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  3. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 63
  4. 2025Open3D-VQA: A Benchmark for Comprehensive Spatial Reasoning with Multimodal Large Language Model in Open Spacecited by 39
  5. 2025Embodied-R: Collaborative Framework for Activating Embodied Spatial Reasoning in Foundation Models via Reinforcement Learningcited by 37
  6. 2025Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimizationcited by 20
  7. 2025AirCopBench: A Benchmark for Multi-drone Collaborative Embodied Perception and Reasoningcited by 20
  8. 2025UAVBench: An Open Benchmark Dataset for Autonomous and Agentic AI UAV Systems via LLM-Generated Flight Scenarioscited by 13
  9. 2026CityCube: Benchmarking Cross-view Spatial Reasoning on Vision-Language Models in Urban Environmentscited by 11
  10. 2025MM-UAVBench: How Well Do Multimodal Large Language Models See, Think, and Plan in Low-Altitude UAV Scenarios?cited by 10
  11. 2025AirScape: An Aerial Generative World Model with Motion Controllabilitycited by 7
  12. 2025Event Camera Meets Mobile Embodied Perception: Abstraction, Algorithm, Acceleration, Applicationcited by 6
24 more
  1. 2026Self in Space: Benchmarking Self-Awareness and Spatial Cognition in UAV Embodied Intelligencecited by 6
  2. 2025USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning of LLMs as Urban Agentscited by 5
  3. 2025EgoPrune: Efficient Token Pruning for Egomotion Video Reasoning in Embodied Agentcited by 4
  4. 2026Are VLMs Lost Between Sky and Space? LinkS2Bench for UAV-Satellite Dynamic Cross-View Spatial Intelligencecited by 3
  5. 2025Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Studycited by 2
  6. 2026Are Video Reasoning Models Ready to Go Outside?cited by 2
  7. 2026SpatialUAV: Benchmarking Spatial Intelligence for Low-Altitude UAV Perception, Collaboration, and Motioncited by 2
  8. 2026Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMscited by 2
  9. 2025JuggleRL: Mastering Ball Juggling with a Quadrotor via Deep Reinforcement Learningcited by 1
  10. 2025OceanGym: A Benchmark Environment for Underwater Embodied Agentscited by 1
  11. 2026EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoningcited by 1
  12. 20263DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understandingcited by 1
  13. 2026ViCoStream: Streaming VideoLLMs Can Run Beyond 100 FPS with Stage-Wise Coordinated Inferencecited by 1
  14. 2026Confidence-Aware Tool Orchestration for Robust Video Understandingcited by 1
  15. 2026Parse, Search, and Confirmation: Training-Free Aerial Vision-and-Dialog Navigation with Chain-of-Thought Reasoning and Structured Spatial Memorycited by 1
  16. 2026Advancing MLLM-based UAV Image Understanding and Reasoning: A Benchmark and a Training-Free Multi-Agent Systemcited by 1
  17. 2025STeP-Diff: Spatio-Temporal Physics-Informed Diffusion Models for Mobile Fine-Grained Pollution Forecastingcited by 0
  18. 2026How Far Are Large Multimodal Models from Human-Level Spatial Action? A Benchmark for Goal-Oriented Embodied Navigation in Urban Airspacecited by 0
  19. 2026Efficient Spatial-Temporal Focal Adapter with SSM for Temporal Action Detectioncited by 0
  20. 2026MulRobBench: A Decision-Level Benchmark for Safe and Security-Policy-Compliant Multimodal UAV Agentscited by 0
  21. 2026MetaSpace: Metamorphic Testing for Spatial Cognition in Embodied Agentscited by 0
  22. 2026Lingjing: A Simulation Testbed for Multi-Agent Embodied Tasks in Open-Ended Citiescited by 0
  23. 2026Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searchingcited by 0
  24. 2026The Gap Is in the Geometry: A Convergent Meta-Analysis of Multi-Facet Aerial and Remote Sensing Video Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025REM: Evaluating LLM Embodied Spatial Reasoning through Multi-Frame Trajectories
  2. 2025RynnEC: Bringing MLLMs into Embodied World
  3. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models
  4. 2025Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents
  5. 2025ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction
  6. 2026Thinker: A vision-language foundation model for embodied intelligence
  7. 2025BLM1: A Boundless Large Model for Cross-Space, Cross-Task, and Cross-Embodiment Learning
  8. 2025FindingDory: A Benchmark to Evaluate Memory in Embodied Agents