mlsanityData to 14 Oct 2026

Understanding Pedestrian Gesture Misrecognition: Insights from Vision-Language Model Reasoning

arXiv 2508.06801 ↗First posted 9 Aug 2025cs.HCcited by 0

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

Pedestrian gestures play an important role in traffic communication, particularly in interactions with autonomous vehicles (AVs), yet their subtle, ambiguous, and context-dependent nature poses persistent challenges for machine interpretation. This study investigates these challenges by using GPT-4V, a vision-language model, not as a performance benchmark but as a diagnostic tool to reveal patterns and causes of gesture misrecognition. We analysed a public dataset of pedestrian-vehicle interactions, combining manual video review with thematic analysis of the model's qualitative reasoning. This dual approach surfaced recurring factors influencing misrecognition, including gesture visibility, pedestrian behaviour, interaction context, and environmental conditions. The findings suggest practical considerations for gesture design, including the value of salience and contextual redundancy, and highlight opportunities to improve AV recognition systems through richer context modelling and uncertainty-aware interpretations. While centred on AV-pedestrian interaction, the method and insights are applicable to other domains where machines interpret human gestures, such as wearable AR and assistive technologies.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 5 of its 25 references that are in the map. First posted on arXiv 9 Aug 2025.

Builds on

5 of 25 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Can Vision-Language Models Understand and Interpret Dynamic Gestures from Pedestrians? Pilot Datasets and Exploration Towards Instructive Nonverbal Commands for Cooperative Autonomous Vehicles15 Apr

Other references · 4

  1. 2023GPT-4V Takes the Wheel: Promises and Challenges for Pedestrian Behavior Predictioncited by 11
  2. 2023Exploring the Potential of Multi-Modal AI for Driving Hazard Predictioncited by 4
  3. 2024Pedestrian-Vehicle Interaction in Shared Space: Insights for Autonomous Vehiclescited by 3
  4. 2024ReStory: VLM-augmentation of Social Human-Robot Interaction Datasetscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Pedestrian Intention Prediction via Vision-Language Foundation Models
  2. 2026PEDESTRIANQA: A Benchmark for Vision-Language Models on Pedestrian Intention and Trajectory Prediction
  3. 2026Decoding Pedestrian Crossing Intention from Egocentric Vision via Vision Language Models
  4. 2025Application of Vision-Language Model to Pedestrians Behavior and Scene Understanding in Autonomous Driving
  5. 2024doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigation
  6. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving
  7. 2024Embodied Understanding of Driving Scenarios
  8. 2026SafeGesture: Evaluating Fine-Grained Hand Gesture Understanding in Vision-Language Models through Scenario-Conditioned Safety Interpretation