# Seeing Beyond Frames: Zero-Shot Pedestrian Intention Prediction with Raw Temporal Video and Multimodal Cues

arXiv: https://arxiv.org/abs/2507.21161 · HTML: https://mlsanity.com/paper/2507.21161/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 25 Jul 2025.

## Builds on (most cited first, 13 of 13)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective](https://mlsanity.com/paper/2305.15408/) (2023)
- [Role-Play with Large Language Models](https://mlsanity.com/paper/2305.16367/) (2023)
- [Vision Language Models in Autonomous Driving: A Survey and Outlook](https://mlsanity.com/paper/2310.14414/) (2023)
- [Long-Term On-Board Prediction of People in Traffic Scenes under Uncertainty](https://mlsanity.com/paper/1711.09026/) (2017)
- [Pedestrian Action Anticipation using Contextual Feature Fusion in Stacked RNNs](https://mlsanity.com/paper/2005.06582/) (2020)
- [Predicting Pedestrian Crossing Intention with Feature Fusion and Spatio-Temporal Attention](https://mlsanity.com/paper/2104.05485/) (2021)
- [Multi-Modal Hybrid Architecture for Pedestrian Action Prediction](https://mlsanity.com/paper/2012.00514/) (2020)
- [IntFormer: Predicting pedestrian intention with the aid of the Transformer architecture](https://mlsanity.com/paper/2105.08647/) (2021)
- [Causal reasoning in typical computer vision tasks](https://mlsanity.com/paper/2307.13992/) (2023)
- [GPT-4V Takes the Wheel: Promises and Challenges for Pedestrian Behavior Prediction](https://mlsanity.com/paper/2311.14786/) (2023)

## Built on by (most cited first, 2 of 2)

- [Decoding Pedestrian Crossing Intention from Egocentric Vision via Vision Language Models](https://mlsanity.com/paper/2606.09142/) (2026)
- [Pedestrian Crossing Intent Prediction via Psychological Features and Transformer Fusion](https://mlsanity.com/paper/2603.19533/) (2026)

## Abstract (arXiv)

Pedestrian intention prediction is essential for autonomous driving in complex urban environments. Conventional approaches depend on supervised learning over frame sequences and require extensive retraining to adapt to new scenarios. Here, we introduce BF-PIP (Beyond Frames Pedestrian Intention Prediction), a zero-shot approach built upon Gemini 2.5 Pro. It infers crossing intentions directly from short, continuous video clips enriched with structured JAAD metadata. In contrast to GPT-4V based methods that operate on discrete frames, BF-PIP processes uninterrupted temporal clips. It also incorporates bounding-box annotations and ego-vehicle speed via specialized multimodal prompts. Without any additional training, BF-PIP achieves 73% prediction accuracy, outperforming a GPT-4V baseline by 18 %. These findings illustrate that combining temporal video inputs with contextual cues enhances spatiotemporal perception and improves intent inference under ambiguous conditions. This approach paves the way for agile, retraining-free perception module in intelligent transportation system.