mlsanityData to 14 Oct 2026

Unified Vision-Centric Pedestrian Crossing Action Prediction via Adaptive Patch Projection and Proactive Spatial Rectification

arXiv 2609.07420 ↗First posted 7 Sep 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

Vision cues are available and informative for pedestrian action prediction, but obtaining stable target-centric representations from video frames remains challenging without frame-level external perception cues. Thus, most methods rely on additional perception modules or multi-source information fusion, leaving the reliability of vision-centric setting an open question. To this end, we propose ViCross, a vision-centric pedestrian crossing action prediction framework powered by multimodal large language models, which maintains target-centric reasoning from video frames without additional perception modules beyond first-frame target initialization. While multimodal large language models exhibit strong visual understanding, applying them directly to vision-centric action prediction faces two challenges. First, accurately perceiving target pedestrians often requires high resolution inputs and dense visual tokenization, making full-frame encoding computationally prohibitive. ViCross tackles this with Variable Resolution Patch Mapping module for efficient token allocation while preserving key pedestrian details. Second, missing spatiotemporal priors hinder consistent cross frame reasoning. ViCross mitigates this with a Spatial Constraint Enhancement Strategy that captures past motion, future locations, and action semantics for training-time proactive spatial rectification. Extensive experiments show that ViCross delivers clear gains in vision-centric prediction settings and is competitive with multi-source fusion approaches in several settings. Code is available at https://github.com/2tianyao1/ViCross.git.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 52 references that are in the map. First posted on arXiv 7 Sep 2026.

Builds on

30 of 52 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  9. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  10. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
  11. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,064
  12. 2017Simple Online and Realtime Tracking with a Deep Association Metriccited by 910
18 more
  1. 2021Conditional DETR for Fast Training Convergencecited by 424
  2. 2017Long-Term On-Board Prediction of People in Traffic Scenes under Uncertaintycited by 63
  3. 2021Predicting Pedestrian Crossing Intention with Feature Fusion and Spatio-Temporal Attentioncited by 48
  4. 2020Bifold and Semantic Reasoning for Pedestrian Behavior Predictioncited by 31
  5. 2019Real-time Intent Prediction of Pedestrians for Autonomous Ground Vehicles via Spatio-Temporal DenseNetcited by 28
  6. 2018Is the Pedestrian going to Cross? Answering by 2D Pose Estimationcited by 25
  7. 2022PedFormer: Pedestrian Behavior Prediction via Cross-Modal Attention Modulation and Gated Multitask Learningcited by 23
  8. 2020RNN-based Pedestrian Crossing Prediction using Activity and Pose-related Featurescited by 21
  9. 2020Pedestrian Trajectory Prediction with Convolutional Neural Networkscited by 21
  10. 2021Is attention to bounding boxes all you need for pedestrian action prediction?cited by 21
  11. 2023GPT-4V Takes the Wheel: Promises and Challenges for Pedestrian Behavior Predictioncited by 11
  12. 2022Deep Virtual-to-Real Distillation for Pedestrian Crossing Predictioncited by 10
  13. 2025Multimodal Large Models Are Effective Action Anticipatorscited by 10
  14. 2022TAMFormer: Multi-Modal Transformer with Learned Attention Mask for Early Intent Predictioncited by 9
  15. 2019Context Model for Pedestrian Intention Prediction using Factored Latent-Dynamic Conditional Random Fieldscited by 8
  16. 2024Synthetic Data Generation Framework, Dataset, and Efficient Deep Model for Pedestrian Intention Predictioncited by 5
  17. 2023Orientation-Aware Leg Movement Learning for Action-Driven Human Motion Predictioncited by 4
  18. 2024Diving Deeper Into Pedestrian Behavior Understanding: Intention Estimation, Action Prediction, and Event Risk Assessmentcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026A Vision-and-Knowledge Enhanced Large Language Model for Generalizable Pedestrian Crossing Behavior Inference
  2. 2026Adaptive Cross-Modal Fusion with Sparse Attention for Pedestrian Crossing Intention Prediction
  3. 2026Decoding Pedestrian Crossing Intention from Egocentric Vision via Vision Language Models
  4. 2026PEDESTRIANQA: A Benchmark for Vision-Language Models on Pedestrian Intention and Trajectory Prediction
  5. 2025SceneAware: Scene-Constrained Pedestrian Trajectory Prediction with LLM-Guided Walkability
  6. 2025GUIDE-CoT: Goal-driven and User-Informed Dynamic Estimation for Pedestrian Trajectory using Chain-of-Thought
  7. 2026Motion-as-Prompt: Enhancing Motion Reasoning in Multimodal Large Language Models via Motion-Guided Cross-Frame Visual Prompting
  8. 2024Gating Syn-to-Real Knowledge for Pedestrian Crossing Prediction in Safe Driving