mlsanityData to 14 Oct 2026

VG4D: Vision-Language Model Goes 4D Video Recognition

arXiv 2404.11605 ↗First posted 17 Apr 2024cs.CV cs.AI cs.ROcited by 11

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Understanding the real world through point cloud video is a crucial aspect of robotics and autonomous driving systems. However, prevailing methods for 4D point cloud recognition have limitations due to sensor resolution, which leads to a lack of detailed information. Recent advances have shown that Vision-Language Models (VLM) pre-trained on web-scale text-image datasets can learn fine-grained visual concepts that can be transferred to various downstream tasks. However, effectively integrating VLM into the domain of 4D point clouds remains an unresolved problem. In this work, we propose the Vision-Language Models Goes 4D (VG4D) framework to transfer VLM knowledge from visual-text pre-trained models to a 4D point cloud network. Our approach involves aligning the 4D encoder's representation with a VLM to learn a shared visual and text space from training on large-scale image-text pairs. By transferring the knowledge of the VLM to the 4D encoder and combining the VLM, our VG4D achieves improved recognition performance. To enhance the 4D encoder, we modernize the classic dynamic point cloud backbone and propose an improved version of PSTNet, im-PSTNet, which can efficiently model point cloud videos. Experiments demonstrate that our method achieves state-of-the-art performance for action recognition on both the NTU RGB+D 60 dataset and the NTU RGB+D 120 dataset. Code is available at urlhttps://github.com/Shark0-0/VG4D.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 56 references that are in the map. First posted on arXiv 17 Apr 2024.

Builds on

42 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition20 Mar
  2. 2022Visual Language Maps for Robot Navigation11 Oct
  3. 2021PointCLIP: Point Cloud Understanding by CLIP4 Dec

Other references · 39

  1. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  3. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,223
  4. 2020Point Transformercited by 1,470
  5. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 686
  6. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  7. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 533
  8. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
  9. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 436
  10. 2018A Short Note about Kinetics-600cited by 333
  11. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 321
  12. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 267
27 more
  1. 2021Revisiting Skeleton-based Action Recognitioncited by 257
  2. 2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 179
  3. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  4. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 160
  5. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 160
  6. 2019MeteorNet: Deep Learning on Dynamic 3D Point Cloud Sequencescited by 117
  7. 2023Vita-CLIP: Video and text adaptive CLIP via Multimodal Promptingcited by 91
  8. 2020Efficient RGB-D Semantic Segmentation for Indoor Scene Analysiscited by 84
  9. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Datacited by 69
  10. 2022PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequencescited by 68
  11. 2020VPN: Learning Video-Pose Embedding for Activities of Daily Livingcited by 57
  12. 2022STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognitioncited by 43
  13. 20203DV: 3D Dynamic Voxel for Action Recognition in Depth Videocited by 41
  14. 2023Explore In-Context Learning for 3D Point Cloud Understandingcited by 40
  15. 2022Point Primitive Transformer for Long-Term 4D Point Cloud Video Understandingcited by 35
  16. 2023Skeleton-in-Context: Unified Skeleton Sequence Modeling with In-Context Learningcited by 27
  17. 2022No Pain, Big Gain: Classify Dynamic Point Cloud Sequences with Static Models by Fitting Feature-level Space-time Surfacescited by 25
  18. 2022Let Images Give You More:Point Cloud Cross-Modal Training for Shape Analysiscited by 24
  19. 2021Spatial-Temporal Transformer for 3D Point Cloud Sequencescited by 22
  20. 2023PointCMP: Contrastive Mask Prediction for Self-supervised Learning on Point Cloud Videoscited by 22
  21. 20194D Generic Video Object Proposalscited by 14
  22. 2021Decoupling and Recoupling Spatiotemporal Representation for RGB-D-based Motion Recognitioncited by 13
  23. 2023GCNext: Towards the Unity of Graph Convolutions for Human Motion Predictioncited by 12
  24. 2023Dynamic Dense Graph Convolutional Network for Skeleton-based Human Motion Predictioncited by 11
  25. 2022MAPLE: Masked Pseudo-Labeling autoEncoder for Semi-supervised Point Cloud Action Recognitioncited by 8
  26. 2022Learning Category-Level Manipulation Tasks from Point Clouds with Dynamic Graph CNNscited by 4
  27. 2024ModelNet-O: A Large-Scale Synthetic Dataset for Occlusion-Aware Point Cloud Classificationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 4.

11 papers.

  1. 2024EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialoguescited by 85
  2. 2026Align then Adapt: Rethinking Parameter-Efficient Transfer Learning in 4D Perceptioncited by 18
  3. 2025LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understandingcited by 17
  4. 2024Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognitioncited by 12
  5. 2026Tri-Efficient Transfer Learning for Point Cloud Videoscited by 6
  6. 20264DPC2hat: Towards Dynamic Point Cloud Understanding with Failure-Aware Bootstrappingcited by 5
  7. 2025Human-in-Context: Unified Cross-Domain 3D Human Motion Modeling via In-Context Learningcited by 3
  8. 2024PointDGMamba: Domain Generalization of Point Cloud Classification via Generalized State Space Modelcited by 2
  9. 2025Heatmap Pooling Network for Action Recognition from RGB Videoscited by 1
  10. 20253D4D: An Interactive, Editable, 4D World Model via 3D Video Generationcited by 0
  11. 2026DAP: Doppler-aware Point Network for Heterogeneous mmWave Action Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CL4D: Contrastive Language-4D Pretraining for Vision-Language Reasoning in Dynamic Scenes
  2. 2023Joint Representation Learning for Text and 3D Point Cloud
  3. 2023Vision-Language Models for Vision Tasks: A Survey
  4. 2025Investigating Vision-Language Model for Point Cloud-based Vehicle Classification
  5. 20264DP-QA: Scalable QA for 4D Perception in Vision Language Models
  6. 2026An overview of 3D Vision-Language Models
  7. 2024Do Pre-trained Vision-Language Models Encode Object States?
  8. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Models