KRAST: Knowledge-Augmented Robotic Action Recognition with Structured Text for Vision-Language Models
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
Accurate vision-based action recognition is crucial for developing autonomous robots that can operate safely and reliably in complex, real-world environments. In this work, we advance video-based recognition of indoor daily actions for robotic perception by leveraging vision-language models (VLMs) enriched with domain-specific knowledge. We adapt a prompt-learning framework in which class-level textual descriptions of each action are embedded as learnable prompts into a frozen pre-trained VLM backbone. Several strategies for structuring and encoding these textual descriptions are designed and evaluated. Experiments on the ETRI-Activity3D dataset demonstrate that our method, using only RGB video inputs at test time, achieves over 95% accuracy and outperforms state-of-the-art approaches. These results highlight the effectiveness of knowledge-augmented prompts in enabling robust action recognition with minimal supervision.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 25 references that are in the map. First posted on arXiv 19 Sep 2025.
Builds on
19 of 25 references.
References · 19
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
- 2017The Kinetics Human Action Video Datasetcited by 1,909
- 2018SlowFast Networks for Video Recognitioncited by 1,867
- 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
- 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
- 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 686
- 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
- 2024YOLOv11: An Overview of the Key Architectural Enhancementscited by 425
- 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
7 more
- 2018Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregationcited by 106
- 2023Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Modelscited by 30
- 2020ETRI-Activity3D: A Large-Scale RGB-D Dataset for Robots to Recognize Daily Activities of the Elderlycited by 26
- 2020Vision-based Estimation of MDS-UPDRS Gait Scores for Assessing Parkinson's Disease Motor Severitycited by 18
- 2017Cooperative Training of Deep Aggregation Networks for RGB-D Action Recognitioncited by 11
- 2024Enhancing Gait Video Analysis in Neurodegenerative Diseases by Knowledge Augmentation in Vision Language Modelcited by 4
- 2023From Detection to Action Recognition: An Edge-Based Pipeline for Robot Human Perceptioncited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
- 2025XR-1: Towards Versatile Vision-Language-Action Models via Learning Unified Vision-Motion Representations
- 2026Task Robustness via Re-Labelling Vision-Action Robot Data
- 2026G0.5: One Autoregressive Stream for Robot Reasoning and Action
- 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challenges
- 2025Vision Language Action Models in Robotic Manipulation: A Systematic Review
- 2026VLAFlow: A Unified Training Framework for Vision-Language-Action Models via Co-training and Future Latent Alignment
- 2025VITA-VLA: Efficiently Teaching Vision-Language Models to Act via Action Expert Distillation