TraMP-LLaMA: Generative Interpretability with Decoupled Instruction Tuning for Facial Expression Quality Assessment
Not on the map. Why?
Abstract (arXiv)
Existing facial expression quality assessment (FEQA) methods typically produce only a severity score, without explicitly communicating the observable facial motion evidence that supports the prediction. This limits interpretability and makes it difficult to inspect the basis of model outputs in Parkinson's disease assessment. To address this gap, we propose TraMP-LLaMA, a unified multimodal framework that jointly predicts severity scores and generates structured textual reports from facial motion cues. The framework integrates RGB appearance and landmark trajectory cues, and adopts a decoupled instruction-tuning strategy to reduce task interference between severity prediction and language generation. To support this task, we further extend the PFED5 dataset with expert-guided textual motion descriptions and construct PFED5-plus. Experiments on PFED5-plus show that TraMP-LLaMA outperforms competitive video-language baselines in report generation and achieves the best severity prediction performance among the compared methods under joint multi-expression training, improving Spearman's rank correlation by at least 4.39 percent over all competing methods. The text annotations and code are available at https://github.com/shuchaoduan/TraMP-LLaMA.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 74 references that are in the map. First posted on arXiv 25 Jun 2026.
Builds on
39 of 74 references.
References · 39
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
- 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
- 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
- 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
- 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
27 more
- 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
- 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
- 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 356
- 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 229
- 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
- 2020DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wildcited by 128
- 2021Sample and Computation Redistribution for Efficient Face Detectioncited by 89
- 2020Uncertainty-aware Score Distribution Learning for Action Quality Assessmentcited by 80
- 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Modelscited by 70
- 2022MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wildcited by 65
- 2023KiUT: Knowledge-injected U-Transformer for Radiology Report Generationcited by 62
- 2024SkateFormer: Skeletal-Temporal Transformer for Human Action Recognitioncited by 47
- 2024EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuningcited by 43
- 2024Automated Radiology Report Generation: A Review of Recent Advancescited by 43
- 2023From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videoscited by 39
- 2023EmoCLIP: A Vision-Language Method for Zero-Shot Video Facial Expression Recognitioncited by 31
- 2024FineCLIPER: Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERscited by 30
- 2024Facial Affective Behavior Analysis with Instruction Tuningcited by 26
- 2024Enhancing Zero-Shot Facial Expression Recognition by LLM Knowledge Transfercited by 18
- 2024CoFInAl: Enhancing Action Quality Assessment with Coarse-to-Fine Instruction Alignmentcited by 17
- 2023Designing Interpretable ML System to Enhance Trust in Healthcare: A Systematic Review to Proposed Responsible Clinician-AI-Collaboration Frameworkcited by 11
- 2025Face-LLaVA: Facial Expression and Attribute Understanding through Instruction Tuningcited by 11
- 2022Facial Expression Analysis Using Decomposed Multiscale Spatiotemporal Networkscited by 4
- 2022Automatic Estimation of Self-Reported Pain by Trajectory Analysis in the Manifold of Fixed Rank Positive Semi-Definite Matricescited by 4
- 2019Image Captioning using Facial Expression and Attentioncited by 3
- 2023QAFE-Net: Quality Assessment of Facial Expressions with Landmark Heatmapscited by 2
- 2025Trajectory-guided Motion Perception for Facial Expression Quality Assessment in Neurological Disorderscited by 1
Built on by
Not cited yet.