mlsanityData to 7 Oct 2026

ExpertAF: Expert Actionable Feedback from Video

arXiv 2408.00672 ↗First posted 1 Aug 2024cs.CVcited by 17

Not on the map. Why?

Abstract (arXiv)

Feedback is essential for learning a new skill or improving one's current skill-level. However, current methods for skill-assessment from video only provide scores or compare demonstrations, leaving the burden of knowing what to do differently on the user. We introduce a novel method to generate actionable feedback (AF) from video of a person doing a physical activity, such as basketball or soccer. Our method takes a video demonstration and its accompanying 3D body pose and generates (1) free-form expert commentary describing what the person is doing well and what they could improve, and (2) a visual expert demonstration that incorporates the required corrections. We show how to leverage Ego-Exo4D's [29] videos of skilled activity and expert commentary together with a strong language model to create a weakly-supervised training dataset for this task, and we devise a multimodal video-language model to infer coaching feedback. Our method is able to reason across multi-modal input combinations to output full spectrum, actionable coaching-expert commentary, expert video retrieval, and expert pose generation-outperforming strong vision-language models on both established metrics and human preference studies.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 65 of its 90 references that are in the map. First posted on arXiv 1 Aug 2024.

Builds on

65 of 90 references.

References · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  7. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  8. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  9. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 683
  10. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  11. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  12. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
38 more
  1. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  2. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 530
  3. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  4. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  5. 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 344
  6. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  7. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  8. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  9. 2022Egocentric Video-Language Pretrainingcited by 231
  10. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  11. 2021Anticipative Video Transformercited by 197
  12. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  13. 2022MotionBERT: A Unified Perspective on Learning Human Motion Representationscited by 188
  14. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  15. 2019Learnable Triangulation of Human Posecited by 160
  16. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  17. 2015Human Pose Estimation with Iterative Error Feedbackcited by 121
  18. 2020Rolling-Unrolling LSTMs for Action Anticipation from First-Person Videocited by 119
  19. 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 106
  20. 2019What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessmentcited by 100
  21. 2019Procedure Planning in Instructional Videoscited by 92
  22. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
  23. 2022Temporal Alignment Networks for Long-term Videocited by 84
  24. 2022DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Videocited by 83
  25. 2023Diffusion-Based 3D Human Pose Estimation with Multi-Hypothesis Aggregationcited by 81
  26. 2020Uncertainty-aware Score Distribution Learning for Action Quality Assessmentcited by 80
  27. 2017RED: Reinforced Encoder-Decoder Networks for Action Anticipationcited by 77
  28. 2021Group-aware Contrastive Regression for Action Quality Assessmentcited by 75
  29. 2018Learning to score the figure skating sports videoscited by 72
  30. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  31. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  32. 2022GRIT: Faster and Better Image captioning Transformer Using Dual Visual Featurescited by 56
  33. 2021Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learningcited by 53
  34. 2021Towards Unified Surgical Skill Assessmentcited by 49
  35. 2023Procedure-Aware Pretraining for Instructional Video Understandingcited by 48
  36. 2023Learning Procedure-aware Video Representation from Instructional Videos and Their Narrationscited by 46
  37. 2023Human Pose as Compositional Tokenscited by 45
  38. 2023Learning Disentangled Avatars with Hybrid 3D Representationscited by 44

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 10.

17 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey of Action Quality Assessment: Method and Benchmarkcited by 20
  2. 2025Vid2Coach: Transforming How-To Videos into Task Assistantscited by 12
  3. 2024FIction: 4D Future Interaction Prediction from Videocited by 8
  4. 2025Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Visioncited by 8
  5. 2025A Decade of Action Quality Assessment: Largest Systematic Survey of Trends, Challenges, and Future Directionscited by 7
  6. 2025Learning Skill-Attributes for Transferable Assessment in Videocited by 4
  7. 2026Learning Consistent Temporal Grounding between Related Tasks in Sports Coachingcited by 1
  8. 2026SportSkills: Physical Skill Learning from Sports Instructional Videoscited by 1
  9. 2026ExpertEdit: Learning Skill-Aware Motion Editing from Expert Videoscited by 1
  10. 2026MMTB: Evaluating Terminal Agents on Multimedia-File Taskscited by 1
  11. 2025Automated Procedural Analysis via Video-Language Models for AI-assisted Nursing Skills Assessmentcited by 0
  12. 2026Latent Visual Diffusion Reasoning with Monte Carlo Tree Searchcited by 0
5 more
  1. 2026SkillSpotter: Pose-Aware Multi-View Skilled Action Detection and Grading in Ego-Exo Videoscited by 0
  2. 2026AIDE: Automated Instruction via Distilled Expertise for Reference-Free Motor Skill Coachingcited by 0
  3. 2026A Survey of Large Models in Sportscited by 0
  4. 2026PuzzleMate: Benchmarking MLLMs for Egocentric Puzzle Assistancecited by 0
  5. 2026Ego-Exo4D Human Meshes Dataset: 4D Human Motion Reconstruction for Ego-Exo Capturescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CricRAG: Retrieval Augmented Vision-Language Models for Personalized Cricket Coaching
  2. 2024Poze: Sports Technique Feedback under Data Constraints
  3. 2026ViSTAR: Virtual Skill Training with Augmented Reality with 3D Avatars and LLM coaching agent
  4. 2025Explainable Action Form Assessment by Exploiting Multimodal Chain-of-Thoughts Reasoning
  5. 2026Can Vision Language Models Judge Action Quality? An Empirical Evaluation
  6. 2026BoT-Feedback: Grounding Multimodal Reasoning in Biomechanical Evidence for Explainable Human Action Feedback
  7. 2026FitAQA: A Benchmark of Fitness Action Quality Assessment for Multimodal Large Language Models
  8. 2025FormCoach: Lift Smarter, Not Harder