mlsanityData to 7 Oct 2026

LEAP: LLM-Generation of Egocentric Action Programs

arXiv 2312.00055 ↗First posted 29 Nov 2023cs.CV cs.LG cs.ROcited by 5

Not on the map. Why?

Abstract (arXiv)

We introduce LEAP (illustrated in Figure 1), a novel method for generating video-grounded action programs through use of a Large Language Model (LLM). These action programs represent the motoric, perceptual, and structural aspects of action, and consist of sub-actions, pre- and post-conditions, and control flows. LEAP's action programs are centered on egocentric video and employ recent developments in LLMs both as a source for program knowledge and as an aggregator and assessor of multimodal video information. We apply LEAP over a majority (87%) of the training set of the EPIC Kitchens dataset, and release the resulting action programs as a publicly available dataset here (https://drive.google.com/drive/folders/1CpkwTI1IIxXdzor0pOXG3rWJWuKU5Ex?usp=drivelink). We employ LEAP as a secondary source of supervision, using its action programs in a loss term applied to action recognition and anticipation networks. We demonstrate sizable improvements in performance in both tasks due to training with the LEAP dataset. Our method achieves 1st place on the EPIC Kitchens Action Recognition leaderboard as of November 17 among the networks restricted to RGB-input (see Supplementary Materials).

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 36 of its 41 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

36 of 41 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2015Fast R-CNNcited by 2,930
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2021ViViT: A Video Vision Transformercited by 1,610
  10. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  11. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  12. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
24 more
  1. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  2. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  3. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  4. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  5. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  6. 2020FineGym: A Hierarchical Video Dataset for Fine-grained Action Understandingcited by 232
  7. 2022Learning Video Representations from Large Language Modelscited by 204
  8. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  9. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  10. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  11. 2020Rolling-Unrolling LSTMs for Action Anticipation from First-Person Videocited by 119
  12. 2022UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormercited by 116
  13. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  14. 2022FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessmentcited by 106
  15. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  16. 2022Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videoscited by 50
  17. 2023EPIC Fields: Marrying 3D Geometry and Video Understandingcited by 49
  18. 2023Epic-Sounds: A Large-scale Dataset of Actions That Soundcited by 47
  19. 2020Intra- and Inter-Action Understanding via Temporal Action Parsingcited by 41
  20. 2021SVIP: Sequence VerIfication for Procedures in Videoscited by 16
  21. 2022SOS! Self-supervised Learning Over Sets Of Handled Objects In Egocentric Action Recognitioncited by 8
  22. 2023Therbligs in Action: Video Understanding through Motion Primitivescited by 7
  23. 2021SAIC_Cambridge-HuPBA-FBK Submission to the EPIC-Kitchens-100 Action Recognition Challenge 2021cited by 3
  24. 20221st Place Solution to the EPIC-Kitchens Action Anticipation Challenge 2022cited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  2. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  3. 2025Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Inputcited by 17
  4. 2026VSMP-IMU: Video-Grounded Semantic Motion Programs for Sensor-Aware Synthetic IMU Generationcited by 0
  5. 2026Language-Augmented Video Action Anticipation: Design Fundamentals, Benchmarks, and Open Challengescited by 0