mlsanityData to 14 Oct 2026

Step Differences in Instructional Video

arXiv 2404.16222 ↗First posted 24 Apr 2024cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

Comparing a user video to a reference how-to video is a key requirement for AR/VR technology delivering personalized assistance tailored to the user's progress. However, current approaches for language-based assistance can only answer questions about a single video. We propose an approach that first automatically generates large amounts of visual instruction tuning data involving pairs of videos from HowTo100M by leveraging existing step annotations and accompanying narrations, and then trains a video-conditioned language model to jointly reason across multiple raw videos. Our model achieves state-of-the-art performance at identifying differences between video pairs and ranking videos based on the severity of these differences, and shows promising ability to perform general reasoning over multiple videos. Project page: https://github.com/facebookresearch/stepdiff

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 54 of its 72 references that are in the map. First posted on arXiv 24 Apr 2024.

Builds on

54 of 72 references.

References · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  10. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  11. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  12. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
38 more
  1. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  2. 2023Instruction Tuning with GPT-4cited by 686
  3. 2017Localizing Moments in Video with Natural Languagecited by 663
  4. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  5. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  6. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  7. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  8. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  9. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  10. 2022SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentationcited by 400
  11. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  12. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  13. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  14. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 287
  15. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  16. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  17. 2018Composing Text and Image for Image Retrieval - An Empirical Odysseycited by 236
  18. 2022Egocentric Video-Language Pretrainingcited by 232
  19. 2022Learning Video Representations from Large Language Modelscited by 205
  20. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  21. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  22. 2018Learning to Describe Differences Between Pairs of Similar Imagescited by 114
  23. 2018Actor and Observer: Joint Modeling of First and Third-Person Videoscited by 103
  24. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  25. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
  26. 2022Video Question Answering: Datasets, Algorithms and Challengescited by 84
  27. 2022Temporal Alignment Networks for Long-term Videocited by 84
  28. 2023Hierarchical Video-Moment Retrieval and Step-Captioningcited by 83
  29. 2022My View is the Best View: Procedure Learning from Egocentric Videoscited by 81
  30. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  31. 2015An end-to-end generative framework for video segmentation and recognitioncited by 61
  32. 2023CoVR-2: Automatic Data Construction for Composed Video Retrievalcited by 52
  33. 2021Describing and Localizing Multiple Changes with Transformerscited by 47
  34. 2019Neural Naturalist: Generating Fine-Grained Image Comparisonscited by 36
  35. 2022Learning to Answer Visual Questions from Web Videoscited by 30
  36. 2021ChangeSim: Towards End-to-End Online Scene Change Detection in Industrial Indoor Environmentscited by 28
  37. 2023Learning to Ground Instructional Articles in Videos through Narrationscited by 28
  38. 2020Visual Question Answering on Image Setscited by 26

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 4.

11 papers.

  1. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  2. 2024ExpertAF: Expert Actionable Feedback from Videocited by 17
  3. 2025Video Action Differencingcited by 14
  4. 2025ConViS-Bench: Estimating Video Similarity Through Semantic Conceptscited by 4
  5. 2025Learning Skill-Attributes for Transferable Assessment in Videocited by 4
  6. 2024It's Just Another Day: Unique Video Captioning by Discriminative Promptingcited by 3
  7. 2025Stitch-a-Demo: Video Demonstrations from Multistep Descriptionscited by 1
  8. 2026SportSkills: Physical Skill Learning from Sports Instructional Videoscited by 1
  9. 2026Let's Split Up: Zero-Shot Classifier Edits for Fine-Grained Video Understandingcited by 0
  10. 2026Anomalous Frame Detection by Grouping Frame Similarities between Two Videos Computed by Vision-Language Model to Extract Expert Workers' Unique Actionscited by 0
  11. 2026Anomalous Frame Detection Using VLM-Based Description Comparison for Extracting Expert-Specific Actions and Contextual Decision-Making Scenes with Intra-Video Self-Similaritycited by 0