mlsanityData to 7 Oct 2026

TutoAI: A Cross-domain Framework for AI-assisted Mixed-media Tutorial Creation on Physical Tasks

arXiv 2403.08049 ↗First posted 12 Mar 2024cs.HC cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Mixed-media tutorials, which integrate videos, images, text, and diagrams to teach procedural skills, offer more browsable alternatives than timeline-based videos. However, manually creating such tutorials is tedious, and existing automated solutions are often restricted to a particular domain. While AI models hold promise, it is unclear how to effectively harness their powers, given the multi-modal data involved and the vast landscape of models. We present TutoAI, a cross-domain framework for AI-assisted mixed-media tutorial creation on physical tasks. First, we distill common tutorial components by surveying existing work; then, we present an approach to identify, assemble, and evaluate AI models for component extraction; finally, we propose guidelines for designing user interfaces (UI) that support tutorial creation based on AI-generated components. We show that TutoAI has achieved higher or similar quality compared to a baseline model in preliminary user studies.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 37 of its 87 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

37 of 87 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
25 more
  1. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  2. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  3. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  4. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  5. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  6. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  7. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  8. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  9. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  10. 2021You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detectioncited by 170
  11. 2020TransNet V2: An effective deep network architecture for fast shot transition detectioncited by 166
  12. 2018MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustmentcited by 145
  13. 2021End-to-End Dense Video Captioning with Parallel Decodingcited by 133
  14. 2023Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarizationcited by 122
  15. 2019Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Makingcited by 109
  16. 2021CLIP-It! Language-Guided Video Summarizationcited by 99
  17. 2021Sparks: Inspiration for Science Writing using Language Modelscited by 80
  18. 2023Align and Attend: Multimodal Summarization with Dual Contrastive Lossescited by 65
  19. 2022Beyond Text Generation: Supporting Writers with Continuous Automatic Text Summariescited by 47
  20. 2023Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspectivecited by 34
  21. 2019Friend, Collaborator, Student, Manager: How Design of an AI-Driven Game Level Editor Affects Creatorscited by 31
  22. 2022End-to-end Dense Video Captioning as Sequence Generationcited by 27
  23. 2022LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexiacited by 27
  24. 2021Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2cited by 15
  25. 2021Soloist: Generating Mixed-Initiative Tutorials from Existing Guitar Instructional Videos Through Audio Processingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025ClearFairy: Capturing Creative Workflows through Decision Structuring, In-Situ Questioning, and Rationale Inferencecited by 4
  2. 2025VideoMix: Aggregating How-To Videos for Task-Oriented Learningcited by 3
  3. 2025NoteIt: A System Converting Instructional Videos to Interactable Notes Through Multimodal Video Understandingcited by 2
  4. 2025Interactive Program Synthesis for Modeling Collaborative Physical Activities from Narrated Demonstrationscited by 2
  5. 2026coTherapist: A Behavior-Aligned Small Language Model to Support Mental Healthcare Expertscited by 1
  6. 2026AutoCue: Multimodal LLM-Assisted Externalization of Implicit Inputs as Instructional Visual Cues in Screencast Tutorialscited by 0