mlsanityData to 7 Oct 2026

How2: A Large-scale Dataset for Multimodal Language Understanding

arXiv 1811.00347 ↗First posted 1 Nov 2018cs.CLcited by 137

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce How2, a multimodal collection of instructional videos with English subtitles and crowdsourced Portuguese translations. We also present integrated sequence-to-sequence baselines for machine translation, automatic speech recognition, spoken language translation, and multimodal summarization. By making available data and code for several multimodal natural language tasks, we hope to stimulate more research on these and similar challenges, to obtain a deeper understanding of multimodality in language processing.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 26; 2022: 17; 2023: 37; 2024: 34; 2025: 19; 2026: 4.

137 papers.

Sort:

By area:

  1. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  2. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  3. 2023A Survey on Video Diffusion Modelscited by 203
  4. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  5. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  6. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 132
  7. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 124
  8. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
  9. 2023Align and Attend: Multimodal Summarization with Dual Contrastive Lossescited by 65
  10. 2021UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-trainingcited by 64
  11. 2021Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videoscited by 62
  12. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
38 more
  1. 2023Neural Sign Actors: A diffusion model for 3D sign language production from textcited by 48
  2. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
  3. 2024Grounded Multi-Hop VideoQA in Long-Form Egocentric Videoscited by 43
  4. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  5. 2021N24News: A New Dataset for Multimodal News Classificationcited by 40
  6. 2022A Survey on Cross-Lingual Summarizationcited by 38
  7. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  8. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  9. 2024MovieBench: A Hierarchical Movie Level Dataset for Long Video Generationcited by 35
  10. 2022SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Taskscited by 34
  11. 2021Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarizationcited by 32
  12. 2023A Large-Scale Outdoor Multi-modal Dataset and Benchmark for Novel View Synthesis and Implicit Scene Reconstructioncited by 32
  13. 2021Instant One-Shot Word-Learning for Context-Specific Neural Sequence-to-Sequence Speech Recognitioncited by 30
  14. 2023SignAvatars: A Large-scale 3D Sign Language Holistic Motion Dataset and Benchmarkcited by 29
  15. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  16. 2021A Survey on Multi-modal Summarizationcited by 26
  17. 2024From Sora What We Can See: A Survey of Text-to-Video Generationcited by 26
  18. 2021Speech Summarization using Restricted Self-Attentioncited by 23
  19. 2022Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translationcited by 23
  20. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  21. 2023Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Modelscited by 21
  22. 2024Exploring the Capability of Mamba in Speech Applicationscited by 19
  23. 2021See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarizationcited by 18
  24. 2022Face-Dubbing++: Lip-Synchronous, Voice Preserving Translation of Videoscited by 18
  25. 2024CrossCheckGPT: Universal Hallucination Ranking for Multimodal Foundation Modelscited by 18
  26. 2021Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct Speech Translationcited by 17
  27. 2024Abstractive Text Summarization: State of the Art, Challenges, and Improvementscited by 17
  28. 2023AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASRcited by 16
  29. 2021An Empirical Study of End-to-end Simultaneous Speech Translation Decoding Strategiescited by 14
  30. 2022Summary-Oriented Vision Modeling for Multimodal Abstractive Summarizationcited by 14
  31. 2021Transcript to Video: Efficient Clip Sequencing from Textscited by 13
  32. 2024MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptionscited by 13
  33. 2024MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languagescited by 13
  34. 2021FIBER: Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Frameworkcited by 12
  35. 2022AVATAR: Unconstrained Audiovisual Speech Recognitioncited by 12
  36. 2022Hierarchical3D Adapters for Long Video-to-text Summarizationcited by 12
  37. 2023Large Scale Multi-Lingual Multi-Modal Summarization Datasetcited by 12
  38. 2023MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videoscited by 12

50 most cited of 137 All 137 (JSON)