mlsanityData to 7 Oct 2026

Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books

arXiv 1506.06724 ↗First posted 22 Jun 2015cs.CV cs.CLcited by 1,041

Not on the map. Why?

Cited by 22 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Books are a rich source of both fine-grained information, how a character, an object or a scene looks like, as well as high-level semantics, what someone is thinking, feeling and how these states evolve through a story. This paper aims to align books to their movie releases in order to provide rich descriptive explanations for visual content that go semantically far beyond the captions available in current datasets. To align movies and books we exploit a neural sentence embedding that is trained in an unsupervised way from a large corpus of books, as well as a video-text neural embedding for computing similarities between movie clips and sentences in the book. We propose a context-aware CNN to combine information from multiple sources. We demonstrate good quantitative performance for movie/book alignment and show several qualitative examples that showcase the diversity of tasks our model can be used for.

Key facts. Cited by 1,041 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Jun 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 236; 2022: 257; 2023: 255; 2024: 158; 2025: 93; 2026: 41.

1,041 papers.

Sort:

By area:

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  9. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  10. 2021Domain Generalization: A Surveycited by 760
  11. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  12. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
38 more
  1. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  2. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  3. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  4. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  5. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  6. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  7. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  8. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  9. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  10. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  11. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  12. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  13. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  14. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  15. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  16. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  17. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 306
  18. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  19. 2023Large Language Model Unlearningcited by 288
  20. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  21. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  22. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  23. 2022Matryoshka Representation Learningcited by 268
  24. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  25. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  26. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  27. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  28. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
  29. 2023Data-centric Artificial Intelligence: A Surveycited by 202
  30. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  31. 2023Power Hungry Processing: Watts Driving the Cost of AI Deployment?cited by 186
  32. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  33. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  34. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  35. 2021Synthesis of Compositional Animations from Textual Descriptionscited by 177
  36. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  37. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  38. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172

50 most cited of 1,041 All 1,041 (JSON)