mlsanityData to 7 Oct 2026

Translating Videos to Natural Language Using Deep Recurrent Neural Networks

arXiv 1412.4729 ↗First posted 15 Dec 2014cs.CV cs.CLcited by 95

Not on the map. Why?

Abstract (arXiv)

Solving the visual symbol grounding problem has long been a goal of artificial intelligence. The field appears to be advancing closer to this goal with recent breakthroughs in deep learning for natural language grounding in static images. In this paper, we propose to translate videos directly to sentences using a unified deep neural network with both convolutional and recurrent structure. Described video datasets are scarce, and most existing methods have been applied to toy domains with a small vocabulary of possible words. By transferring knowledge from 1.2M+ images with category labels and 100,000+ images with captions, our method is able to create sentence descriptions of open-domain videos with large vocabularies. We compare our approach with recent work using language generation metrics, subject, verb, and object prediction accuracy, and a human evaluation.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Dec 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 0; 2019: 0; 2020: 0; 2021: 33; 2022: 25; 2023: 19; 2024: 11; 2025: 5; 2026: 1.

95 papers.

Sort:

By area:

  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  2. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  3. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  4. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  5. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  6. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  7. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  8. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  9. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  10. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoningcited by 152
  11. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  12. 2021End-to-End Dense Video Captioning with Parallel Decodingcited by 133
38 more
  1. 2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
  2. 2023ADAPT: Action-aware Driving Caption Transformercited by 78
  3. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  4. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  5. 2021Contrastive Attention for Automatic Chest X-ray Report Generationcited by 71
  6. 2021Object-aware Video-language Pre-training for Retrievalcited by 57
  7. 2023Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learningcited by 45
  8. 2022LocVTP: Video-Text Pre-training for Temporal Localizationcited by 44
  9. 2024Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrievalcited by 36
  10. 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
  11. 2021Hierarchical Modular Network for Video Captioningcited by 31
  12. 2022Visual Abductive Reasoningcited by 30
  13. 2023A Review of Deep Learning for Video Captioningcited by 28
  14. 2023Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generationcited by 23
  15. 2017Reinforced Video Captioning with Entailment Rewardscited by 21
  16. 2021Video-Text Pre-training with Learned Regionscited by 20
  17. 2022Text-Adaptive Multiple Visual Prototype Matching for Video-Text Retrievalcited by 20
  18. 2023Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Bankscited by 20
  19. 2021O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioningcited by 18
  20. 2022Refined Semantic Enhancement towards Frequency Diffusion for Video Captioningcited by 18
  21. 2023Tagging before Alignment: Integrating Multi-Modal Tags for Video-Text Retrievalcited by 18
  22. 2022TempCLR: Temporal Alignment Representation with Contrastive Learningcited by 15
  23. 2021Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and Visioncited by 14
  24. 2022Aligning Source Visual and Target Language Domains for Unpaired Video Captioningcited by 14
  25. 2021Transcript to Video: Efficient Clip Sequencing from Textscited by 13
  26. 2022Sports Video Analysis on Large-Scale Datacited by 13
  27. 2024A Survey on Integrated Sensing, Communication, and Computationcited by 12
  28. 2021CLIP Meets Video Captioning: Concept-Aware Representation Learning Does Mattercited by 11
  29. 2022LGDN: Language-Guided Denoising Network for Video-Language Modelingcited by 11
  30. 2023CLEVRER-Humans: Describing Physical and Causal Events the Human Waycited by 11
  31. 2021X-modaler: A Versatile and High-performance Codebase for Cross-modal Analyticscited by 10
  32. 2021Video and Text Matching with Conditioned Embeddingscited by 10
  33. 2021Sensor-Augmented Egocentric-Video Captioning with Dynamic Modal Attentioncited by 9
  34. 2021Controllable Video Captioning with an Exemplar Sentencecited by 9
  35. 2024Enhancing Traffic Safety with Parallel Dense Video Captioning for End-to-End Event Analysiscited by 9
  36. 2024MICap: A Unified Model for Identity-aware Movie Descriptionscited by 9
  37. 2021PGT: A Progressive Method for Training Models on Long Videoscited by 7
  38. 2024Sentiment-oriented Transformer-based Variational Autoencoder Network for Live Video Commentingcited by 7

50 most cited of 95 All 95 (JSON)