mlsanityData to 7 Oct 2026

Sequence to Sequence -- Video to Text

arXiv 1505.00487 ↗First posted 3 May 2015cs.CVcited by 157

Not on the map. Why?

Abstract (arXiv)

Real-world videos often have complex dynamics; and methods for generating open-domain video descriptions should be sensitive to temporal structure and allow both input (sequence of frames) and output (sequence of words) of variable length. To approach this problem, we propose a novel end-to-end sequence-to-sequence model to generate captions for videos. For this we exploit recurrent neural networks, specifically LSTMs, which have demonstrated state-of-the-art performance in image caption generation. Our LSTM model is trained on video-sentence pairs and learns to associate a sequence of video frames to a sequence of words in order to generate a description of the event in the video clip. Our model naturally is able to learn the temporal structure of the sequence of frames as well as the sequence model of the generated sentences, i.e. a language model. We evaluate several variants of our model that exploit different visual features on a standard set of YouTube videos and two movie description datasets (M-VAD and MPII-MD).

Key facts. Cited by 157 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 May 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 0; 2019: 0; 2020: 0; 2021: 48; 2022: 26; 2023: 27; 2024: 29; 2025: 19; 2026: 7.

157 papers.

Sort:

By area:

  1. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  2. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  3. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 357
  4. 2021End-to-End Dense Video Captioning with Parallel Decodingcited by 133
  5. 2023Large Language Models in Law: A Surveycited by 128
  6. 2024Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoningcited by 122
  7. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  8. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  9. 2021CLIP4Caption: CLIP for Video Captioncited by 88
  10. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  11. 2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
  12. 2023ADAPT: Action-aware Driving Caption Transformercited by 78
38 more
  1. 2021Cross Modal Retrieval with Querybank Normalisationcited by 71
  2. 2021Video Corpus Moment Retrieval with Contrastive Learningcited by 60
  3. 2021Target Adaptive Context Aggregation for Video Scene Graph Generationcited by 51
  4. 2023AutoAD: Movie Description in Contextcited by 50
  5. 2024Understanding the Impact of Negative Prompts: When and How Do They Take Effect?cited by 50
  6. 2021RODNet: A Real-Time Radar Object Detection Network Cross-Supervised by Camera-Radar Fused Object 3D Localizationcited by 49
  7. 2021On Semantic Similarity in Video Retrievalcited by 45
  8. 2021Grounding Open-Domain Instructions to Automate Web Support Taskscited by 43
  9. 2023Towards Surveillance Video-and-Language Understanding: New Dataset, Baselines, and Challengescited by 42
  10. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  11. 2021EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matchingcited by 36
  12. 2024Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrievalcited by 36
  13. 2021Reconstructive Sequence-Graph Network for Video Summarizationcited by 34
  14. 2021Hierarchical Modular Network for Video Captioningcited by 31
  15. 2022Visual Abductive Reasoningcited by 30
  16. 2023GOAL: A Challenging Knowledge-grounded Video Captioning Benchmark for Real-time Soccer Commentary Generationcited by 28
  17. 2021A comparison of LSTM and GRU networks for learning symbolic sequencescited by 24
  18. 2021Towards Diverse Paragraph Captioning for Untrimmed Videoscited by 22
  19. 2017Reinforced Video Captioning with Entailment Rewardscited by 21
  20. 2022Meta Spatio-Temporal Debiasing for Video Scene Graph Generationcited by 20
  21. 2024Samba: Synchronized Set-of-Sequences Modeling for Multiple Object Trackingcited by 20
  22. 2023Models See Hallucinations: Evaluating the Factuality in Video Captioningcited by 19
  23. 2021Adaptive Hierarchical Graph Reasoning with Semantic Coherence for Video-and-Language Inferencecited by 18
  24. 2022Selective Query-guided Debiasing for Video Corpus Moment Retrievalcited by 18
  25. 2021Question Answering Survey: Directions, Challenges, Datasets, Evaluation Matricescited by 17
  26. 2023Fine-grained Audible Video Descriptioncited by 17
  27. 2022GL-RG: Global-Local Representation Granularity for Video Captioningcited by 16
  28. 2024Generative Emergent Communication: Large Language Model is a Collective World Modelcited by 15
  29. 2021Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and Visioncited by 14
  30. 2023PointMBF: A Multi-scale Bidirectional Fusion Network for Unsupervised RGB-D Point Cloud Registrationcited by 14
  31. 2024FMM-Attack: A Flow-based Multi-modal Adversarial Attack on Video-based LLMscited by 14
  32. 2021Beam Search with Bidirectional Strategies for Neural Response Generationcited by 12
  33. 2021Relational Graph Learning for Grounded Video Description Generationcited by 12
  34. 2024HICEScore: A Hierarchical Metric for Image Captioning Evaluationcited by 12
  35. 2021PiSLTRc: Position-informed Sign Language Transformer with Content-aware Convolutioncited by 11
  36. 2021CLIP Meets Video Captioning: Concept-Aware Representation Learning Does Mattercited by 11
  37. 2022Non-Parallel Text Style Transfer with Self-Parallel Supervisioncited by 11
  38. 2022Unsupervised Flow-Aligned Sequence-to-Sequence Learning for Video Restorationcited by 11

50 most cited of 157 All 157 (JSON)