mlsanityData to 7 Oct 2026

MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning

arXiv 2005.05402 ↗First posted 11 May 2020cs.CL cs.CV cs.LGcited by 79

Not on the map. Why?

Abstract (arXiv)

Generating multi-sentence descriptions for videos is one of the most challenging captioning tasks due to its high requirements for not only visual relevance but also discourse-based coherence across the sentences in the paragraph. Towards this goal, we propose a new approach called Memory-Augmented Recurrent Transformer (MART), which uses a memory module to augment the transformer architecture. The memory module generates a highly summarized memory state from the video segments and the sentence history so as to help better prediction of the next sentence (w.r.t. coreference and repetition aspects), thus encouraging coherent paragraph generation. Extensive experiments, human evaluations, and qualitative analyses on two popular datasets ActivityNet Captions and YouCookII show that MART generates more coherent and less repetitive paragraph captions than baseline methods, while maintaining relevance to the input video events. All code is available open-source at: https://github.com/jayleicn/recurrent-transformer

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 24; 2023: 13; 2024: 18; 2025: 8; 2026: 1.

79 papers.

Sort:

By area:

  1. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  2. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  3. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  4. 2022Recurrent Memory Transformercited by 197
  5. 2022Recurrent Video Restoration Transformer with Guided Deformable Attentioncited by 168
  6. 2021End-to-End Dense Video Captioning with Parallel Decodingcited by 133
  7. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 115
  8. 2021CLIP-It! Language-Guided Video Summarizationcited by 99
  9. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 97
  10. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  11. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  12. 2023DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Trainingcited by 84
38 more
  1. 2022StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuationcited by 74
  2. 2021CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representationscited by 72
  3. 2022VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captionerscited by 69
  4. 2022CLIP-TSA: CLIP-Assisted Temporal Self-Attention for Weakly-Supervised Video Anomaly Detectioncited by 67
  5. 2022Video Transformers: A Surveycited by 63
  6. 2022Real-time Online Video Detection with Temporal Smoothing Transformerscited by 58
  7. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  8. 2022P3IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervisioncited by 52
  9. 2021Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualizationcited by 51
  10. 2021Improving Generation and Evaluation of Visual Stories via Semantic Consistencycited by 43
  11. 2022Local-Global Context Aware Transformer for Language-Guided Video Segmentationcited by 41
  12. 2024In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Misscited by 36
  13. 2022Visual Abductive Reasoningcited by 30
  14. 2023A Review of Deep Learning for Video Captioningcited by 28
  15. 2023Text with Knowledge Graph Augmented Transformer for Video Captioningcited by 27
  16. 2022VLTinT: Visual-Linguistic Transformer-in-Transformer for Coherent Video Paragraph Captioningcited by 25
  17. 2024MECD: Unlocking Multi-Event Causal Discovery in Video Reasoningcited by 23
  18. 2021Towards Diverse Paragraph Captioning for Untrimmed Videoscited by 22
  19. 2022VLCap: Vision-Language with Contrastive Learning for Coherent Video Paragraph Captioningcited by 19
  20. 2023Models See Hallucinations: Evaluating the Factuality in Video Captioningcited by 19
  21. 2021Video-aided Unsupervised Grammar Inductioncited by 17
  22. 2023Recurrent Action Transformer with Memorycited by 17
  23. 2024Learning Video Context as Interleaved Multimodal Sequencescited by 16
  24. 2022AutoTransition: Learning to Recommend Video Transition Effectscited by 13
  25. 2024A Modular Approach for Multimodal Summarization of TV Showscited by 13
  26. 2022Hierarchical3D Adapters for Long Video-to-text Summarizationcited by 12
  27. 2022Explain My Surprise: Learning Efficient Long-Term Memory by Predicting Uncertain Outcomescited by 11
  28. 2021Sensor-Augmented Egocentric-Video Captioning with Dynamic Modal Attentioncited by 9
  29. 2023Learning Grounded Vision-Language Representation for Versatile Understanding in Untrimmed Videoscited by 8
  30. 2024From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Modelscited by 8
  31. 2025VideoA11y: Method and Dataset for Accessible Video Descriptioncited by 8
  32. 2023Story Visualization by Online Text Augmentation with Context Memorycited by 7
  33. 2021CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learningcited by 6
  34. 2022Relevance-based Margin for Contrastively-trained Video Retrieval Modelscited by 6
  35. 2024RAP: Retrieval-Augmented Planner for Adaptive Procedure Planning in Instructional Videoscited by 6
  36. 2021A Comprehensive Review of the Video-to-Text Problemcited by 5
  37. 2022CREATE: A Benchmark for Chinese Short Video Retrieval and Title Generationcited by 5
  38. 2022Contextual Explainable Video Representation: Human Perception-based Understandingcited by 5

50 most cited of 79 All 79 (JSON)