mlsanityData to 7 Oct 2026

Long-term Recurrent Convolutional Networks for Visual Recognition and Description

arXiv 1411.4389 ↗First posted 17 Nov 2014cs.CVcited by 483

Not on the map. Why?

Abstract (arXiv)

Models based on deep convolutional networks have dominated recent image interpretation tasks; we investigate whether models which are also recurrent, or "temporally deep", are effective for tasks involving sequences, visual and otherwise. We develop a novel recurrent convolutional architecture suitable for large-scale visual learning which is end-to-end trainable, and demonstrate the value of these models on benchmark video recognition tasks, image description and retrieval problems, and video narration challenges. In contrast to current models which assume a fixed spatio-temporal receptive field or simple temporal averaging for sequential processing, recurrent convolutional models are "doubly deep"' in that they can be compositional in spatial and temporal "layers". Such models may have advantages when target concepts are complex and/or training data are limited. Learning long-term dependencies is possible when nonlinearities are incorporated into the network state updates. Long-term RNN models are appealing in that they directly can map variable-length inputs (e.g., video frames) to variable length outputs (e.g., natural language text) and can model complex temporal dynamics; yet they can be optimized with backpropagation. Our recurrent long-term models are directly connected to modern visual convnet models and can be jointly trained to simultaneously learn temporal dynamics and convolutional perceptual representations. Our results show such models have distinct advantages over state-of-the-art models for recognition or generation which are separately defined and/or optimized.

Key facts. Cited by 483 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Nov 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 0; 2021: 155; 2022: 117; 2023: 78; 2024: 50; 2025: 48; 2026: 33.

483 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  3. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  4. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  5. 2021Video Transformer Networkcited by 271
  6. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  7. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  8. 2022Learning Video Representations from Large Language Modelscited by 204
  9. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  10. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  11. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 181
  12. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
38 more
  1. 2021ASFormer: Transformer for Action Segmentationcited by 166
  2. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  3. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  4. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  5. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  6. 2022Natural Language Descriptions of Deep Visual Featurescited by 136
  7. 2021MoViNets: Mobile Video Networks for Efficient Video Recognitioncited by 135
  8. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  9. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 104
  10. 2021SGCN:Sparse Graph Convolution Network for Pedestrian Trajectory Predictioncited by 97
  11. 2021Spatial-Temporal Transformer for Dynamic Scene Graph Generationcited by 95
  12. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93
  13. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learningcited by 90
  14. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  15. 2021Long Short-Term Transformer for Online Action Detectioncited by 87
  16. 2021ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spottingcited by 84
  17. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  18. 2021OadTR: Online Action Detection with Transformerscited by 83
  19. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  20. 2021Adaptive Focus for Efficient Video Recognitioncited by 76
  21. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 74
  22. 2021Object-Region Video Transformerscited by 73
  23. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  24. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  25. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  26. 2022FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videoscited by 69
  27. 2023Deep Learning for Time Series Classification and Extrinsic Regression: A Current Surveycited by 68
  28. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  29. 2021iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysiscited by 63
  30. 2021Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Frameworkcited by 52
  31. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52
  32. 2021E2(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognitioncited by 50
  33. 2023AutoAD: Movie Description in Contextcited by 50
  34. 2021AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognitioncited by 48
  35. 2021Split, embed and merge: An accurate table structure recognizercited by 42
  36. 2021Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signalscited by 41
  37. 2021An Overview of Backdoor Attacks Against Deep Neural Networks and Possible Defencescited by 41
  38. 2021JRDB-Act: A Large-scale Dataset for Spatio-temporal Action, Social Group and Activity Detectioncited by 38

50 most cited of 483 All 483 (JSON)