mlsanityData to 7 Oct 2026

Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

arXiv 1412.6632 ↗First posted 20 Dec 2014cs.CV cs.CL cs.LGcited by 94

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a multimodal Recurrent Neural Network (m-RNN) model for generating novel image captions. It directly models the probability distribution of generating a word given previous words and an image. Image captions are generated by sampling from this distribution. The model consists of two sub-networks: a deep recurrent neural network for sentences and a deep convolutional network for images. These two sub-networks interact with each other in a multimodal layer to form the whole m-RNN model. The effectiveness of our model is validated on four benchmark datasets (IAPR TC-12, Flickr 8K, Flickr 30K and MS COCO). Our model outperforms the state-of-the-art methods. In addition, we apply the m-RNN model to retrieval tasks for retrieving images or sentences, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval. The project page of this work is: www.stat.ucla.edu/~junhua.mao/m-RNN.html .

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Dec 2014.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2019: 1; 2020: 0; 2021: 29; 2022: 29; 2023: 15; 2024: 15; 2025: 5.

94 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  2. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  3. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  4. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  5. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  6. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  7. 2022Make-A-Story: Visual Memory Conditioned Consistent Story Generationcited by 76
  8. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  9. 2022X-DETR: A Versatile Architecture for Instance-wise Vision-Language Taskscited by 41
  10. 2022Comprehending and Ordering Semantics for Image Captioningcited by 33
  11. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
  12. 2023Multi-modal Machine Learning in Engineering Design: A Review and Future Directionscited by 27
38 more
  1. 2022Inference of captions from histopathological patchescited by 26
  2. 2023Towards Local Visual Modeling for Image Captioningcited by 26
  3. 2021Towards Open-World Text-Guided Face Image Generation and Manipulationcited by 25
  4. 2022Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioningcited by 25
  5. 2022Zero-Shot Video Captioning with Evolving Pseudo-Tokenscited by 25
  6. 2021A comparison of LSTM and GRU networks for learning symbolic sequencescited by 24
  7. 2021Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrievalcited by 23
  8. 2023Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generationcited by 23
  9. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  10. 2022Noise-aware Learning from Web-crawled Image-Text Data for Image Captioningcited by 22
  11. 2021OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generationcited by 20
  12. 2021Writing by Memorizing: Hierarchical Retrieval-based Medical Report Generationcited by 19
  13. 2023Foundation Model is Efficient Multimodal Multitask Model Selectorcited by 16
  14. 2021Parts2Words: Learning Joint Embedding of Point Clouds and Texts by Bidirectional Matching between Parts and Wordscited by 14
  15. 2022Cross-modal Semantic Enhanced Interaction for Image-Sentence Retrievalcited by 14
  16. 2023A Systematic Review of Deep Learning-based Research on Radiology Report Generationcited by 14
  17. 2021Group-based Distinctive Image Captioning with Memory Attentioncited by 13
  18. 2022CODER: Coupled Diversity-Sensitive Momentum Contrastive Learning for Image-Text Retrievalcited by 13
  19. 2022Learning Distinct and Representative Styles for Image Captioningcited by 12
  20. 2024VIXEN: Visual Text Comparison Network for Image Difference Captioningcited by 12
  21. 2024A Survey on Integrated Sensing, Communication, and Computationcited by 12
  22. 2022What is Where by Looking: Weakly-Supervised Open-World Phrase-Grounding without Text Inputscited by 11
  23. 2022CLIP-Diffusion-LM: Apply Diffusion Model on Image Captioningcited by 11
  24. 2024Resolving Word Vagueness with Scenario-guided Adapter for Natural Language Inferencecited by 11
  25. 2021Sensor-Augmented Egocentric-Video Captioning with Dynamic Modal Attentioncited by 9
  26. 2023NICE: CVPR 2023 Challenge on Zero-shot Image Captioningcited by 9
  27. 2024Atari-GPT: Benchmarking Multimodal Large Language Models as Low-Level Policies in Atari Gamescited by 8
  28. 2022Unpaired Image Captioning by Image-level Weakly-Supervised Visual Concept Recognitioncited by 7
  29. 2022On Distinctive Image Captioning via Comparing and Reweightingcited by 7
  30. 2022Explicit Image Caption Editingcited by 7
  31. 2023Reverse Stable Diffusion: What prompt was used to generate this image?cited by 7
  32. 2021Integrating Information Theory and Adversarial Learning for Cross-modal Retrievalcited by 6
  33. 2022Automatic Rule Induction for Interpretable Semi-Supervised Learningcited by 6
  34. 2022Distinctive Image Captioning via CLIP Guided Group Optimizationcited by 6
  35. 2021Image Captioning using Deep Stacked LSTMs, Contextual Word Embeddings and Data Augmentationcited by 5
  36. 2021A Comprehensive Review of the Video-to-Text Problemcited by 5
  37. 2021Topos and Stacks of Deep Neural Networkscited by 5
  38. 2021Geometry Attention Transformer with Position-aware LSTMs for Image Captioningcited by 5

50 most cited of 94 All 94 (JSON)