mlsanityData to 7 Oct 2026

A Hierarchical Approach for Generating Descriptive Image Paragraphs

arXiv 1611.06607 ↗First posted 20 Nov 2016cs.CV cs.CLcited by 104

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent progress on image captioning has made it possible to generate novel sentences describing images in natural language, but compressing an image into a single sentence can describe visual content in only coarse detail. While one new captioning approach, dense captioning, can potentially describe images in finer levels of detail by captioning many regions within an image, it in turn is unable to produce a coherent story for an image. In this paper we overcome these limitations by generating entire paragraphs for describing images, which can tell detailed, unified stories. We develop a model that decomposes both images and paragraphs into their constituent parts, detecting semantic regions in images and using a hierarchical recurrent neural network to reason about language. Linguistic analysis confirms the complexity of the paragraph generation task, and thorough experiments on a new dataset of image and paragraph pairs demonstrate the effectiveness of our approach.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 14; 2023: 27; 2024: 27; 2025: 12; 2026: 5.

104 papers.

Sort:

By area:

  1. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  2. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  3. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  4. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  5. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  6. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  7. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 160
  8. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  9. 2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
  10. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  11. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  12. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
38 more
  1. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  2. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  3. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  4. 2022Competence-based Multimodal Curriculum Learning for Medical Report Generationcited by 74
  5. 2021Contrastive Attention for Automatic Chest X-ray Report Generationcited by 71
  6. 2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Modelscited by 68
  7. 2022AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generationcited by 65
  8. 2024Red Teaming Visual Language Modelscited by 62
  9. 2022NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Taskscited by 60
  10. 2022Cross-modal Prototype Driven Network for Radiology Report Generationcited by 58
  11. 2021GaitSet: Cross-view Gait Recognition through Utilizing Gait as a Deep Setcited by 57
  12. 2023DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedbackcited by 57
  13. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 56
  14. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  15. 2021Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generationcited by 47
  16. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 46
  17. 2024Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratingscited by 44
  18. 2023Visual Instruction Tuning with Polite Flamingocited by 38
  19. 2024DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perceptioncited by 32
  20. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  21. 2021Panoptic Narrative Groundingcited by 27
  22. 2024Safety Alignment for Vision Language Modelscited by 27
  23. 2024VILA2: VILA Augmented VILAcited by 25
  24. 2021Medical-VLBERT: Medical Visual Language BERT for COVID-19 CT Report Generation With Alternate Learningcited by 24
  25. 2021Automated Generation of Accurate & Fluent Medical X-ray Reportscited by 24
  26. 2023Connecting Vision and Language with Video Localized Narrativescited by 24
  27. 2023Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generationcited by 23
  28. 2021Towards Diverse Paragraph Captioning for Untrimmed Videoscited by 22
  29. 2022Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioningcited by 21
  30. 2022CAMANet: Class Activation Map Guided Attention Network for Radiology Report Generationcited by 21
  31. 2024The Evolution of Multimodal Model Architecturescited by 21
  32. 2023IC3: Image Captioning by Committee Consensuscited by 20
  33. 2021Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generationcited by 19
  34. 2022Attributed Abnormality Graph Embedding for Clinically Accurate X-Ray Report Generationcited by 17
  35. 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
  36. 2024SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Datacited by 14
  37. 2025MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decodingcited by 14
  38. 2021FixMyPose: Pose Correctional Captioning and Retrievalcited by 12

50 most cited of 104 All 104 (JSON)