mlsanityData to 14 Oct 2026

Deep Visual-Semantic Alignments for Generating Image Descriptions

arXiv 1412.2306 ↗First posted 7 Dec 2014cs.CVcited by 968

Not on the map. Why?

Cited by 169 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions significantly outperform retrieval baselines on both full images and on a new dataset of region-level annotations.

Key facts. Cited by 968 papers in the map (data through 14 Oct 2026). First posted on arXiv 7 Dec 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 2; 2019: 1; 2020: 1; 2021: 202; 2022: 213; 2023: 236; 2024: 134; 2025: 119; 2026: 60.

968 papers.

Sort:

By area:

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  6. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  8. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  9. 2023A Survey on Multimodal Large Language Modelscited by 909
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  11. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  12. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
38 more
  1. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  2. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  3. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  4. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  5. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  6. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  7. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  8. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  9. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 485
  10. 2018An Algorithmic Perspective on Imitation Learningcited by 459
  11. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  12. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  13. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 410
  14. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  15. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 349
  16. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  17. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  18. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  19. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  20. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  21. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
  22. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  23. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  24. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  25. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  26. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  27. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  28. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 220
  29. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 215
  30. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  31. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  32. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  33. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 187
  34. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 183
  35. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  36. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  37. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
  38. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 166

50 most cited of 968 All 968 (JSON)