mlsanityData to 14 Oct 2026

From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledge

arXiv 1511.03292 ↗First posted 10 Nov 2015cs.CV cs.AI cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

In this paper we propose the construction of linguistic descriptions of images. This is achieved through the extraction of scene description graphs (SDGs) from visual scenes using an automatically constructed knowledge base. SDGs are constructed using both vision and reasoning. Specifically, commonsense reasoning is applied on (a) detections obtained from existing perception methods on given images, (b) a "commonsense" knowledge base constructed using natural language processing of image annotations and (c) lexical ontological knowledge from resources such as WordNet. Amazon Mechanical Turk(AMT)-based evaluations on Flickr8k, Flickr30k and MS-COCO datasets show that in most cases, sentences auto-constructed from SDGs obtained by our method give a more relevant and thorough description of an image than a recent state-of-the-art image caption based approach. Our Image-Sentence Alignment Evaluation results are also comparable to that of the recent state-of-the art approaches.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 0; 2023: 5; 2024: 12; 2025: 6; 2026: 5.

35 papers.

Sort:

By area:

  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  2. 2021A Comprehensive Survey of Scene Graphs: Generation and Applicationcited by 125
  3. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  4. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 53
  5. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 50
  6. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
  7. 2024FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Modelcited by 45
  8. 2023CLAIR: Evaluating Image Captions with Large Language Modelscited by 43
  9. 2021Transparent Human Evaluation for Image Captioningcited by 40
  10. 2021UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learningcited by 37
  11. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  12. 2025Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectivescited by 18
23 more
  1. 2024Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Trainingcited by 16
  2. 2021QACE: Asking Questions to Evaluate an Image Captioncited by 15
  3. 2024BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cuescited by 13
  4. 2024HICEScore: A Hierarchical Metric for Image Captioning Evaluationcited by 12
  5. 2024DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioningcited by 11
  6. 2023Adaptive Visual Scene Understanding: Incremental Scene Graph Generationcited by 8
  7. 2024Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extractioncited by 7
  8. 2025EXPERT: An Explainable Image Captioning Evaluation Metric with Structured Explanationscited by 7
  9. 2023PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioningcited by 5
  10. 2024Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimizationcited by 5
  11. 2025Evaluation of Multilingual Image Captioning: How far can we get with CLIP models?cited by 5
  12. 2025VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captionscited by 5
  13. 2021iReason: Multimodal Commonsense Reasoning using Videos and Natural Language with Interpretabilitycited by 4
  14. 2023MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Typescited by 4
  15. 2025L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Trainingcited by 2
  16. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 1
  17. 2026MLLM-as-a-Judge Exhibits Model Preference Biascited by 1
  18. 2026Rigel: Self-Distilled Score Adaptation for Image and Video Captioning Evaluationcited by 1
  19. 2024Generated Contents Enrichmentcited by 0
  20. 2025A Conformal Risk Control Framework for Granular Word Assessment and Uncertainty Calibration of CLIPScore Quality Estimatescited by 0
  21. 2026ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMscited by 0
  22. 2026MSD-Score: Multi-Scale Distributional Scoring for Reference-Free Image Caption Evaluationcited by 0
  23. 2026BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Modelcited by 0