mlsanityData to 14 Oct 2026

Auto-Encoding Scene Graphs for Image Captioning

arXiv 1812.02378 ↗First posted 6 Dec 2018cs.CVcited by 255

Not on the map. Why?

Cited by 20 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose Scene Graph Auto-Encoder (SGAE) that incorporates the language inductive bias into the encoder-decoder image captioning framework for more human-like captions. Intuitively, we humans use the inductive bias to compose collocations and contextual inference in discourse. For example, when we see the relation `person on bike', it is natural to replace `on' with `ride' and infer `person riding bike on a road' even the `road' is not evident. Therefore, exploiting such bias as a language prior is expected to help the conventional encoder-decoder models less likely overfit to the dataset bias and focus on reasoning. Specifically, we use the scene graph --- a directed graph (G) where an object node is connected by adjective nodes and relationship nodes --- to represent the complex structural layout of both image (I) and sentence (S). In the textual domain, we use SGAE to learn a dictionary (D) that helps to reconstruct sentences in the Srightarrow G rightarrow D rightarrow S pipeline, where D encodes the desired language prior; in the vision-language domain, we use the shared D to guide the encoder-decoder in the Irightarrow Grightarrow D rightarrow S pipeline. Thanks to the scene graph representation and shared dictionary, the inductive bias is transferred across domains in principle. We validate the effectiveness of SGAE on the challenging MS-COCO image captioning benchmark, e.g., our SGAE-based single-model achieves a new state-of-the-art 127.8 CIDEr-D on the Karpathy split, and a competitive 125.5 CIDEr-D (c40) on the official server even compared to other ensemble models.

Key facts. Cited by 255 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 76; 2022: 70; 2023: 41; 2024: 32; 2025: 21; 2026: 15.

255 papers.

Sort:

By area:

  1. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  2. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  3. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  4. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 138
  5. 2021Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generationcited by 137
  6. 2021A Comprehensive Survey of Scene Graphs: Generation and Applicationcited by 125
  7. 2021Similarity Reasoning and Filtration for Image-Text Matchingcited by 121
  8. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  9. 2021Causal Attention for Vision-Language Taskscited by 111
  10. 2021Energy-Based Learning for Scene Graph Generationcited by 104
  11. 2021SGTR: End-to-end Scene Graph Generation with Transformercited by 103
  12. 2021Spatial-Temporal Transformer for Dynamic Scene Graph Generationcited by 96
38 more
  1. 2022RelTR: Relation Transformer for Scene Graph Generationcited by 96
  2. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  3. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  4. 2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
  5. 2022Scene Graph Generation: A Comprehensive Surveycited by 73
  6. 2021Dual-Level Collaborative Transformer for Image Captioningcited by 71
  7. 2021Recovering the Unbiased Scene Graphs from the Biased Onescited by 71
  8. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  9. 2021Learning of Visual Relations: The Devil is in the Tailscited by 65
  10. 2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
  11. 2021Fully Convolutional Scene Graph Generationcited by 57
  12. 2022Fine-Grained Scene Graph Generation with Data Transfercited by 57
  13. 2022GRIT: Faster and Better Image captioning Transformer Using Dual Visual Featurescited by 56
  14. 2022A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspectivecited by 56
  15. 2021CPTR: Full Transformer Network for Image Captioningcited by 55
  16. 2022Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuningcited by 54
  17. 2023Action Scene Graphs for Long-Form Understanding of Egocentric Videoscited by 51
  18. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  19. 2021Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generationcited by 47
  20. 2024EGTR: Extracting Graph from Transformer for Scene Graph Generationcited by 47
  21. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
  22. 2024An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Modelscited by 45
  23. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
  24. 2021Structured Sparse R-CNN for Direct Scene Graph Generationcited by 44
  25. 2022Deep Learning Approaches on Image Captioning: A Reviewcited by 43
  26. 2024From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Modelscited by 43
  27. 2021Unifying Multimodal Transformer for Bi-directional Image and Text Generationcited by 41
  28. 2021Injecting Semantic Concepts into End-to-End Image Captioningcited by 41
  29. 2021Exemplar-Based Open-Set Panoptic Segmentation Networkcited by 38
  30. 2022Retrieval-Augmented Transformer for Image Captioningcited by 38
  31. 2022Semantic-Conditional Diffusion Networks for Image Captioningcited by 38
  32. 2023Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representationscited by 37
  33. 2021Probabilistic Modeling of Semantic Ambiguity for Scene Graph Generationcited by 35
  34. 2024MeaCap: Memory-Augmented Zero-shot Image Captioningcited by 35
  35. 2022Comprehending and Ordering Semantics for Image Captioningcited by 33
  36. 2023ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishingcited by 33
  37. 2023Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionalitycited by 32
  38. 2024Enhancing Video-Language Representations with Structural Spatio-Temporal Alignmentcited by 32

50 most cited of 255 All 255 (JSON)