mlsanityData to 7 Oct 2026

ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph

arXiv 2006.16934 ↗First posted 30 Jun 2020cs.CV cs.CLcited by 211

Not on the map. Why?

Cited by 45 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose a knowledge-enhanced approach, ERNIE-ViL, which incorporates structured knowledge obtained from scene graphs to learn joint representations of vision-language. ERNIE-ViL tries to build the detailed semantic connections (objects, attributes of objects and relationships between objects) across vision and language, which are essential to vision-language cross-modal tasks. Utilizing scene graphs of visual scenes, ERNIE-ViL constructs Scene Graph Prediction tasks, i.e., Object Prediction, Attribute Prediction and Relationship Prediction tasks in the pre-training phase. Specifically, these prediction tasks are implemented by predicting nodes of different types in the scene graph parsed from the sentence. Thus, ERNIE-ViL can learn the joint representations characterizing the alignments of the detailed semantics across vision and language. After pre-training on large scale image-text aligned datasets, we validate the effectiveness of ERNIE-ViL on 5 cross-modal downstream tasks. ERNIE-ViL achieves state-of-the-art performances on all these tasks and ranks the first place on the VCR leaderboard with an absolute improvement of 3.7%.

Key facts. Cited by 211 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 56; 2022: 75; 2023: 48; 2024: 22; 2025: 7; 2026: 3.

211 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  5. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  6. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  7. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  8. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  9. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  10. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  11. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  12. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
38 more
  1. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  2. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  3. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  4. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  5. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  6. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  7. 2021General Facial Representation Learning in a Visual-Linguistic Mannercited by 193
  8. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  9. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  10. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182
  11. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  12. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  13. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  14. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  15. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  16. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  17. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 138
  18. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  19. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  20. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoningcited by 119
  21. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  22. 2021Causal Attention for Vision-Language Taskscited by 111
  23. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 110
  24. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  25. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  26. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98
  27. 2021M6: A Chinese Multimodal Pretrainercited by 91
  28. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  29. 2022CLIP-Event: Connecting Text and Images with Event Structurescited by 86
  30. 2021WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Trainingcited by 84
  31. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learningcited by 84
  32. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  33. 2021Detecting Harmful Memes and Their Targetscited by 77
  34. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  35. 2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
  36. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  37. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 71
  38. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillationcited by 70

50 most cited of 211 All 211 (JSON)