Table and Image Generation for Investigating Knowledge of Entities in Pre-trained Vision and Language Models
Not on the map. Why?
Abstract (arXiv)
In this paper, we propose a table and image generation task to verify how the knowledge about entities acquired from natural language is retained in Vision & Language (V&L) models. This task consists of two parts: the first is to generate a table containing knowledge about an entity and its related image, and the second is to generate an image from an entity with a caption and a table containing related knowledge of the entity. In both tasks, the model must know the entities used to perform the generation properly. We created the Wikipedia Table and Image Generation (WikiTIG) dataset from about 200,000 infoboxes in English Wikipedia articles to perform the proposed tasks. We evaluated the performance on the tasks with respect to the above research question using the V&L model OFA, which has achieved state-of-the-art results in multiple tasks. Experimental results show that OFA forgets part of its entity knowledge by pre-training as a complement to improve the performance of image related tasks.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 3 Jun 2023.
Builds on
29 of 40 references.
References · 29
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2021Zero-Shot Text-to-Image Generationcited by 4,276
- 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
- 2016Improved Techniques for Training GANscited by 2,938
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
17 more
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
- 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
- 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
- 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
- 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
- 2016Generative Adversarial Text to Image Synthesiscited by 735
- 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
- 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
- 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
- 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
- 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
- 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
- 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
- 2021Text-to-Table: A New Way of Information Extractioncited by 47
- 2021Does Vision-and-Language Pretraining Improve Lexical Grounding?cited by 25
- 2021Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Modelscited by 18
- 2022How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input?cited by 4
Built on by
Citing papers per year: 2025: 1.
1 paper.