Constructing Image-Text Pair Dataset from Books
Not on the map. Why?
Abstract (arXiv)
Digital archiving is becoming widespread owing to its effectiveness in protecting valuable books and providing knowledge to many people electronically. In this paper, we propose a novel approach to leverage digital archives for machine learning. If we can fully utilize such digitized data, machine learning has the potential to uncover unknown insights and ultimately acquire knowledge autonomously, just like humans read books. As a first step, we design a dataset construction pipeline comprising an optical character reader (OCR), an object detector, and a layout analyzer for the autonomous extraction of image-text pairs. In our experiments, we apply our pipeline on old photo books to construct an image-text pair dataset, showing its effectiveness in image-text retrieval and insight extraction.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 3 Oct 2023.
Builds on
21 of 33 references.
References · 21
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2017Mask R-CNNcited by 7,277
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
- 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
- 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
- 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
- 2015VQA: Visual Question Answeringcited by 2,506
- 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
- 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
9 more
- 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
- 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
- 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
- 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
- 2014Show and Tell: A Neural Image Caption Generatorcited by 879
- 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
- 2019Character Region Awareness for Text Detectioncited by 278
- 2019What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysiscited by 191
- 2021FooDI-ML: a large multi-language dataset of food, drinks and groceries images and descriptionscited by 4
Built on by
Citing papers per year: 2024: 2; 2025: 1.
3 papers.