mlsanityData to 13 Oct 2026

Constructing Multimodal Datasets from Scratch for Rapid Development of a Japanese Visual Language Model

arXiv 2410.22736 ↗First posted 30 Oct 2024cs.CLcited by 6

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

To develop high-performing Visual Language Models (VLMs), it is essential to prepare multimodal resources, such as image-text pairs, interleaved data, and instruction data. While multimodal resources for English are abundant, there is a significant lack of corresponding resources for non-English languages, such as Japanese. To address this problem, we take Japanese as a non-English language and propose a method for rapidly creating Japanese multimodal datasets from scratch. We collect Japanese image-text pairs and interleaved data from web archives and generate Japanese instruction data directly from images using an existing VLM. Our experimental results show that a VLM trained on these native datasets outperforms those relying on machine-translated content.

Key facts. Cited by 6 papers in the map (data through 13 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 34 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

22 of 34 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  3. 2023VILA: On Pre-training for Visual Language Models12 Dec
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,141
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,068
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,649
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,506
  8. 2016A Diagram Is Worth A Dozen Imagescited by 796
  9. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  10. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 339
  11. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  12. 2024Evolutionary Optimization of Model Merging Recipescited by 204
5 more
  1. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  2. 2024LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMscited by 32
  3. 2024Heron-Bench: A Benchmark for Evaluating Vision Language Models in Japanesecited by 12
  4. 2024JDocQA: Japanese Document Question Answering Dataset for Generative Language Modelscited by 9
  5. 2016Semantic Parsing to Probabilistic Programs for Situated Question Answeringcited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2026JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluationcited by 3
  2. 2025Harnessing PDF Data for Improving Japanese Large Multimodal Modelscited by 1
  3. 2025DEJIMA: A Novel Large-scale Japanese Dataset for Image Captioning and Visual Question Answeringcited by 1
  4. 2025WAON: A Large-Scale Japanese Image-Text Dataset for Cultural Adaptation in Contrastive Vision-Language Modelscited by 0
  5. 2025JMMMU-Pro: Image-based Japanese Multi-discipline Multimodal Understanding Benchmark via Vibe Benchmark Constructioncited by 0
  6. 2026Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Multimodal Large Language Models on Vertically Written Japanese Text
  2. 2024Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models
  3. 2024The Revolution of Multimodal Large Language Models: A Survey
  4. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  5. 2025Aya Vision: Advancing the Frontier of Multilingual Multimodality
  6. 2026JaWildText: A Benchmark for Vision-Language Models on Japanese Scene Text Understanding
  7. 2024Visualization Literacy of Multimodal Large Language Models: A Comparative Study
  8. 2023MLLMs-Augmented Visual-Language Representation Learning