mlsanityData to 7 Oct 2026

World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filtering

arXiv 2409.20424 ↗First posted 30 Sep 2024cs.CV cs.AIcited by 14

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

also in 1 more
Abstract (arXiv)

Recent advances in Vision-Language Models (VLMs) and the scarcity of high-quality multi-modal alignment data have inspired numerous researches on synthetic VLM data generation. The conventional norm in VLM data construction uses a mixture of specialists in caption and OCR, or stronger VLM APIs and expensive human annotation. In this paper, we present World to Code (W2C), a meticulously curated multi-modal data construction pipeline that organizes the final generation output into a Python code format. The pipeline leverages the VLM itself to extract cross-modal information via different prompts and filter the generated outputs again via a consistency filtering strategy. Experiments have demonstrated the high quality of W2C by improving various existing visual question answering and visual grounding benchmarks across different VLMs. Further analysis also demonstrates that the new code parsing ability of VLMs presents better cross-modal equivalence than the commonly used detail caption ability. Our code is available at https://github.com/foundation-multimodal-models/World2Code.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 52 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

34 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation22 Nov
  2. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  3. 2023Tag2Text: Guiding Vision-Language Model via Image Tagging10 Mar

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
19 more
  1. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  2. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  3. 2022Large Language Models Can Self-Improvecited by 656
  4. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  5. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  6. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  7. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  8. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  9. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  10. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  11. 2022DETRs with Collaborative Hybrid Assignments Trainingcited by 273
  12. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  13. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  14. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  15. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  16. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 98
  17. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  18. 2021Document Collection Visual Question Answeringcited by 45
  19. 2023Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognitioncited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 2.

14 papers.

Sort:

By area:

  1. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 67
  2. 2025Scalable Vision Language Model Training via High Quality Data Curationcited by 54
  3. 2025VideoWorld: Exploring Knowledge Learning from Unlabeled Videoscited by 42
  4. 2025The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformercited by 28
  5. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  6. 2025Scheduling Your LLM Reinforcement Learning with Reasoning Treescited by 11
  7. 2025Thinking With Bounding Boxes: Enhancing Spatio-Temporal Video Grounding via Reinforcement Fine-Tuningcited by 11
  8. 2026VideoWorld 2: Learning Transferable Knowledge from Real-world Videoscited by 11
  9. 2025SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Modelcited by 10
  10. 2025UVE: Are MLLMs Unified Evaluators for AI-Generated Videos?cited by 6
  11. 2025SAILViT: Towards Robust and Generalizable Visual Backbones for MLLMs via Gradual Feature Refinementcited by 4
  12. 2025Scale-Distribution Decoupling: Enabling Stable and Effective Training of Large Language Modelscited by 3
2 more
  1. 2025SVAC: Scaling Is All You Need For Referring Video Object Segmentationcited by 1
  2. 2026Evidence-Grounded Trustworthy Multimodal Reasoning and Evaluation Benchmark in Complex Urban Scenescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SynthVLM: Towards High-Quality and Efficient Synthesis of Image-Caption Datasets for Vision-Language Models
  2. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  3. 2025Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
  4. 2025Text-Only Data Synthesis for Vision Language Model Training
  5. 2023Enhancing Vision-Language Pre-Training with Jointly Learned Questioner and Dense Captioner
  6. 2025VisCon-100K: Leveraging Contextual Web Data for Fine-tuning Vision Language Models
  7. 2025Vision-Language Models Struggle to Align Entities across Modalities
  8. 2025Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoring