mlsanityData to 7 Oct 2026

Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation

arXiv 2502.14846 ↗First posted 20 Feb 2025cs.CV cs.CLcited by 54

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Reasoning about images with rich text, such as charts and documents, is a critical application of vision-language models (VLMs). However, VLMs often struggle in these domains due to the scarcity of diverse text-rich vision-language data. To address this challenge, we present CoSyn, a framework that leverages the coding capabilities of text-only large language models (LLMs) to automatically create synthetic text-rich multimodal data. Given input text describing a target domain (e.g., "nutrition fact labels"), CoSyn prompts an LLM to generate code (Python, HTML, LaTeX, etc.) for rendering synthetic images. With the underlying code as textual representations of the synthetic images, CoSyn can generate high-quality instruction-tuning data, again relying on a text-only LLM. Using CoSyn, we constructed a dataset comprising 400K images and 2.7M rows of vision-language instruction-tuning data. Comprehensive experiments on seven benchmarks demonstrate that models trained on our synthetic data achieve state-of-the-art performance among competitive open-source models, including Llama 3.2, and surpass proprietary models such as GPT-4V and Gemini 1.5 Flash. Furthermore, CoSyn can produce synthetic pointing data, enabling VLMs to ground information within input images, showcasing its potential for developing multimodal agents capable of acting in real-world environments.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 76 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

49 of 76 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models9 Dec
  2. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  3. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  4. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation14 May

Other references · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2019Towards VQA Models That Can Readcited by 1,696
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
33 more
  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  3. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  4. 2016A Diagram Is Worth A Dozen Imagescited by 793
  5. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  6. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  7. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  8. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  9. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  10. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  11. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  12. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  13. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  14. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  15. 2019Scene Text Visual Question Answeringcited by 327
  16. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  17. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  18. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  19. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  20. 2016Driving in the Matrix: Can Virtual Worlds Replace Human-Generated Annotations for Real World Tasks?cited by 194
  21. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  22. 2018TallyQA: Answering Complex Counting Questionscited by 152
  23. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 127
  24. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 116
  25. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  26. 2024TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domainscited by 68
  27. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 66
  28. 2024OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMscited by 64
  29. 2024DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflowscited by 43
  30. 2024Distill Visual Chart Reasoning Ability from LLMs to MLLMscited by 30
  31. 2024Chart-based Reasoning: Transferring Capabilities from LLMs to VLMscited by 20
  32. 2024Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustnesscited by 17
  33. 2024SBS Figures: Pre-training Figure QA from Stage-by-Stage Synthesized Imagescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 21; 2026: 33.

54 papers.

Sort:

By area:

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  2. 2025FineVision: Open Data Is All You Needcited by 51
  3. 2025T2I-ReasonBench: Benchmarking Reasoning-Informed Text-to-Image Generationcited by 38
  4. 2026STEP3-VL-10B Technical Reportcited by 28
  5. 2025VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learningcited by 18
  6. 2025VisCoder: Fine-Tuning LLMs for Executable Python Visualization Code Generationcited by 13
  7. 2025BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuningcited by 13
  8. 2025DeepSketcher: Internalizing Visual Manipulation for Multimodal Reasoningcited by 13
  9. 2026DM0: An Embodied-Native Vision-Language-Action Model towards Physical AIcited by 12
  10. 2026MolmoPoint: Better Pointing for VLMs with Grounding Tokenscited by 11
  11. 2026OCRVerse: Towards Holistic OCR in End-to-End Vision-Language Modelscited by 10
  12. 2025VACoT: Rethinking Visual Data Augmentation with VLMscited by 8
38 more
  1. 2026ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratchcited by 8
  2. 2026Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utilitycited by 8
  3. 2026Phi-4-reasoning-vision-15B Technical Reportcited by 8
  4. 2025ChartGen: Scaling Chart Understanding Via Code-Guided Synthetic Chart Generationcited by 7
  5. 2025Be My Eyes: Extending Large Language Models to New Modalities Through Multi-Agent Collaborationcited by 5
  6. 2026ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understandingcited by 5
  7. 2026Infinity-Parser2 Technical Reportcited by 4
  8. 2026Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillationcited by 4
  9. 2025VisCoder2: Building Multi-Language Visualization Coding Agentscited by 3
  10. 2026Enhancing Multi-Modal LLMs Reasoning via Difficulty-Aware Group Normalizationcited by 3
  11. 2026Omni-I2C: A Holistic Benchmark for High-Fidelity Image-to-Code Generationcited by 3
  12. 2026OmniDiagram: Advancing Unified Diagram Code Generation via Visual Interrogation Rewardcited by 3
  13. 2026MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judgecited by 3
  14. 2025Learning from Reasoning Failures via Synthetic Data Generationcited by 2
  15. 2025GRAFT: GRaPH and Table Reasoning for Textual Alignment -- A Benchmark for Structured Instruction Following and Visual Reasoningcited by 2
  16. 2025MindVL: Towards Efficient and Effective Training of Multimodal Large Language Models on Ascend NPUscited by 2
  17. 2025RECODE: Reasoning Through Code Generation for Visual Question Answeringcited by 2
  18. 2025BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languagescited by 2
  19. 2025DAVE: A VLM Vision Encoder for Document Understanding and Web Agentscited by 2
  20. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alonecited by 2
  21. 2025Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Trainingcited by 1
  22. 2025InteractScience: Programmatic and Visually-Grounded Evaluation of Interactive Scientific Demonstration Code Generationcited by 1
  23. 2026Typhoon OCR: Open Vision-Language Model For Thai Document Extractioncited by 1
  24. 2026Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancementcited by 1
  25. 2026Q-Mask: Query-driven Causal Masks for Text Anchoring in OCR-Oriented Vision-Language Modelscited by 1
  26. 2026Erase Persona, Forget Lore: Benchmarking Multimodal Copyright Unlearning in Large Vision Language Modelscited by 1
  27. 2026Program-as-Weights: A Programming Paradigm for Fuzzy Functionscited by 1
  28. 2026OpenVisTool: An Open Recipe for Synthesizing Instructive Visual Tool-Use Trajectoriescited by 1
  29. 2026UI-Venus-2 Technical Reportcited by 1
  30. 2025Rule-Based Reinforcement Learning for Document Image Classification with Vision Language Modelscited by 0
  31. 2025Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Taskscited by 0
  32. 2026Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Modelscited by 0
  33. 2026TableVision: A Large-Scale Benchmark for Spatially Grounded Reasoning over Complex Hierarchical Tablescited by 0
  34. 2026Multilingual Training and Evaluation Resources for Vision-Language Modelscited by 0
  35. 2026"Inctelegi Romanecste?'' A Recipe for Romanian Vision-Language Modelscited by 0
  36. 2026AMALIA-VL: A Native European Portuguese Open-Source Vision and Language Modelcited by 0
  37. 2026PhysBrain 1.5: From Vision-Language Models to Physical Foundation Modelscited by 0
  38. 2026SyntheticDoc: A Large Synthetic Dataset for Document Unwarping and Illumination Correctioncited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Text-Only Data Synthesis for Vision Language Model Training
  2. 2025Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models
  3. 2025MM-GEN: Enhancing Task Performance Through Targeted Multimodal Data Curation
  4. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generation
  5. 2024TRINS: Towards Multimodal Language Models that Can Read
  6. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  7. 2023Generating Images with Multimodal Language Models
  8. 2024Synth2: Boosting Visual-Language Models with Synthetic Captions and Image Embeddings