mlsanityData to 13 Oct 2026

AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ

arXiv 2310.00367 ↗First posted 30 Sep 2023cs.CL cs.CVcited by 76

Not on the map. Why?

Abstract (arXiv)

Generating bitmap graphics from text has gained considerable attention, yet for scientific figures, vector graphics are often preferred. Given that vector graphics are typically encoded using low-level graphics primitives, generating them directly is difficult. To address this, we propose the use of TikZ, a well-known abstract graphics language that can be compiled to vector graphics, as an intermediate representation of scientific figures. TikZ offers human-oriented, high-level commands, thereby facilitating conditional language modeling with any large language model. To this end, we introduce DaTikZ, the first large-scale TikZ dataset consisting of 120k TikZ drawings aligned with captions. We fine-tune LLaMA on DaTikZ, as well as our new model CLiMA, which augments LLaMA with multimodal CLIP embeddings. In both human and automatic evaluation, CLiMA and LLaMA outperform commercial GPT-4 and Claude 2 in terms of similarity to human-created figures, with CLiMA additionally improving text-image alignment. Our detailed analysis shows that all models generalize well and are not susceptible to memorization. GPT-4 and Claude 2, however, tend to generate more simplistic figures compared to both humans and our models. We make our framework, AutomaTikZ, along with model weights and datasets, publicly available.

Key facts. Cited by 76 papers in the map (data through 13 Oct 2026). Builds on 79 of its 111 references that are in the map. First posted on arXiv 30 Sep 2023.

Builds on

79 of 111 references.

References · 79

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  9. 2023Visual Instruction Tuningcited by 7,827
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,564
  11. 2020Scaling Laws for Neural Language Modelscited by 6,494
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,339
38 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,484
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,314
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,284
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,785
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,388
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,237
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,864
  8. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,619
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,586
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,407
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,130
  12. 2017Decoupled Weight Decay Regularizationcited by 1,788
  13. 2022Competition-Level Code Generation with AlphaCodecited by 1,445
  14. 2018Demystifying MMD GANscited by 1,277
  15. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  16. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,164
  17. 2017Mixed Precision Trainingcited by 1,147
  18. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,120
  19. 2023A Survey on Multimodal Large Language Modelscited by 905
  20. 2023StarCoder: may the source be with you!cited by 797
  21. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  22. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 744
  23. 2016Generative Adversarial Text to Image Synthesiscited by 735
  24. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 692
  25. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 641
  26. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 577
  27. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 497
  28. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  29. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  30. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 408
  31. 2022A Systematic Evaluation of Large Language Models of Codecited by 362
  32. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 339
  33. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 334
  34. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 316
  35. 2017A Neural Representation of Sketch Drawingscited by 311
  36. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  37. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 273
  38. 2022Efficient Training of Language Models to Fill in the Middlecited by 251

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 3; 2024: 13; 2025: 28; 2026: 32.

76 papers.

Sort:

By area:

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 827
  2. 2024What matters when building vision-language models?cited by 323
  3. 2024Building and better understanding vision-language models: insights and future directionscited by 172
  4. 2023StarVector: Generating Scalable Vector Graphics Code from Images and Textcited by 96
  5. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 88
  6. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 76
  7. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 76
  8. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  9. 2025AutoPresent: Designing Structured Visuals from Scratchcited by 65
  10. 2025FineVision: Open Data Is All You Needcited by 53
  11. 2024MMEvol: Empowering Multimodal Large Language Models with Evol-Instructcited by 42
  12. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 41
38 more
  1. 2024Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Modelscited by 37
  2. 2023DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planningcited by 29
  3. 2024What Can Natural Language Processing Do for Peer Review?cited by 24
  4. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  5. 2026PaperBanana: Automating Academic Illustration for AI Scientistscited by 23
  6. 2025TikZero: Zero-Shot Text-Guided Graphics Program Synthesiscited by 22
  7. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 22
  8. 2026AutoFigure: Generating and Refining Publication-Ready Scientific Illustrationscited by 21
  9. 2025Factuality Matters: When Image Generation and Editing Meet Structured Visualscited by 18
  10. 2024SciDoc2Diagrammer-MAF: Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinementcited by 15
  11. 2025SVGen: Interpretable Vector Graphics Generation with Large Language Modelscited by 14
  12. 2024ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?cited by 13
  13. 2026AutoFigure-Edit: Generating Editable Scientific Illustrationcited by 13
  14. 2024OmChat: A Recipe to Train Multimodal Language Models with Strong Long Context and Video Understandingcited by 10
  15. 2025Talk to Your Slides: High-Efficiency Slide Editing via Language-Driven Structured Data Manipulationcited by 10
  16. 2025A Survey of AI Scientistscited by 10
  17. 2025MLLM-Selector: Necessity and Diversity-driven High-Value Data Selection for Enhanced Visual Instruction Tuningcited by 9
  18. 2026Crafter: A Multi-Agent Harness for Editable Scientific Figure Generation from Diverse Inputscited by 9
  19. 2025DiagramEval: Evaluating LLM-Generated Diagrams via Graphscited by 8
  20. 2025LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sightcited by 8
  21. 2026Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utilitycited by 8
  22. 2026SciFig: Towards Automating Editable Figure Generation for Scientific Paperscited by 7
  23. 2026Scientific Graphics Program Synthesis via Dual Self-Consistency Reinforcement Learningcited by 6
  24. 2025From Text to Visuals: Using LLMs to Generate Math Diagrams with Vector Graphicscited by 5
  25. 2025LLM Code Customization with Visual Results: A Benchmark on TikZcited by 5
  26. 2025MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generationcited by 5
  27. 2026Automatic Method Illustration Generation for AI Scientific Papers via Drawing Middleware Creation, Evolution, and Orchestrationcited by 5
  28. 2026VectorGym: A Multi-Task Benchmark for SVG Code Generation, Sketching and Editingcited by 5
  29. 2026Beyond NL2Code: A Structured Survey of Multimodal Code Intelligencecited by 5
  30. 2024TurtleBench: A Visual Programming Benchmark in Turtle Geometrycited by 4
  31. 2025Generating Pedagogically Meaningful Visuals for Math Word Problems: A New Benchmark and Analysis of Text-to-Image Modelscited by 4
  32. 2025From Pixels to Paths: A Multi-Agent Framework for Editable Scientific Illustrationcited by 4
  33. 2025SciPostGen: Bridging the Gap between Scientific Papers and Poster Layoutscited by 4
  34. 2026TikZilla: Scaling Text-to-TikZ with High-Quality Data and Reinforcement Learningcited by 4
  35. 2026PairCoder++: Pair Programming as a Universal Paradigm for Verified Code-Driven Multimodal and Structured-Artifact Generationcited by 4
  36. 2025Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understandingcited by 3
  37. 2025MindVL: Towards Efficient and Effective Training of Multimodal Large Language Models on Ascend NPUscited by 3
  38. 2026PaperX: A Unified Framework for Multimodal Academic Presentation Generation with Scholar DAGcited by 3

50 most cited of 76 All 76 (JSON)