# SciDoc2Diagrammer-MAF: Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinement

arXiv: https://arxiv.org/abs/2409.19242 · HTML: https://mlsanity.com/paper/2409.19242/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 63 references that are in the map.
- First posted on arXiv 28 Sep 2024.

## Builds on (most cited first, 30 of 38)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders](https://mlsanity.com/paper/1703.10960/) (2017)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [DePlot: One-shot visual language reasoning by plot-to-table translation](https://mlsanity.com/paper/2212.10505/) (2022)
- [UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2305.14761/) (2023)
- [Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences](https://mlsanity.com/paper/2404.12272/) (2024)
- [MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering](https://mlsanity.com/paper/2212.09662/) (2022)
- [ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation](https://mlsanity.com/paper/2406.09961/) (2024)
- [GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks](https://mlsanity.com/paper/2311.01361/) (2023)
- [Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision](https://mlsanity.com/paper/2311.07362/) (2023)
- [AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ](https://mlsanity.com/paper/2310.00367/) (2023)
- [SciCap: Generating Captions for Scientific Figures](https://mlsanity.com/paper/2110.11624/) (2021)
- [DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents](https://mlsanity.com/paper/2101.11796/) (2021)
- [LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts](https://mlsanity.com/paper/2310.10640/) (2023)
- [D2S: Document-to-Slide Generation Via Query-Based Text Summarization](https://mlsanity.com/paper/2105.03664/) (2021)
- [Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problems](https://mlsanity.com/paper/1811.00720/) (2018)
- [Reasoning with Sarcasm by Reading In-between](https://mlsanity.com/paper/1805.02856/) (2018)
- [PEDANTS: Cheap but Effective and Interpretable Answer Equivalence](https://mlsanity.com/paper/2402.11161/) (2024)
- [DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning](https://mlsanity.com/paper/2310.12128/) (2023)

## Built on by (most cited first, 15 of 15)

- [A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges](https://mlsanity.com/paper/2501.02189/) (2025)
- [Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation](https://mlsanity.com/paper/2502.05151/) (2025)
- [SlideTailor: Personalized Presentation Slide Generation for Scientific Papers](https://mlsanity.com/paper/2512.20292/) (2025)
- [DiagramEval: Evaluating LLM-Generated Diagrams via Graphs](https://mlsanity.com/paper/2510.25761/) (2025)
- [SciFig: Towards Automating Editable Figure Generation for Scientific Papers](https://mlsanity.com/paper/2601.04390/) (2026)
- [Paper2SysArch: Structure-Constrained System Architecture Generation from Scientific Papers](https://mlsanity.com/paper/2511.18036/) (2025)
- [SciFigDetect: A Benchmark for AI-Generated Scientific Figure Detection](https://mlsanity.com/paper/2604.08211/) (2026)
- [Large Language Models Struggle to Describe the Haystack without Human Help: Human-in-the-loop Evaluation of Topic Models](https://mlsanity.com/paper/2502.14748/) (2025)
- [TechImage-Bench: Rubric-Based Evaluation for Technical Image Generation](https://mlsanity.com/paper/2512.12220/) (2025)
- [Exploring Agentic Workflows for Generating High Quality Math Visual Aids](https://mlsanity.com/paper/2607.09839/) (2026)
- [SciForma: Structure-Faithful Generation of Scientific Diagrams](https://mlsanity.com/paper/2607.18091/) (2026)
- [EdiTikZ: Scientific Figure Editing from Revision Trajectories](https://mlsanity.com/paper/2609.01409/) (2026)
- [When Looks Do Not Lie: Discourse Structure Guided In-Context Learning for Faithful Diagram Generation](https://mlsanity.com/paper/2601.20476/) (2026)
- [DiagramBank: A Quality-Audited Dataset of Scientific Schematic Diagrams with Multi-Level Document Context](https://mlsanity.com/paper/2604.20857/) (2026)
- [InfoEdit: Probing Global Layout Reasoning in Infographic Editing](https://mlsanity.com/paper/2609.33286/) (2026)

## Abstract (arXiv)

Automating the creation of scientific diagrams from academic papers can significantly streamline the development of tutorials, presentations, and posters, thereby saving time and accelerating the process. Current text-to-image models struggle with generating accurate and visually appealing diagrams from long-context inputs. We propose SciDoc2Diagram, a task that extracts relevant information from scientific papers and generates diagrams, along with a benchmarking dataset, SciDoc2DiagramBench. We develop a multi-step pipeline SciDoc2Diagrammer that generates diagrams based on user intentions using intermediate code generation. We observed that initial diagram drafts were often incomplete or unfaithful to the source, leading us to develop SciDoc2Diagrammer-Multi-Aspect-Feedback (MAF), a refinement strategy that significantly enhances factual correctness and visual appeal and outperforms existing models on both automatic and human judgement.