mlsanityData to 14 Oct 2026

SciDoc2Diagrammer-MAF: Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinement

arXiv 2409.19242 ↗First posted 28 Sep 2024cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

Automating the creation of scientific diagrams from academic papers can significantly streamline the development of tutorials, presentations, and posters, thereby saving time and accelerating the process. Current text-to-image models struggle with generating accurate and visually appealing diagrams from long-context inputs. We propose SciDoc2Diagram, a task that extracts relevant information from scientific papers and generates diagrams, along with a benchmarking dataset, SciDoc2DiagramBench. We develop a multi-step pipeline SciDoc2Diagrammer that generates diagrams based on user intentions using intermediate code generation. We observed that initial diagram drafts were often incomplete or unfaithful to the source, leading us to develop SciDoc2Diagrammer-Multi-Aspect-Feedback (MAF), a refinement strategy that significantly enhances factual correctness and visual appeal and outperforms existing models on both automatic and human judgement.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 38 of its 63 references that are in the map. First posted on arXiv 28 Sep 2024.

Builds on

38 of 63 references.

References · 38

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  8. 2015VQA: Visual Question Answeringcited by 2,506
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  12. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
26 more
  1. 2017Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoderscited by 222
  2. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 169
  3. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 160
  4. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 160
  5. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 144
  6. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 133
  7. 2024ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generationcited by 129
  8. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 119
  9. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revisioncited by 98
  10. 2023AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZcited by 79
  11. 2021SciCap: Generating Captions for Scientific Figurescited by 77
  12. 2021DOC2PPT: Automatic Presentation Slides Generation from Scientific Documentscited by 70
  13. 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Promptscited by 54
  14. 2021D2S: Document-to-Slide Generation Via Query-Based Text Summarizationcited by 50
  15. 2018Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problemscited by 49
  16. 2018Reasoning with Sarcasm by Reading In-betweencited by 38
  17. 2024PEDANTS: Cheap but Effective and Interpretable Answer Equivalencecited by 31
  18. 2023DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planningcited by 29
  19. 2024Guiding Large Language Models to Post-Edit Machine Translation with Error Annotationscited by 22
  20. 2021End-to-End NLP Knowledge Graph Constructioncited by 21
  21. 2023LIQUID: A Framework for List Question Answering Dataset Generationcited by 19
  22. 2024Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysiscited by 10
  23. 2018TutorialBank: A Manually-Collected Corpus for Prerequisite Chains, Survey Extraction and Resource Recommendationcited by 9
  24. 2024Assessing the Aesthetic Evaluation Capabilities of GPT-4 with Vision: Insights from Group and Individual Assessmentscited by 4
  25. 2018QA4IE: A Question Answering based Framework for Information Extractioncited by 3
  26. 2017MORSE: Semantic-ally Drive-n MORpheme SEgment-ercited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 8.

15 papers.

Sort:

By area:

  1. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 47
  2. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 42
  3. 2025SlideTailor: Personalized Presentation Slide Generation for Scientific Paperscited by 13
  4. 2025DiagramEval: Evaluating LLM-Generated Diagrams via Graphscited by 8
  5. 2026SciFig: Towards Automating Editable Figure Generation for Scientific Paperscited by 8
  6. 2025Paper2SysArch: Structure-Constrained System Architecture Generation from Scientific Paperscited by 7
  7. 2026SciFigDetect: A Benchmark for AI-Generated Scientific Figure Detectioncited by 4
  8. 2025Large Language Models Struggle to Describe the Haystack without Human Help: Human-in-the-loop Evaluation of Topic Modelscited by 2
  9. 2025TechImage-Bench: Rubric-Based Evaluation for Technical Image Generationcited by 2
  10. 2026Exploring Agentic Workflows for Generating High Quality Math Visual Aidscited by 1
  11. 2026SciForma: Structure-Faithful Generation of Scientific Diagramscited by 1
  12. 2026EdiTikZ: Scientific Figure Editing from Revision Trajectoriescited by 1
3 more
  1. 2026When Looks Do Not Lie: Discourse Structure Guided In-Context Learning for Faithful Diagram Generationcited by 0
  2. 2026DiagramBank: A Quality-Audited Dataset of Scientific Schematic Diagrams with Multi-Level Document Contextcited by 0
  3. 2026InfoEdit: Probing Global Layout Reasoning in Infographic Editingcited by 0