mlsanityData to 7 Oct 2026

Markup-to-Image Diffusion Models with Scheduled Sampling

arXiv 2210.05147 ↗First posted 11 Oct 2022cs.LG cs.CL cs.CVcited by 6

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Building on recent advances in image generation, we present a fully data-driven approach to rendering markup into images. The approach is based on diffusion models, which parameterize the distribution of data using a sequence of denoising operations on top of a Gaussian noise distribution. We view the diffusion denoising process as a sequential decision making process, and show that it exhibits compounding errors similar to exposure bias issues in imitation learning problems. To mitigate these issues, we adapt the scheduled sampling algorithm to diffusion training. We conduct experiments on four markup datasets: mathematical formulas (LaTeX), table layouts (HTML), sheet music (LilyPond), and molecular images (SMILES). These experiments each verify the effectiveness of the diffusion process and the use of scheduled sampling to fix generation issues. These results also show that the markup-to-image task presents a useful controlled compositional setting for diagnosing and analyzing generative image models.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 40 of its 61 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

40 of 61 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2013Auto-Encoding Variational Bayescited by 3,980
  12. 2016Improved Techniques for Training GANscited by 2,938
25 more
  1. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  4. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  5. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  6. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  7. 2016Generative Adversarial Text to Image Synthesiscited by 735
  8. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  9. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  10. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  11. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  12. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  13. 2017Non-Autoregressive Neural Machine Translationcited by 457
  14. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 332
  15. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  16. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  17. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  18. 2020iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenescited by 151
  19. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  20. 2016Learning What and Where to Drawcited by 105
  21. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  22. 2021Learning to Compose Visual Relationscited by 72
  23. 2021Feedback in Imitation Learning: The Three Regimes of Covariate Shiftcited by 68
  24. 2019Generating Multiple Objects at Spatially Distinct Locationscited by 41
  25. 2020Text-to-Image Generation Grounded by Fine-Grained User Attentioncited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 2; 2026: 3.

6 papers.

  1. 2025NeuralOS: Towards Simulating Operating Systems via Neural Generative Modelscited by 17
  2. 2026Exposure Bias Can Alleviate Itself via Directional and Frequency Rectification in Flow Matchingcited by 4
  3. 2026Program-as-Weights: A Programming Paradigm for Fuzzy Functionscited by 1
  4. 2023Contrast-augmented Diffusion Model with Fine-grained Sequence Alignment for Markup-to-Image Generationcited by 0
  5. 2025ReflexFlow: Rethinking Learning Objective for Exposure Bias Alleviation in Flow Matchingcited by 0
  6. 2026Let It Go or Learn to Self-Correct: Continuous Diffusion for Constrained Discrete Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Guided Image Synthesis via Initial Image Editing in Diffusion Model
  2. 2024Diffusion Model-Based Image Editing: A Survey
  3. 2024Conditional Image Synthesis with Diffusion Models: A Survey
  4. 2023Nested Diffusion Processes for Anytime Image Generation
  5. 2024Controllable Generation with Text-to-Image Diffusion Models: A Survey
  6. 2023Diffusion Brush: A Latent Diffusion Model-based Editing Tool for AI-generated Images
  7. 2024Diffusion Models Need Visual Priors for Image Generation
  8. 2022On Conditioning the Input Noise for Controlled Image Generation with Diffusion Models