# Let the Chart Spark: Embedding Semantic Context into Chart with Text-to-Image Generative Model

arXiv: https://arxiv.org/abs/2304.14630 · HTML: https://mlsanity.com/paper/2304.14630/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 66 references that are in the map.
- First posted on arXiv 28 Apr 2023.

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://mlsanity.com/paper/2106.14843/) (2021)
- [StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators](https://mlsanity.com/paper/2108.00946/) (2021)
- [Highly Accurate Dichotomous Image Segmentation](https://mlsanity.com/paper/2203.03041/) (2022)
- [Word-As-Image for Semantic Typography](https://mlsanity.com/paper/2303.01818/) (2023)
- [Text-to-Viz: Automatic Generation of Infographics from Proportion-Related Natural Language Statements](https://mlsanity.com/paper/1907.09091/) (2019)
- [Towards Natural Language-Based Visualization Authoring](https://mlsanity.com/paper/2208.10947/) (2022)
- [Towards Automated Infographic Design: Deep Learning-based Auto-Extraction of Extensible Timeline](https://mlsanity.com/paper/1907.13550/) (2019)
- [Retrieve-Then-Adapt: Example-based Automatic Generation for Proportion-related Infographics](https://mlsanity.com/paper/2008.01177/) (2020)
- [Structure-aware Visualization Retrieval](https://mlsanity.com/paper/2202.05960/) (2022)
- [viz2viz: Prompt-driven stylized visualization generation using a diffusion model](https://mlsanity.com/paper/2304.01919/) (2023)

## Built on by (most cited first, 21 of 21)

- [Foundation Models Meet Visualizations: Challenges and Opportunities](https://mlsanity.com/paper/2310.05771/) (2023)
- [Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuning](https://mlsanity.com/paper/2407.20174/) (2024)
- [Generative AI for Visualization: State of the Art and Future Directions](https://mlsanity.com/paper/2404.18144/) (2024)
- [IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation](https://mlsanity.com/paper/2601.04498/) (2026)
- [InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions](https://mlsanity.com/paper/2503.04110/) (2025)
- [TypeDance: Creating Semantic Typographic Logos from Image through Personalized Generation](https://mlsanity.com/paper/2401.11094/) (2024)
- [Natural Language Generation for Visualizations: State of the Art, Challenges and Future Directions](https://mlsanity.com/paper/2409.19747/) (2024)
- [SketchFlex: Facilitating Spatial-Semantic Coherence in Text-to-Image Generation with Region-Based Sketches](https://mlsanity.com/paper/2502.07556/) (2025)
- [Evaluating LLMs for Visualization Generation and Understanding](https://mlsanity.com/paper/2507.22890/) (2025)
- [ShowTable: Unlocking Creative Table Visualization with Collaborative Reflection and Refinement](https://mlsanity.com/paper/2512.13303/) (2025)
- [LLM-Assisted Visual Analytics: Opportunities and Challenges](https://mlsanity.com/paper/2409.02691/) (2024)
- [Structuring the Unstructured: A Systematic Review of Text-to-Structure Generation for Agentic AI with a Universal Evaluation Framework](https://mlsanity.com/paper/2508.12257/) (2025)
- [ChArtist: Generating Pictorial Charts with Unified Spatial and Subject Control](https://mlsanity.com/paper/2603.14209/) (2026)
- [The Contemporary Art of Image Search: Iterative User Intent Expansion via Vision-Language Model](https://mlsanity.com/paper/2312.01656/) (2023)
- [Reflection on Data Storytelling Tools in the Generative AI Era from the Human-AI Collaboration Perspective](https://mlsanity.com/paper/2503.02631/) (2025)
- [DashChat: Interactive Authoring of Performance Dashboard Design Prototypes through Conversation with LLM-Powered Agent](https://mlsanity.com/paper/2504.12865/) (2025)
- [A Survey on Text-Driven 360-Degree Panorama Generation](https://mlsanity.com/paper/2502.14799/) (2025)
- [Data Therapist: Eliciting Domain Knowledge from Subject Matter Experts Using Large Language Models](https://mlsanity.com/paper/2505.00455/) (2025)
- [Evaluating LLMs for Visualization Tasks](https://mlsanity.com/paper/2506.10996/) (2025)
- [Semantic-Structural Alignment for Generative Pictorial Charts](https://mlsanity.com/paper/2606.06498/) (2026)
- [ArtChart: Faithful Artistic Chart Generation with Integrated Text Rendering](https://mlsanity.com/paper/2607.16060/) (2026)

## Abstract (arXiv)

Pictorial visualization seamlessly integrates data and semantic context into visual representation, conveying complex information in a manner that is both engaging and informative. Extensive studies have been devoted to developing authoring tools to simplify the creation of pictorial visualizations. However, mainstream works mostly follow a retrieving-and-editing pipeline that heavily relies on retrieved visual elements from a dedicated corpus, which often compromise the data integrity. Text-guided generation methods are emerging, but may have limited applicability due to its predefined recognized entities. In this work, we propose ChartSpark, a novel system that embeds semantic context into chart based on text-to-image generative model. ChartSpark generates pictorial visualizations conditioned on both semantic context conveyed in textual inputs and data information embedded in plain charts. The method is generic for both foreground and background pictorial generation, satisfying the design practices identified from an empirical research into existing pictorial visualizations. We further develop an interactive visual interface that integrates a text analyzer, editing module, and evaluation module to enable users to generate, modify, and assess pictorial visualizations. We experimentally demonstrate the usability of our tool, and conclude with a discussion of the potential of using text-to-image generative model combined with interactive interface for visualization design.