# DiSK: A Diffusion Model for Structured Knowledge

arXiv: https://arxiv.org/abs/2312.05253 · HTML: https://mlsanity.com/paper/2312.05253/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- Builds on 23 of its 45 references that are in the map.
- First posted on arXiv 8 Dec 2023.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [A Continuous Time Framework for Discrete Denoising Models](https://mlsanity.com/paper/2205.14987/) (2022)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [TabDDPM: Modelling Tabular Data with Diffusion Models](https://mlsanity.com/paper/2209.15421/) (2022)
- [Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer](https://mlsanity.com/paper/2203.03466/) (2022)
- [Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://mlsanity.com/paper/2309.14316/) (2023)
- [CTAB-GAN: Effective Table Data Synthesizing](https://mlsanity.com/paper/2102.08369/) (2021)
- [Physics of Language Models: Part 3.2, Knowledge Manipulation](https://mlsanity.com/paper/2309.14402/) (2023)
- [Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs](https://mlsanity.com/paper/1906.01195/) (2019)
- [CoDi: Co-evolving Contrastive Diffusion Models for Mixed-type Tabular Synthesis](https://mlsanity.com/paper/2304.12654/) (2023)

## Built on by (most cited first, 16 of 16)

- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [Simplified and Generalized Masked Diffusion for Discrete Data](https://mlsanity.com/paper/2406.04329/) (2024)
- [Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding](https://mlsanity.com/paper/2505.22618/) (2025)
- [LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning](https://mlsanity.com/paper/2505.16933/) (2025)
- [Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking](https://mlsanity.com/paper/2505.24857/) (2025)
- [Attention Is All You Need for KV Cache in Diffusion LLMs](https://mlsanity.com/paper/2510.14973/) (2025)
- [Transformers Can Navigate Mazes With Multi-Step Prediction](https://mlsanity.com/paper/2412.05117/) (2024)
- [Anomaly Detection and Generation with Diffusion Models: A Survey](https://mlsanity.com/paper/2506.09368/) (2025)
- [Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning](https://mlsanity.com/paper/2509.15188/) (2025)
- [Membership Inference Attacks Against Fine-tuned Diffusion Language Models](https://mlsanity.com/paper/2601.20125/) (2026)
- [Unmasking Trees for Tabular Data](https://mlsanity.com/paper/2407.05593/) (2024)
- [Count Bridges enable Modeling and Deconvolving Transcriptomic Data](https://mlsanity.com/paper/2603.04730/) (2026)
- [Structured Object Language Modeling (SoLM): Native Structured Objects Generation Conforming to Complex Schemas with Self-Supervised Denoising](https://mlsanity.com/paper/2411.19301/) (2024)
- [Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention](https://mlsanity.com/paper/2605.19726/) (2026)
- [Learning Distributions over Permutations and Rankings with Factorized Representations](https://mlsanity.com/paper/2505.24664/) (2025)
- [From Noise to Order: Learning to Rank via Denoising Diffusion](https://mlsanity.com/paper/2602.11453/) (2026)

## Abstract (arXiv)

Structured (dictionary-like) data presents challenges for left-to-right language models, as they can struggle with structured entities for a wide variety of reasons such as formatting and sensitivity to the order in which attributes are presented. Tabular generative models suffer from a different set of limitations such as their lack of flexibility. We introduce Diffusion Models of Structured Knowledge (DiSK) - a new architecture and training approach specialized for structured data. DiSK handles text, categorical, and continuous numerical data using a Gaussian mixture model approach, which allows for improved precision when dealing with numbers. It employs diffusion training to model relationships between properties. Experiments demonstrate DiSK's state-of-the-art performance on tabular data modeling, synthesis, and imputation on over 15 datasets across diverse domains. DiSK provides an effective inductive bias for generative modeling and manipulation of structured data. The techniques we propose could open the door to improved knowledge manipulation in future language models.