mlsanityData to 7 Oct 2026

DiSK: A Diffusion Model for Structured Knowledge

arXiv 2312.05253 ↗First posted 8 Dec 2023cs.LG cs.AIcited by 16

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Structured (dictionary-like) data presents challenges for left-to-right language models, as they can struggle with structured entities for a wide variety of reasons such as formatting and sensitivity to the order in which attributes are presented. Tabular generative models suffer from a different set of limitations such as their lack of flexibility. We introduce Diffusion Models of Structured Knowledge (DiSK) - a new architecture and training approach specialized for structured data. DiSK handles text, categorical, and continuous numerical data using a Gaussian mixture model approach, which allows for improved precision when dealing with numbers. It employs diffusion training to model relationships between properties. Experiments demonstrate DiSK's state-of-the-art performance on tabular data modeling, synthesis, and imputation on over 15 datasets across diverse domains. DiSK provides an effective inductive bias for generative modeling and manipulation of structured data. The techniques we propose could open the door to improved knowledge manipulation in future language models.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 23 of its 45 references that are in the map. First posted on arXiv 8 Dec 2023.

Builds on

23 of 45 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  5. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
  6. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  7. 2021Variational Diffusion Modelscited by 1,145
  8. 2019RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Spacecited by 893
  9. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  10. 2019Modeling Tabular data using Conditional GANcited by 695
  11. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  12. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
11 more
  1. 2015Deep Metric Learning via Lifted Structured Feature Embeddingcited by 449
  2. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  3. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  4. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  5. 2022TabDDPM: Modelling Tabular Data with Diffusion Modelscited by 373
  6. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  7. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  8. 2021CTAB-GAN: Effective Table Data Synthesizingcited by 156
  9. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 142
  10. 2019Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphscited by 125
  11. 2023CoDi: Co-evolving Contrastive Diffusion Models for Mixed-type Tabular Synthesiscited by 78

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 4.

16 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 872
  2. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  3. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  4. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  5. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmaskingcited by 128
  6. 2025Attention Is All You Need for KV Cache in Diffusion LLMscited by 32
  7. 2024Transformers Can Navigate Mazes With Multi-Step Predictioncited by 11
  8. 2025Anomaly Detection and Generation with Diffusion Models: A Surveycited by 8
  9. 2025Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuningcited by 8
  10. 2026Membership Inference Attacks Against Fine-tuned Diffusion Language Modelscited by 5
  11. 2024Unmasking Trees for Tabular Datacited by 4
  12. 2026Count Bridges enable Modeling and Deconvolving Transcriptomic Datacited by 3
4 more
  1. 2024Structured Object Language Modeling (SoLM): Native Structured Objects Generation Conforming to Complex Schemas with Self-Supervised Denoisingcited by 2
  2. 2026Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attentioncited by 1
  3. 2025Learning Distributions over Permutations and Rankings with Factorized Representationscited by 0
  4. 2026From Noise to Order: Learning to Rank via Denoising Diffusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Comprehensive Survey on Generative Diffusion Models for Structured Data
  2. 2026CDMD: A Cross-Dataset Mixed-Type Diffusion Model for Tabular Data
  3. 2024Improving Discrete Diffusion Models via Structured Preferential Generation
  4. 2024Diffusion-nested Auto-Regressive Synthesis of Heterogeneous Tabular Data
  5. 2026TabDLM: Free-Form Tabular Data Generation via Joint Numerical-Language Diffusion
  6. 2025Tabby: A Language Model Architecture for Tabular and Structured Data Synthesis
  7. 2025Discrete Diffusion Models for Language Generation
  8. 2025How Compositional Generalization and Creativity Improve as Diffusion Models are Trained