# Deep Generative Models with Learnable Knowledge Constraints

arXiv: https://arxiv.org/abs/1806.09764 · HTML: https://mlsanity.com/paper/1806.09764/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jun 2018.

## Built on by (most cited first, 23 of 23)

- [COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics](https://mlsanity.com/paper/2202.11705/) (2022)
- [Composable Text Controls in Latent Space with ODEs](https://mlsanity.com/paper/2208.00638/) (2022)
- [Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models](https://mlsanity.com/paper/2209.06970/) (2022)
- [A Pseudo-Semantic Loss for Autoregressive Models with Logical Constraints](https://mlsanity.com/paper/2312.03905/) (2023)
- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)
- [Toward a `Standard Model' of Machine Learning](https://mlsanity.com/paper/2108.07783/) (2021)
- [Unsupervised Editing for Counterfactual Stories](https://mlsanity.com/paper/2112.05417/) (2021)
- [RLSS: A Deep Reinforcement Learning Algorithm for Sequential Scene Generation](https://mlsanity.com/paper/2206.02544/) (2022)
- [Deep Generative Modeling on Limited Data with Regularization by Nontransferable Pre-trained Models](https://mlsanity.com/paper/2208.14133/) (2022)
- [Harnessing Knowledge and Reasoning for Human-Like Natural Language Generation: A Brief Review](https://mlsanity.com/paper/2212.03747/) (2022)
- [Logic Tensor Network-Enhanced Generative Adversarial Network](https://mlsanity.com/paper/2601.03839/) (2026)
- [Distantly Supervised Semantic Text Detection and Recognition for Broadcast Sports Videos Understanding](https://mlsanity.com/paper/2111.00629/) (2021)
- [Training Deep Normalizing Flow Models in Highly Incomplete Data Scenarios with Prior Regularization](https://mlsanity.com/paper/2104.01482/) (2021)
- [fAshIon after fashion: A Report of AI in Fashion](https://mlsanity.com/paper/2105.03050/) (2021)
- [Machine Learning for Combinatorial Optimisation of Partially-Specified Problems: Regret Minimisation as a Unifying Lens](https://mlsanity.com/paper/2205.10157/) (2022)
- [Physically-admissible polarimetric data augmentation for road-scene analysis](https://mlsanity.com/paper/2206.07431/) (2022)
- [Bridging the Intent Gap: Knowledge-Enhanced Visual Generation](https://mlsanity.com/paper/2405.12538/) (2024)
- [A Roadmap to Domain Knowledge Integration in Machine Learning](https://mlsanity.com/paper/2212.05712/) (2022)
- [Human-AI Co-Creation Approach to Find Forever Chemicals Replacements](https://mlsanity.com/paper/2304.05389/) (2023)
- [All by Myself: Learning Individualized Competitive Behaviour with a Contrastive Reinforcement Learning optimization](https://mlsanity.com/paper/2310.00964/) (2023)
- [How to Train Text Summarization Model with Weak Supervisions](https://mlsanity.com/paper/2409.00098/) (2024)
- [LatentMimic: Terrain-Adaptive Locomotion via Latent Space Imitation](https://mlsanity.com/paper/2604.16440/) (2026)
- [Generate in the Chart, Not on the Boundary: Function-Symbol Grounding for Hard Constraints in LTN-GANs](https://mlsanity.com/paper/2608.21605/) (2026)

## Abstract (arXiv)

The broad set of deep generative models (DGMs) has achieved remarkable advances. However, it is often difficult to incorporate rich structured domain knowledge with the end-to-end DGMs. Posterior regularization (PR) offers a principled framework to impose structured constraints on probabilistic models, but has limited applicability to the diverse DGMs that can lack a Bayesian formulation or even explicit density evaluation. PR also requires constraints to be fully specified a priori, which is impractical or suboptimal for complex knowledge with learnable uncertain parts. In this paper, we establish mathematical correspondence between PR and reinforcement learning (RL), and, based on the connection, expand PR to learn constraints as the extrinsic reward in RL. The resulting algorithm is model-agnostic to apply to any DGMs, and is flexible to adapt arbitrary constraints with the model jointly. Experiments on human image generation and templated sentence generation show models with learned knowledge constraints by our algorithm greatly improve over base generative models.