# Learning the Structure of Generative Models without Labeled Data

arXiv: https://arxiv.org/abs/1703.00854 · HTML: https://mlsanity.com/paper/1703.00854/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2017.

## Built on by (most cited first, 30 of 50)

- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [A Survey on Programmatic Weak Supervision](https://mlsanity.com/paper/2202.05433/) (2022)
- [Language Models in the Loop: Incorporating Prompting into Weak Supervision](https://mlsanity.com/paper/2205.02318/) (2022)
- [End-to-End Weak Supervision](https://mlsanity.com/paper/2107.02233/) (2021)
- [Weakly Supervised Named Entity Tagging with Learnable Logical Rules](https://mlsanity.com/paper/2107.02282/) (2021)
- [DP-SSL: Towards Robust Semi-supervised Learning with A Few Labeled Samples](https://mlsanity.com/paper/2110.13740/) (2021)
- [Learning from Multiple Noisy Partial Labelers](https://mlsanity.com/paper/2106.04530/) (2021)
- [Creating Training Sets via Weak Indirect Supervision](https://mlsanity.com/paper/2110.03484/) (2021)
- [GLaRA: Graph-based Labeling Rule Augmentation for Weakly Supervised Named Entity Recognition](https://mlsanity.com/paper/2104.06230/) (2021)
- [Trustable Co-label Learning from Multiple Noisy Annotators](https://mlsanity.com/paper/2203.04199/) (2022)
- [Jointly Improving Language Understanding and Generation with Quality-Weighted Weak Supervision of Automatic Labeling](https://mlsanity.com/paper/2102.03551/) (2021)
- [Losses over Labels: Weakly Supervised Learning via Direct Loss Construction](https://mlsanity.com/paper/2212.06921/) (2022)
- [Transferring Annotator- and Instance-dependent Transition Matrix for Learning from Crowds](https://mlsanity.com/paper/2306.03116/) (2023)
- [How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Ranking](https://mlsanity.com/paper/2312.01619/) (2023)
- [Comparing the Value of Labeled and Unlabeled Data in Method-of-Moments Latent Variable Estimation](https://mlsanity.com/paper/2103.02761/) (2021)
- [Knodle: Modular Weakly Supervised Learning with PyTorch](https://mlsanity.com/paper/2104.11557/) (2021)
- [Dependency Structure Misspecification in Multi-Source Weak Supervision Models](https://mlsanity.com/paper/2106.10302/) (2021)
- [Leveraging Instance Features for Label Aggregation in Programmatic Weak Supervision](https://mlsanity.com/paper/2210.02724/) (2022)
- [WeCheck: Strong Factual Consistency Checker via Weakly Supervised Learning](https://mlsanity.com/paper/2212.10057/) (2022)
- [Generalizing Trust: Weak-to-Strong Trustworthiness in Language Models](https://mlsanity.com/paper/2501.00418/) (2024)
- [PInKS: Preconditioned Commonsense Inference with Minimal Supervision](https://mlsanity.com/paper/2206.07920/) (2022)
- [Analyzing the impact of climate change on critical infrastructure from the scientific literature: A weakly supervised NLP approach](https://mlsanity.com/paper/2302.01887/) (2023)
- [Making Binary Classification from Multiple Unlabeled Datasets Almost Free of Supervision](https://mlsanity.com/paper/2306.07036/) (2023)
- [Zero-to-Strong Generalization: Eliciting Strong Capabilities of Large Language Models Iteratively without Gold Labels](https://mlsanity.com/paper/2409.12425/) (2024)
- [Evaluating multiple models using labeled and unlabeled data](https://mlsanity.com/paper/2501.11866/) (2025)
- [Big Machinery Data Preprocessing Methodology for Data-Driven Models in Prognostics and Health Management](https://mlsanity.com/paper/2110.04256/) (2021)
- [WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue](https://mlsanity.com/paper/2108.01487/) (2021)
- [Addressing Bias Through Ensemble Learning and Regularized Fine-Tuning](https://mlsanity.com/paper/2402.00910/) (2024)
- [A data-centric weak supervised learning for highway traffic incident detection](https://mlsanity.com/paper/2112.09792/) (2021)

## Abstract (arXiv)

Curating labeled training data has become the primary bottleneck in machine learning. Recent frameworks address this bottleneck with generative models to synthesize labels at scale from weak supervision sources. The generative model's dependency structure directly affects the quality of the estimated labels, but selecting a structure automatically without any labeled data is a distinct challenge. We propose a structure estimation method that maximizes the ℓ1-regularized marginal pseudolikelihood of the observed data. Our analysis shows that the amount of unlabeled data required to identify the true structure scales sublinearly in the number of possible dependencies for a broad class of models. Simulations show that our method is 100× faster than a maximum likelihood approach and selects 1/4 as many extraneous dependencies. We also show that our method provides an average of 1.5 F1 points of improvement over existing, user-developed information extraction applications on real-world data such as PubMed journal abstracts.