# Combining Probabilistic Logic and Deep Learning for Self-Supervised Learning

arXiv: https://arxiv.org/abs/2107.12591 · HTML: https://mlsanity.com/paper/2107.12591/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 80 references that are in the map.
- First posted on arXiv 27 Jul 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Neural Probabilistic Logic Programming in DeepProbLog](https://mlsanity.com/paper/1907.08194/) (2019)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [End-to-End Differentiable Proving](https://mlsanity.com/paper/1705.11040/) (2017)
- [Harnessing Deep Neural Networks with Logic Rules](https://mlsanity.com/paper/1603.06318/) (2016)
- [Composing graphical models with neural networks for structured representations and fast inference](https://mlsanity.com/paper/1603.06277/) (2016)
- [Training Complex Models with Multi-Task Weak Supervision](https://mlsanity.com/paper/1810.02840/) (2018)
- [Cross-Sentence N-ary Relation Extraction with Graph LSTMs](https://mlsanity.com/paper/1708.03743/) (2017)
- [Loopy Belief Propagation for Approximate Inference: An Empirical Study](https://mlsanity.com/paper/1301.6725/) (2013)
- [Probabilistic Logic Neural Networks for Reasoning](https://mlsanity.com/paper/1906.08495/) (2019)
- [Distant Supervision for Relation Extraction beyond the Sentence Boundary](https://mlsanity.com/paper/1609.04873/) (2016)
- [Learning the Structure of Generative Models without Labeled Data](https://mlsanity.com/paper/1703.00854/) (2017)
- [Deep Probabilistic Programming](https://mlsanity.com/paper/1701.03757/) (2017)
- [Inferring Generative Model Structure with Static Analysis](https://mlsanity.com/paper/1709.02477/) (2017)
- [Adaptive Rule Discovery for Labeling Text Data](https://mlsanity.com/paper/2005.06133/) (2020)
- [Leveraging Deep Neural Networks and Knowledge Graphs for Entity Disambiguation](https://mlsanity.com/paper/1504.07678/) (2015)
- [Incidental Supervision: Moving beyond Supervised Learning](https://mlsanity.com/paper/2005.12339/) (2020)
- [Learning Graphical Model Parameters with Approximate Marginal Inference](https://mlsanity.com/paper/1301.3193/) (2013)
- [Heterogeneous Supervision for Relation Extraction: A Representation Learning Approach](https://mlsanity.com/paper/1707.00166/) (2017)
- [Identifying civilians killed by police with distantly supervised entity-event extraction](https://mlsanity.com/paper/1707.07086/) (2017)
- [Deep Probabilistic Logic: A Unifying Framework for Indirect Supervision](https://mlsanity.com/paper/1808.08485/) (2018)
- [Self-supervised self-supervision by combining deep learning and probabilistic logic](https://mlsanity.com/paper/2012.12474/) (2020)
- [EZLearn: Exploiting Organic Supervision in Large-Scale Data Annotation](https://mlsanity.com/paper/1709.08600/) (2017)

## Abstract (arXiv)

Deep learning has proven effective for various application tasks, but its applicability is limited by the reliance on annotated examples. Self-supervised learning has emerged as a promising direction to alleviate the supervision bottleneck, but existing work focuses on leveraging co-occurrences in unlabeled data for task-agnostic representation learning, as exemplified by masked language model pretraining. In this chapter, we explore task-specific self-supervision, which leverages domain knowledge to automatically annotate noisy training examples for end applications, either by introducing labeling functions for annotating individual instances, or by imposing constraints over interdependent label decisions. We first present deep probabilistic logic(DPL), which offers a unifying framework for task-specific self-supervision by composing probabilistic logic with deep learning. DPL represents unknown labels as latent variables and incorporates diverse self-supervision using probabilistic logic to train a deep neural network end-to-end using variational EM. Next, we present self-supervised self-supervision(S4), which adds to DPL the capability to learn new self-supervision automatically. Starting from an initial seed self-supervision, S4 iteratively uses the deep neural network to propose new self supervision. These are either added directly (a form of structured self-training) or verified by a human expert (as in feature-based active learning). Experiments on real-world applications such as biomedical machine reading and various text classification tasks show that task-specific self-supervision can effectively leverage domain expertise and often match the accuracy of supervised methods with a tiny fraction of human effort.