# Self-supervised self-supervision by combining deep learning and probabilistic logic

arXiv: https://arxiv.org/abs/2012.12474 · HTML: https://mlsanity.com/paper/2012.12474/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2020.

## Built on by (most cited first, 7 of 7)

- [Theoretical Analysis of Weak-to-Strong Generalization](https://mlsanity.com/paper/2405.16043/) (2024)
- [Pareto Optimal Learning for Estimating Large Language Model Errors](https://mlsanity.com/paper/2306.16564/) (2023)
- [Towards Structuring Real-World Data at Scale: Deep Learning for Extracting Key Oncology Information from Clinical Text with Patient-Level Supervision](https://mlsanity.com/paper/2203.10442/) (2022)
- [Neuro-Symbolic Recommendation Model based on Logic Query](https://mlsanity.com/paper/2309.07594/) (2023)
- [Combining Probabilistic Logic and Deep Learning for Self-Supervised Learning](https://mlsanity.com/paper/2107.12591/) (2021)
- [Generative Modeling Helps Weak Supervision (and Vice Versa)](https://mlsanity.com/paper/2203.12023/) (2022)
- [Aiding Humans in Financial Fraud Decision Making: Toward an XAI-Visualization Framework](https://mlsanity.com/paper/2408.14552/) (2024)

## Abstract (arXiv)

Labeling training examples at scale is a perennial challenge in machine learning. Self-supervision methods compensate for the lack of direct supervision by leveraging prior knowledge to automatically generate noisy labeled examples. Deep probabilistic logic (DPL) is a unifying framework for self-supervised learning that represents unknown labels as latent variables and incorporates diverse self-supervision using probabilistic logic to train a deep neural network end-to-end using variational EM. While DPL is successful at combining pre-specified self-supervision, manually crafting self-supervision to attain high accuracy may still be tedious and challenging. In this paper, we propose Self-Supervised Self-Supervision (S4), which adds to DPL the capability to learn new self-supervision automatically. Starting from an initial "seed," S4 iteratively uses the deep neural network to propose new self supervision. These are either added directly (a form of structured self-training) or verified by a human expert (as in feature-based active learning). Experiments show that S4 is able to automatically propose accurate self-supervision and can often nearly match the accuracy of supervised methods with a tiny fraction of the human effort.