# Functional Regularization for Representation Learning: A Unified Theoretical Perspective

arXiv: https://arxiv.org/abs/2008.02447 · HTML: https://mlsanity.com/paper/2008.02447/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Aug 2020.

## Built on by (most cited first, 18 of 18)

- [Pre-training Molecular Graph Representation with 3D Geometry](https://mlsanity.com/paper/2110.07728/) (2021)
- [Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matching](https://mlsanity.com/paper/2206.13602/) (2022)
- [Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning](https://mlsanity.com/paper/2102.06866/) (2021)
- [Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning](https://mlsanity.com/paper/2402.15017/) (2024)
- [Sample Efficiency of Data Augmentation Consistency Regularization](https://mlsanity.com/paper/2202.12230/) (2022)
- [The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learning](https://mlsanity.com/paper/2303.00106/) (2023)
- [On the Surrogate Gap between Contrastive and Supervised Losses](https://mlsanity.com/paper/2110.02501/) (2021)
- [Provable and Efficient Continual Representation Learning](https://mlsanity.com/paper/2203.02026/) (2022)
- [Learning with Explanation Constraints](https://mlsanity.com/paper/2303.14496/) (2023)
- [Empirical Evaluation and Theoretical Analysis for Representation Learning: A Survey](https://mlsanity.com/paper/2204.08226/) (2022)
- [An Augmentation Overlap Theory of Contrastive Learning](https://mlsanity.com/paper/2511.03114/) (2025)
- [Adaptation of Autoencoder for Sparsity Reduction From Clinical Notes Representation Learning](https://mlsanity.com/paper/2209.12831/) (2022)
- [Provable Pathways: Learning Multiple Tasks over Multiple Paths](https://mlsanity.com/paper/2303.04338/) (2023)
- [HairFormer: Transformer-Based Dynamic Neural Hair Simulation](https://mlsanity.com/paper/2507.12600/) (2025)
- [A Framework of Meta Functional Learning for Regularising Knowledge Transfer](https://mlsanity.com/paper/2203.14840/) (2022)
- [A Quantitative Approach to Predicting Representational Learning and Performance in Neural Networks](https://mlsanity.com/paper/2307.07575/) (2023)
- [Randomized Dimension Reduction with Statistical Guarantees](https://mlsanity.com/paper/2310.01739/) (2023)
- [PREFAB: PREFerence-based Affective Modeling for Low-Budget Self-Annotation](https://mlsanity.com/paper/2601.13904/) (2026)

## Abstract (arXiv)

Unsupervised and self-supervised learning approaches have become a crucial tool to learn representations for downstream prediction tasks. While these approaches are widely used in practice and achieve impressive empirical gains, their theoretical understanding largely lags behind. Towards bridging this gap, we present a unifying perspective where several such approaches can be viewed as imposing a regularization on the representation via a learnable function using unlabeled data. We propose a discriminative theoretical framework for analyzing the sample complexity of these approaches, which generalizes the framework of (Balcan and Blum, 2010) to allow learnable regularization functions. Our sample complexity bounds show that, with carefully chosen hypothesis classes to exploit the structure in the data, these learnable regularization functions can prune the hypothesis space, and help reduce the amount of labeled data needed. We then provide two concrete examples of functional regularization, one using auto-encoders and the other using masked self-supervision, and apply our framework to quantify the reduction in the sample complexity bound of labeled data. We also provide complementary empirical results to support our analysis.