# Toward a `Standard Model' of Machine Learning

arXiv: https://arxiv.org/abs/2108.07783 · HTML: https://mlsanity.com/paper/2108.07783/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 112 references that are in the map.
- First posted on arXiv 17 Aug 2021.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [Self-Supervised Exploration via Disagreement](https://mlsanity.com/paper/1906.04161/) (2019)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [Conditional Random Fields as Recurrent Neural Networks](https://mlsanity.com/paper/1502.03240/) (2015)
- [AutoML-Zero: Evolving Machine Learning Algorithms From Scratch](https://mlsanity.com/paper/2003.03384/) (2020)
- [Harnessing Deep Neural Networks with Logic Rules](https://mlsanity.com/paper/1603.06318/) (2016)
- [Learning in Implicit Generative Models](https://mlsanity.com/paper/1610.03483/) (2016)
- [Neural Variational Inference and Learning in Belief Networks](https://mlsanity.com/paper/1402.0030/) (2014)
- [Composing graphical models with neural networks for structured representations and fast inference](https://mlsanity.com/paper/1603.06277/) (2016)
- [On Learning Intrinsic Rewards for Policy Gradient Methods](https://mlsanity.com/paper/1804.06459/) (2018)
- [Stochastic Variational Deep Kernel Learning](https://mlsanity.com/paper/1611.00336/) (2016)
- [Reward Augmented Maximum Likelihood for Neural Structured Prediction](https://mlsanity.com/paper/1609.00150/) (2016)
- [Unsupervised Text Style Transfer using Language Models as Discriminators](https://mlsanity.com/paper/1805.11749/) (2018)
- [Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense](https://mlsanity.com/paper/2004.09044/) (2020)
- [Generalized Variational Inference: Three arguments for deriving new Posteriors](https://mlsanity.com/paper/1904.02063/) (2019)
- [Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale](https://mlsanity.com/paper/1812.00417/) (2018)

## Built on by (most cited first, 8 of 8)

- [ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings](https://mlsanity.com/paper/2305.11554/) (2023)
- [Language Models Meet World Models: Embodied Experiences Enhance Language Models](https://mlsanity.com/paper/2305.10626/) (2023)
- [Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planning](https://mlsanity.com/paper/2312.05230/) (2023)
- [Critique of World Model](https://mlsanity.com/paper/2507.05169/) (2025)
- [BertNet: Harvesting Knowledge Graphs with Arbitrary Relations from Pretrained Language Models](https://mlsanity.com/paper/2206.14268/) (2022)
- [Unified Generation, Reconstruction, and Representation: Generalized Diffusion with Adaptive Latent Encoding-Decoding](https://mlsanity.com/paper/2402.19009/) (2024)
- [Text Alignment Is An Efficient Unified Model for Massive NLP Tasks](https://mlsanity.com/paper/2307.02729/) (2023)
- [Take Goodhart Seriously: Principled Limit on General-Purpose AI Optimization](https://mlsanity.com/paper/2510.02840/) (2025)

## Abstract (arXiv)

Machine learning (ML) is about computational methods that enable machines to learn concepts from experience. In handling a wide variety of experience ranging from data instances, knowledge, constraints, to rewards, adversaries, and lifelong interaction in an ever-growing spectrum of tasks, contemporary ML/AI (artificial intelligence) research has resulted in a multitude of learning paradigms and methodologies. Despite the continual progresses on all different fronts, the disparate narrowly focused methods also make standardized, composable, and reusable development of ML approaches difficult, and preclude the opportunity to build AI agents that panoramically learn from all types of experience. This article presents a standardized ML formalism, in particular a `standard equation' of the learning objective, that offers a unifying understanding of many important ML algorithms in the supervised, unsupervised, knowledge-constrained, reinforcement, adversarial, and online learning paradigms, respectively -- those diverse algorithms are encompassed as special cases due to different choices of modeling components. The framework also provides guidance for mechanical design of new ML approaches and serves as a promising vehicle toward panoramic machine learning with all experience.