mlsanityData to 7 Oct 2026

On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligence

arXiv 2207.04630 ↗First posted 11 Jul 2022cs.AI cs.CV cs.ITcited by 50

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Ten years into the revival of deep networks and artificial intelligence, we propose a theoretical framework that sheds light on understanding deep networks within a bigger picture of Intelligence in general. We introduce two fundamental principles, Parsimony and Self-consistency, that address two fundamental questions regarding Intelligence: what to learn and how to learn, respectively. We believe the two principles are the cornerstones for the emergence of Intelligence, artificial or natural. While these two principles have rich classical roots, we argue that they can be stated anew in entirely measurable and computable ways. More specifically, the two principles lead to an effective and efficient computational framework, compressive closed-loop transcription, that unifies and explains the evolution of modern deep networks and many artificial intelligence practices. While we mainly use modeling of visual data as an example, we believe the two principles will unify understanding of broad families of autonomous intelligent systems and provide a framework for understanding the brain.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 54 of its 152 references that are in the map. First posted on arXiv 11 Jul 2022.

Builds on

54 of 152 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2013Intriguing properties of neural networkscited by 5,114
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  10. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  11. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  12. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
38 more
  1. 2016Group Equivariant Convolutional Networkscited by 1,154
  2. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  3. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  4. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  5. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  6. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  7. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  8. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  9. 2019Deep Equilibrium Modelscited by 603
  10. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  11. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 277
  12. 2015Highway Networkscited by 257
  13. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  14. 2018A General Theory of Equivariant CNNs on Homogeneous Spacescited by 223
  15. 2021Fake It Till You Make It: Face analysis in the wild using synthetic data alonecited by 200
  16. 2017VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learningcited by 199
  17. 2018Why do deep convolutional networks generalize so poorly to small image transformations?cited by 198
  18. 2020The Next Decade in AI: Four Steps Towards Robust Artificial Intelligencecited by 194
  19. 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 183
  20. 2021The Principles of Deep Learning Theorycited by 169
  21. 2020Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reductioncited by 141
  22. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
  23. 2019Implicit Deep Learningcited by 124
  24. 2015Deep Convolutional Inverse Graphics Networkcited by 121
  25. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
  26. 2020Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Modelcited by 108
  27. 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
  28. 2021ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reductioncited by 84
  29. 2017Exploring the Landscape of Spatial Robustnesscited by 81
  30. 2014PCANet: A Simple Deep Learning Baseline for Image Classification?cited by 70
  31. 2020Learning Physical Graph Representations from Visual Scenescited by 50
  32. 2015A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extractioncited by 46
  33. 2021The Dimpled Manifold Model of Adversarial Examples in Machine Learningcited by 41
  34. 2020Deep Networks and the Multiple Manifold Problemcited by 35
  35. 2017A Geometric View of Optimal Transportation and Generative Modelcited by 31
  36. 2019Convergence of Learning Dynamics in Stackelberg Gamescited by 28
  37. 2019A Rate-Distortion Framework for Explaining Neural Network Decisionscited by 24
  38. 2021Closed-Loop Data Transcription to an LDR via Minimaxing Rate Reductioncited by 17

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 20; 2024: 11; 2025: 7; 2026: 6.

50 papers.

Sort:

By area:

  1. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  2. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  3. 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 107
  4. 2023White-Box Transformers via Sparse Rate Reductioncited by 77
  5. 2022The Hidden Uniform Cluster Prior in Self-Supervised Learningcited by 57
  6. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 49
  7. 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Modelscited by 48
  8. 2022Green Learning: Introduction, Examples and Outlookcited by 30
  9. 2023Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researcherscited by 25
  10. 2023GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generationcited by 25
  11. 2023Structuring Representation Geometry with Rotationally Equivariant Contrastive Learningcited by 24
  12. 2023Emergence of Segmentation with Minimalistic White-Box Transformerscited by 23
38 more
  1. 2023Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regressioncited by 20
  2. 2025Block-Recurrent Dynamics in Vision Transformerscited by 16
  3. 2024A Global Geometric Analysis of Maximal Coding Rate Reductioncited by 13
  4. 2023Representation Learning via Manifold Flattening and Reconstructioncited by 11
  5. 2024Masked Completion via Structured Diffusion with White-Box Transformerscited by 9
  6. 2024GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Costcited by 9
  7. 2024Applications of Generative AI (GAI) for Mobile and Wireless Networking: A Surveycited by 9
  8. 2023Clustering-based Domain-Incremental Learningcited by 8
  9. 2023Attribute-Aware Deep Hashing with Self-Consistency for Large-Scale Fine-Grained Image Retrievalcited by 8
  10. 2022Minimalistic Unsupervised Learning with the Sparse Manifold Transformcited by 6
  11. 2023Eight challenges in developing theory of intelligencecited by 6
  12. 2024Task structure and nonlinearity jointly determine learned representational geometrycited by 5
  13. 2025Attention-Only Transformers via Unrolled Subspace Denoisingcited by 5
  14. 2023Closed-Loop Transcription via Convolutional Sparse Codingcited by 4
  15. 2023Large Language Models' Understanding of Math: Source Criticism and Extrapolationcited by 4
  16. 2024Disentangling Representations through Multi-task Learningcited by 4
  17. 2024Out-of-Distribution Detection with Prototypical Outlier Proxycited by 4
  18. 2026Understanding Self-Supervised Learning via Latent Distribution Matchingcited by 4
  19. 2023Differentiable modeling to unify machine learning and physical models and advance Geosciencescited by 3
  20. 2023Understanding the World to Solve Social Dilemmas Using Multi-Agent Reinforcement Learningcited by 2
  21. 2023Pathway toward prior knowledge-integrated machine learning in engineeringcited by 2
  22. 2025Semantic Communication meets System 2 ML: How Abstraction, Compositionality and Emergent Languages Shape Intelligencecited by 2
  23. 2023Inferring Inferencecited by 1
  24. 2024A mathematical framework of intelligence and consciousness based on Riemannian Geometrycited by 1
  25. 2024LoLaFL: Low-Latency Federated Learning via Forward-only Propagationcited by 1
  26. 2025Twofold Debiasing Enhances Fine-Grained Learning with Coarse Labelscited by 1
  27. 2025A Representationalist, Functionalist and Naturalistic Conception of Intelligence as a Foundation for AGIcited by 1
  28. 2025Physics Informed Constrained Learning of Dynamics from Static Datacited by 1
  29. 2022Pathway to Future Symbiotic Creativitycited by 0
  30. 2022Statistical Foundation Behind Machine Learning and Its Impact on Computer Visioncited by 0
  31. 2023Enhancing Text Generation with Cooperative Trainingcited by 0
  32. 2024The Extrapolation Power of Implicit Modelscited by 0
  33. 2025Scaling Non-Parametric Sampling with Representationcited by 0
  34. 2026SPIKE: Sparse Koopman Regularization for Physics-Informed Neural Networkscited by 0
  35. 2026Epistemic Compression: The Case for Deliberate Ignorance in High-Stakes AIcited by 0
  36. 2026Semantic-based Distributed Learning for Diverse and Discriminative Representationscited by 0
  37. 2026Estimating Implicit Regularization in Deep Learningcited by 0
  38. 2026On the Limits of Maximal Coding Rate Reduction for Out-of-Distribution Generalisationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021A brain basis of dynamical intelligence for AI and computational neuroscience
  2. 2024Possible Principles for Aligned Structure Learning Agents
  3. 2022Designing Ecosystems of Intelligence from First Principles
  4. 2022A World-Self Model Towards Understanding Intelligence
  5. 2025Cognitive Loop via In-Situ Optimization: Self-Adaptive Reasoning for Science
  6. 2025The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain
  7. 2025The Recursive Coherence Principle: A Formal Constraint on Scalable Intelligence, Alignment, and Reasoning Architecture
  8. 2025From generative AI to the brain: five takeaways