mlsanityData to 14 Oct 2026

A picture of the space of typical learnable tasks

arXiv 2210.17011 ↗First posted 31 Oct 2022cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

We develop information geometric techniques to understand the representations learned by deep networks when they are trained on different tasks using supervised, meta-, semi-supervised and contrastive learning. We shed light on the following phenomena that relate to the structure of the space of tasks: (1) the manifold of probabilistic models trained on different tasks using different representation learning methods is effectively low-dimensional; (2) supervised learning on one task results in a surprising amount of progress even on seemingly dissimilar tasks; progress on other tasks is larger if the training task has diverse classes; (3) the structure of the space of tasks indicated by our analysis is consistent with parts of the Wordnet phylogenetic tree; (4) episodic meta-learning algorithms and supervised learning traverse different trajectories during training but they fit similar models eventually; (5) contrastive and semi-supervised learning methods traverse trajectories similar to those of supervised learning. We use classification tasks constructed from the CIFAR-10 and Imagenet datasets to study these phenomena.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 44 of its 88 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

44 of 88 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2016Wide Residual Networkscited by 3,070
  4. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  5. 2016Matching Networks for One Shot Learningcited by 2,255
  6. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  8. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
  9. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  10. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  11. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  12. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
32 more
  1. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  2. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  3. 2019Benign Overfitting in Linear Regressioncited by 549
  4. 2019Making Convolutional Networks Shift-Invariant Againcited by 391
  5. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  6. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 300
  7. 2019A Baseline for Few-Shot Image Classificationcited by 298
  8. 2019Revisiting Self-Supervised Visual Representation Learningcited by 251
  9. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  10. 2019Task2Vec: Task Embedding for Meta-Learningcited by 205
  11. 2017Multi-task Self-Supervised Visual Learningcited by 200
  12. 2022Datamodels: Predicting Predictions from Training Datacited by 189
  13. 2018A mathematical theory of semantic development in deep neural networkscited by 186
  14. 2018Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learningcited by 174
  15. 2020Provable Meta-Learning of Linear Representationscited by 161
  16. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  17. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 109
  18. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  19. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  20. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 91
  21. 2019Meta-Q-Learningcited by 62
  22. 2020Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learningcited by 62
  23. 2019Understanding Generalization through Visualizationscited by 54
  24. 2021Model Zoo: A Growing "Brain" That Learns Continuallycited by 54
  25. 2021Optimal Representations for Covariate Shiftcited by 51
  26. 2020A No-Free-Lunch Theorem for MultiTask Learningcited by 28
  27. 2022Deconstructing Distributions: A Pointwise Framework of Learningcited by 19
  28. 2022Is Self-Supervised Learning More Robust Than Supervised Learning?cited by 17
  29. 2020An Information-Geometric Distance on the Space of Taskscited by 15
  30. 2021Does the Data Induce Capacity Control in Deep Learning?cited by 14
  31. 2018PCA of high dimensional random walks with comparison to neural network trainingcited by 11
  32. 2020A Free-Energy Principle for Representation Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 4; 2026: 1.

7 papers.

  1. 2025MorphMark: Flexible Adaptive Watermarking for Large Language Modelscited by 24
  2. 2026Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentioncited by 10
  3. 2023Learning Capacity: A Measure of the Effective Dimensionality of a Modelcited by 4
  4. 2024Many Perception Tasks are Highly Redundant Functions of their Input Datacited by 4
  5. 2025Rethinking Few Shot CLIP Benchmarks: A Critical Analysis in the Inductive Settingcited by 4
  6. 2025An Analytical Characterization of Sloppiness in Neural Networks: Insights from Linear Modelscited by 1
  7. 2025Ensemble-MIX: Enhancing Sample-Efficiency in Multi-Agent RL via Selective Ensemble-Based Explorationcited by 1