mlsanityData to 14 Oct 2026

The Information Complexity of Learning Tasks, their Structure and their Distance

arXiv 1904.03292 ↗First posted 5 Apr 2019cs.LG cs.IT math.ITcited by 36

Not on the map. Why?

Abstract (arXiv)

We introduce an asymmetric distance in the space of learning tasks, and a framework to compute their complexity. These concepts are foundational for the practice of transfer learning, whereby a parametric model is pre-trained for a task, and then fine-tuned for another. The framework we develop is non-asymptotic, captures the finite nature of the training dataset, and allows distinguishing learning from memorization. It encompasses, as special cases, classical notions from Kolmogorov complexity, Shannon, and Fisher Information. However, unlike some of those frameworks, it can be applied to large-scale models and real-world datasets. Our framework is the first to measure complexity in a way that accounts for the effect of the optimization scheme, which is critical in Deep Learning.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). First posted on arXiv 5 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 7; 2023: 8; 2024: 5; 2025: 8; 2026: 3.

36 papers.

Sort:

By area:

  1. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
  2. 2023The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learningcited by 63
  3. 2022Less Data, More Knowledge: Building Next Generation Semantic Communication Networkscited by 62
  4. 2025What Has a Foundation Model Found? Using Inductive Bias to Probe for World Modelscited by 47
  5. 2021A linearized framework and a new benchmark for model selection for fine-tuningcited by 41
  6. 2021Probing transfer learning with a model of synthetic correlated datasetscited by 25
  7. 2024Concept-skill Transferability-based Data Selection for Large Vision-Language Modelscited by 24
  8. 2023Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language Datacited by 19
  9. 2025Selective Task Group Updates for Multi-Task Optimizationcited by 17
  10. 2024What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biasescited by 13
  11. 2022The Curse of Low Task Diversity: On the Failure of Transfer Learning to Outperform MAML and Their Empirical Equivalencecited by 10
  12. 2023Limits of Model Selection under Transfer Learningcited by 9
24 more
  1. 2023Model-agnostic Measure of Generalization Difficultycited by 7
  2. 2021Task Fingerprinting for Meta Learning in Biomedical Image Analysiscited by 6
  3. 2022X-Learner: Learning Cross Sources and Tasks for Universal Visual Representationcited by 6
  4. 2022Hierarchical Optimal Transport for Comparing Histopathology Datasetscited by 6
  5. 2025Bridging Kolmogorov Complexity and Deep Learning: Asymptotically Optimal Description Length Objectives for Transformerscited by 6
  6. 2022On the Learnability of Physical Concepts: Can a Neural Network Understand What's Real?cited by 5
  7. 2022Meta-Learning for Unsupervised Outlier Detection with Optimal Transportcited by 5
  8. 2023Task Grouping for Automated Multi-Task Machine Learning via Task Affinity Predictioncited by 4
  9. 2022Disentangling Learnable and Memorizable Data via Contrastive Learning for Semantic Communicationscited by 3
  10. 2023Provable Pathways: Learning Multiple Tasks over Multiple Pathscited by 3
  11. 2025Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learningcited by 3
  12. 2023Semantic Communication of Learnable Conceptscited by 2
  13. 2024Beyond Knowledge Silos: Task Fingerprinting for Democratization of Medical Imaging AIcited by 2
  14. 2025Measuring Fine-Grained Relatedness in Multitask Learning via Data Attributioncited by 1
  15. 2025The Advantage of Fine-Grained Trainingcited by 1
  16. 2025Task Specific Sharpness Aware O-RAN Resource Management using Multi Agent Reinforcement Learningcited by 1
  17. 2025Efficiently Estimating Data Efficiency for Language Model Fine-tuningcited by 1
  18. 2021The Curse of Zero Task Diversity: On the Failure of Transfer Learning to Outperform MAML and their Empirical Equivalencecited by 0
  19. 2023Model Complexity of Program Phasescited by 0
  20. 2024CLAMS: A System for Zero-Shot Model Selection for Clusteringcited by 0
  21. 2024LargePiG: Your Large Language Model is Secretly a Pointer Generatorcited by 0
  22. 2026Ensemble Prediction of Task Affinity for Efficient Multi-Task Learningcited by 0
  23. 2026Data Difficulty and the Generalization--Extrapolation Tradeoff in LLM Fine-Tuningcited by 0
  24. 2026Kernelized Parameter Sensitivity Atlases for Amortized Conditional Interpretabilitycited by 0