Statistical Deficiency for Task Inclusion Estimation
Not on the map. Why?
Abstract (arXiv)
Tasks are central in machine learning, as they are the most natural objects to assess the capabilities of current models. The trend is to build general models able to address any task. Even though transfer learning and multitask learning try to leverage the underlying task space, no well-founded tools are available to study its structure. This study proposes a theoretically grounded setup to define the notion of task and to compute the bf inclusion between two tasks from a statistical deficiency point of view. We propose a tractable proxy as information sufficiency to estimate the degree of inclusion between tasks, show its soundness on synthetic data, and use it to reconstruct empirically the classic NLP pipeline.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 50 of its 107 references that are in the map. First posted on arXiv 7 Mar 2025.
Builds on
50 of 107 references.
References · 50
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
- 2016Understanding intermediate layers using linear classifier probescited by 1,353
- 2020Gradient Surgery for Multi-Task Learningcited by 1,256
- 2017A Survey on Multi-Task Learningcited by 840
- 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
- 2020A Primer in BERTology: What we know about how BERT workscited by 815
- 2021Robust fine-tuning of zero-shot modelscited by 797
- 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
38 more
- 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
- 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
- 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
- 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
- 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
- 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
- 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 287
- 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 215
- 2020Information-Theoretic Probing for Linguistic Structurecited by 207
- 2019Task2Vec: Task Embedding for Meta-Learningcited by 205
- 2015The Benefit of Multitask Representation Learningcited by 195
- 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 184
- 2020A Theory of Usable Information Under Computational Constraintscited by 162
- 2020Exploring and Predicting Transferability across NLP Taskscited by 128
- 2020Understanding and Improving Information Transfer in Multi-Task Learningcited by 114
- 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
- 2020A unifying mutual information view of metric learning: cross-entropy vs. pairwise lossescited by 100
- 2024Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Datacited by 85
- 2022An Information-Theoretic Approach to Transferability in Task Transfer Learningcited by 81
- 2021OTCE: A Transferability Metric for Cross-Domain Cross-Task Representationscited by 62
- 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
- 2021Probing BERT in Hyperbolic Spacescited by 50
- 2020On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformerscited by 39
- 2019A Principled Approach for Learning Task Similarity in Multitask Learningcited by 32
- 2021How transfer learning impacts linguistic knowledge in deep NLP models?cited by 32
- 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26
- 2024A deeper look at depth pruning of LLMscited by 23
- 2024Ethos: Rectifying Language Models in Orthogonal Parameter Spacecited by 20
- 2022On the Transformation of Latent Space in Fine-Tuned NLP Modelscited by 19
- 2025Optimal Task Order for Continual Learning of Multiple Taskscited by 14
- 2023Investigating semantic subspaces of Transformer sentence embeddings through linear structural probingcited by 12
- 2024Task Arithmetic Through The Lens Of One-Shot Federated Learningcited by 12
- 2021To Share or not to Share: Predicting Sets of Sources for Model Transfer Learningcited by 11
- 2024When is an Embedding Model More Promising than Another?cited by 7
- 2025Leveraging the true depth of LLMscited by 7
- 2020On the Estimation of Information Measures of Continuous Distributionscited by 6
- 2024textttCOSMIC: Mutual Information for Task-Agnostic Summarization Evaluationcited by 6
- 2024Large Language Models Are Overparameterized Text Encoderscited by 6
Built on by
Citing papers per year: 2025: 1.
1 paper.