Data to 14 Oct 2026

Taskonomy: Disentangling Task Transfer Learning

arXiv 1804.08328 ↗First posted 23 Apr 2018cs.CV cs.AI cs.LGcited by 634

Not on the map. Why?

Abstract (arXiv)

Do visual tasks have a relationship, or are they unrelated? For instance, could having surface normals simplify estimating the depth of an image? Intuition answers these questions positively, implying existence of a structure among visual tasks. Knowing this structure has notable values; it is the concept underlying transfer learning and provides a principled way for identifying redundancies across tasks, e.g., to seamlessly reuse supervision among related tasks or solve many tasks in one system without piling up the complexity. We proposes a fully computational approach for modeling the structure of space of visual tasks. This is done via finding (first and higher-order) transfer learning dependencies across a dictionary of twenty six 2D, 2.5D, 3D, and semantic tasks in a latent space. The product is a computational taxonomic map for task transfer learning. We study the consequences of this structure, e.g. nontrivial emerged relationships, and exploit them to reduce the demand for labeled data. For example, we show that the total number of labeled datapoints needed for solving a set of 10 tasks can be reduced by roughly 2/3 (compared to training independently) while keeping the performance nearly the same. We provide a set of tools for computing and probing this taxonomical structure including a solver that users can employ to devise efficient supervision policies for their use cases.

Key facts. Cited by 634 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 116; 2022: 111; 2023: 137; 2024: 95; 2025: 94; 2026: 81.

634 papers.

Sort:

By area:

2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 692
2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 415
2024Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimationcited by 395
2024UniDepth: Universal Monocular Metric Depth Estimationcited by 382
2023Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Imagecited by 365
2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
2025π3: Permutation-Equivariant Visual Geometry Learningcited by 319
2025MapAnything: Universal Feed-Forward Metric 3D Reconstructioncited by 314
2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302

38 more

2024MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervisioncited by 299
2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
2025MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Detailscited by 261
2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
2025Video models are zero-shot learners and reasonerscited by 220
2022Omnivore: A Single Model for Many Visual Modalitiescited by 215
2021Concealed Object Detectioncited by 212
2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 200
2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
2025UniDepthV2: Universal Monocular Metric Depth Estimation Made Simplercited by 173
2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 172
2021Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Studycited by 168
2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 160
2021Geometry Uncertainty Projection Network for Monocular 3D Object Detectioncited by 156
2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
2023Neural Architecture Search: Insights from 1000 Paperscited by 139
2022Multimodal Token Fusion for Vision Transformerscited by 136
2024Rethinking Inductive Biases for Surface Normal Estimationcited by 124
2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 121
20234M: Massively Multimodal Masked Modelingcited by 120
20223D Common Corruptions and Data Augmentationcited by 119
2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
2022The CLRS Algorithmic Reasoning Benchmarkcited by 106
2021Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Mergingcited by 104
2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
2023Soft Merging of Experts with Adaptive Routingcited by 90
2021Ensembling Off-the-shelf Models for GAN Trainingcited by 86
2021Multi-Task Self-Training for Learning General Representationscited by 85
2021RotoGrad: Gradient Homogenization in Multitask Learningcited by 83
2022An Information-Theoretic Approach to Transferability in Task Transfer Learningcited by 81
2021Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimationcited by 80
2023FAMO: Fast Adaptive Multitask Optimizationcited by 78
2024MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learningcited by 77
2022Offline Visual Representation Learning for Embodied Navigationcited by 75

50 most cited of 634 All 634 (JSON)