# Multi-task Collaborative Pre-training and Individual-adaptive-tokens Fine-tuning: A Unified Framework for Brain Representation Learning

arXiv: https://arxiv.org/abs/2306.11378 · HTML: https://mlsanity.com/paper/2306.11378/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 54 references that are in the map.
- First posted on arXiv 20 Jun 2023.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis](https://mlsanity.com/paper/1908.06912/) (2019)
- [Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learning](https://mlsanity.com/paper/2102.10680/) (2021)
- [Controllable Artistic Text Style Transfer via Shape-Matching GAN](https://mlsanity.com/paper/1905.01354/) (2019)
- [Multi-Target Domain Adaptation with Collaborative Consistency Learning](https://mlsanity.com/paper/2106.03418/) (2021)
- [Latent Correlation Representation Learning for Brain Tumor Segmentation with Missing MRI Modalities](https://mlsanity.com/paper/2104.06231/) (2021)
- [Geometric Visual Similarity Learning in 3D Medical Image Self-supervised Pre-training](https://mlsanity.com/paper/2303.00874/) (2023)
- [Global-Local Transformer for Brain Age Estimation](https://mlsanity.com/paper/2109.01663/) (2021)
- [One-pass Multi-task Networks with Cross-task Guided Attention for Brain Tumor Segmentation](https://mlsanity.com/paper/1906.01796/) (2019)
- [AGTGAN: Unpaired Image Translation for Photographic Ancient Character Generation](https://mlsanity.com/paper/2303.07012/) (2023)
- [Preclinical Stage Alzheimer's Disease Detection Using Magnetic Resonance Image Scans](https://mlsanity.com/paper/2011.14139/) (2020)
- [Brain MRI-based 3D Convolutional Neural Networks for Classification of Schizophrenia and Controls](https://mlsanity.com/paper/2003.08818/) (2020)
- [Collaborative Image Understanding](https://mlsanity.com/paper/2210.11907/) (2022)
- [Connectional-Style-Guided Contextual Representation Learning for Brain Disease Diagnosis](https://mlsanity.com/paper/2306.05297/) (2023)

## Abstract (arXiv)

Structural magnetic resonance imaging (sMRI) provides accurate estimates of the brain's structural organization and learning invariant brain representations from sMRI is an enduring issue in neuroscience. Previous deep representation learning models ignore the fact that the brain, as the core of human cognitive activity, is distinct from other organs whose primary attribute is anatomy. Therefore, capturing the semantic structure that dominates interindividual cognitive variability is key to accurately representing the brain. Given that this high-level semantic information is subtle, distributed, and interdependently latent in the brain structure, sMRI-based models need to capture fine-grained details and understand how they relate to the overall global structure. However, existing models are optimized by simple objectives, making features collapse into homogeneity and worsening simultaneous representation of fine-grained information and holistic semantics, causing a lack of biological plausibility and interpretation of cognition. Here, we propose MCIAT, a unified framework that combines Multi-task Collaborative pre-training and Individual-Adaptive-Tokens fine-tuning. Specifically, we first synthesize restorative learning, age prediction auxiliary learning and adversarial learning as a joint proxy task for deep semantic representation learning. Then, a mutual-attention-based token selection method is proposed to highlight discriminative features. The proposed MCIAT achieves state-of-the-art diagnosis performance on the ADHD-200 dataset compared with several sMRI-based approaches and shows superior generalization on the MCIC and OASIS datasets. Moreover, we studied 12 behavioral tasks and found significant associations between cognitive functions and MCIAT-established representations, which verifies the interpretability of our proposed framework.