# MTNeuro: A Benchmark for Evaluating Representations of Brain Structure Across Multiple Levels of Abstraction

arXiv: https://arxiv.org/abs/2301.00345 · HTML: https://mlsanity.com/paper/2301.00345/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 76 references that are in the map.
- First posted on arXiv 1 Jan 2023.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Pyramid Attention Network for Semantic Segmentation](https://mlsanity.com/paper/1805.10180/) (2018)
- [Superhuman Accuracy on the SNEMI3D Connectomics Challenge](https://mlsanity.com/paper/1706.00120/) (2017)
- [On the Compactness, Efficiency, and Representation of 3D Convolutional Networks: Brain Parcellation as a Pretext Task](https://mlsanity.com/paper/1707.01992/) (2017)
- [Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction](https://mlsanity.com/paper/2102.10106/) (2021)
- [A Study of the Generalizability of Self-Supervised Representations](https://mlsanity.com/paper/2109.09150/) (2021)
- [Small-World Brain Networks Revisited](https://mlsanity.com/paper/1608.05665/) (2016)
- [AxonEM Dataset: 3D Axon Instance Segmentation of Brain Cortical Regions](https://mlsanity.com/paper/2107.05451/) (2021)
- [Automatic Assessment of Alzheimer's Disease Diagnosis Based on Deep Learning Techniques](https://mlsanity.com/paper/2105.08446/) (2021)
- [Quantifying mesoscale neuroanatomy using X-ray microtomography](https://mlsanity.com/paper/1604.03629/) (2016)

## Built on by (most cited first, 7 of 7)

- [What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2306.15521/) (2023)
- [WBCAtt: A White Blood Cell Dataset Annotated with Detailed Morphological Attributes](https://mlsanity.com/paper/2306.13531/) (2023)
- [BRACTIVE: A Brain Activation Approach to Human Visual Brain Learning](https://mlsanity.com/paper/2405.18808/) (2024)
- [A Large-scale Benchmark on Geological Fault Delineation Models: Domain Shift, Training Dynamics, Generalizability, Evaluation and Inferential Behavior](https://mlsanity.com/paper/2505.08585/) (2025)
- [Brainformer: Mimic Human Visual Brain Functions to Machine Vision Models via fMRI](https://mlsanity.com/paper/2312.00236/) (2023)
- [Scale-Aware Self-Supervised Learning for Segmentation of Small and Sparse Structures](https://mlsanity.com/paper/2601.18619/) (2026)
- [WBCAtt+: Fine-Grained Pixel-Level Morphological Annotations for White Blood Cell Images](https://mlsanity.com/paper/2605.19692/) (2026)

## Abstract (arXiv)

There are multiple scales of abstraction from which we can describe the same image, depending on whether we are focusing on fine-grained details or a more global attribute of the image. In brain mapping, learning to automatically parse images to build representations of both small-scale features (e.g., the presence of cells or blood vessels) and global properties of an image (e.g., which brain region the image comes from) is a crucial and open challenge. However, most existing datasets and benchmarks for neuroanatomy consider only a single downstream task at a time. To bridge this gap, we introduce a new dataset, annotations, and multiple downstream tasks that provide diverse ways to readout information about brain structure and architecture from the same image. Our multi-task neuroimaging benchmark (MTNeuro) is built on volumetric, micrometer-resolution X-ray microtomography images spanning a large thalamocortical section of mouse brain, encompassing multiple cortical and subcortical regions. We generated a number of different prediction challenges and evaluated several supervised and self-supervised models for brain-region prediction and pixel-level semantic segmentation of microstructures. Our experiments not only highlight the rich heterogeneity of this dataset, but also provide insights into how self-supervised approaches can be used to learn representations that capture multiple attributes of a single image and perform well on a variety of downstream tasks. Datasets, code, and pre-trained baseline models are provided at: https://mtneuro.github.io/ .