# Boosting 3D Neuron Segmentation with 2D Vision Transformer Pre-trained on Natural Images

arXiv: https://arxiv.org/abs/2405.02686 · HTML: https://mlsanity.com/paper/2405.02686/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 20 references that are in the map.
- First posted on arXiv 4 May 2024.

## Builds on (most cited first, 5 of 5)

- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Adapting Pre-trained Vision Transformers from 2D to 3D through Weight Inflation Improves Medical Image Segmentation](https://mlsanity.com/paper/2302.04303/) (2023)
- [Single Neuron Segmentation using Graph-based Global Reasoning with Auxiliary Skeleton Loss from 3D Optical Microscope Images](https://mlsanity.com/paper/2101.08910/) (2021)
- [PointNeuron: 3D Neuron Reconstruction via Geometry and Topology Learning of Point Clouds](https://mlsanity.com/paper/2210.08305/) (2022)
- [Voxel-wise Cross-Volume Representation Learning for 3D Neuron Reconstruction](https://mlsanity.com/paper/2108.06522/) (2021)

## Built on by (most cited first, 2 of 2)

- [NeuroSeg Meets DINOv3: Transferring 2D Self-Supervised Visual Priors to 3D Neuron Segmentation via DINOv3 Initialization](https://mlsanity.com/paper/2603.23104/) (2026)
- [DINeuro: Distilling Knowledge from 2D Natural Images via Deformable Tubular Transferring Strategy for 3D Neuron Reconstruction](https://mlsanity.com/paper/2410.22078/) (2024)

## Abstract (arXiv)

Neuron reconstruction, one of the fundamental tasks in neuroscience, rebuilds neuronal morphology from 3D light microscope imaging data. It plays a critical role in analyzing the structure-function relationship of neurons in the nervous system. However, due to the scarcity of neuron datasets and high-quality SWC annotations, it is still challenging to develop robust segmentation methods for single neuron reconstruction. To address this limitation, we aim to distill the consensus knowledge from massive natural image data to aid the segmentation model in learning the complex neuron structures. Specifically, in this work, we propose a novel training paradigm that leverages a 2D Vision Transformer model pre-trained on large-scale natural images to initialize our Transformer-based 3D neuron segmentation model with a tailored 2D-to-3D weight transferring strategy. Our method builds a knowledge sharing connection between the abundant natural and the scarce neuron image domains to improve the 3D neuron segmentation ability in a data-efficiency manner. Evaluated on a popular benchmark, BigNeuron, our method enhances neuron segmentation performance by 8.71% over the model trained from scratch with the same amount of training samples.