# Self-Supervised Neuron Segmentation with Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2310.04148 · HTML: https://mlsanity.com/paper/2310.04148/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 54 references that are in the map.
- First posted on arXiv 6 Oct 2023.

## Builds on (most cited first, 24 of 24)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Adversarial AutoAugment](https://mlsanity.com/paper/1912.11188/) (2019)
- [SimCVD: Simple Contrastive Voxel-Wise Representation Distillation for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2108.06227/) (2021)
- [Superhuman Accuracy on the SNEMI3D Connectomics Challenge](https://mlsanity.com/paper/1706.00120/) (2017)
- [Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentation](https://mlsanity.com/paper/2306.04811/) (2023)
- [AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoders](https://mlsanity.com/paper/2211.09120/) (2022)
- [Iteratively-Refined Interactive 3D Medical Image Segmentation with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1911.10334/) (2019)
- [Learning Multiscale Consistency for Self-supervised Electron Microscopy Instance Segmentation](https://mlsanity.com/paper/2308.09917/) (2023)
- [Machine learning of hierarchical clustering to segment 2D and 3D images](https://mlsanity.com/paper/1303.6163/) (2013)
- [SelectAugment: Hierarchical Deterministic Sample Selection for Data Augmentation](https://mlsanity.com/paper/2112.02862/) (2021)
- [Local Patch AutoAugment with Multi-Agent Collaboration](https://mlsanity.com/paper/2103.11099/) (2021)

## Built on by (most cited first, 19 of 19)

- [HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrieval](https://mlsanity.com/paper/2512.02792/) (2025)
- [BIMCV-R: A Landmark Dataset for 3D CT Text-Image Retrieval](https://mlsanity.com/paper/2403.15992/) (2024)
- [Research on Adverse Drug Reaction Prediction Model Combining Knowledge Graph Embedding and Deep Learning](https://mlsanity.com/paper/2407.16715/) (2024)
- [Rumor Detection with a novel graph neural network approach](https://mlsanity.com/paper/2403.16206/) (2024)
- [Mask Factory: Towards High-quality Synthetic Data Generation for Dichotomous Image Segmentation](https://mlsanity.com/paper/2412.19080/) (2024)
- [Image Captioning in news report scenario](https://mlsanity.com/paper/2403.16209/) (2024)
- [Research on Image Recognition Technology Based on Multimodal Deep Learning](https://mlsanity.com/paper/2405.03091/) (2024)
- [SED-MVS: Segmentation-Driven and Edge-Aligned Deformation Multi-View Stereo with Depth Restoration and Occlusion Constraint](https://mlsanity.com/paper/2503.13721/) (2025)
- [What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generation](https://mlsanity.com/paper/2602.11499/) (2026)
- [TokenUnify: Scaling Up Autoregressive Pretraining for Neuron Segmentation](https://mlsanity.com/paper/2405.16847/) (2024)
- [EAGLE: An Edge-Aware Gradient Localization Enhanced Loss for CT Image Reconstruction](https://mlsanity.com/paper/2403.10695/) (2024)
- [Efficient In-Context Medical Segmentation with Meta-driven Visual Prompt Selection](https://mlsanity.com/paper/2407.11188/) (2024)
- [DVP-MVS++: Synergize Depth-Normal-Edge and Harmonized Visibility Prior for Multi-View Stereo](https://mlsanity.com/paper/2506.13215/) (2025)
- [End-To-End Training and Testing Gamification Framework to Learn Human Highway Driving](https://mlsanity.com/paper/2404.10849/) (2024)
- [IndusAgent: Reinforcing Open-Vocabulary Industrial Anomaly Detection with Agentic Tools](https://mlsanity.com/paper/2605.20682/) (2026)
- [E-commerce Webpage Recommendation Scheme Base on Semantic Mining and Neural Networks](https://mlsanity.com/paper/2409.07033/) (2024)
- [Dual form Complementary Masking for Domain-Adaptive Image Segmentation](https://mlsanity.com/paper/2507.12008/) (2025)
- [NeuroMamba: Multi-Perspective Feature Interaction with Visual Mamba for Neuron Segmentation](https://mlsanity.com/paper/2601.15929/) (2026)
- [NeuroRefiner: Morphology-Aware Multi-Agent Refinement for 3D Fluorescence Microscopy Neuron Segmentation](https://mlsanity.com/paper/2608.09636/) (2026)

## Abstract (arXiv)

The performance of existing supervised neuron segmentation methods is highly dependent on the number of accurate annotations, especially when applied to large scale electron microscopy (EM) data. By extracting semantic information from unlabeled data, self-supervised methods can improve the performance of downstream tasks, among which the mask image model (MIM) has been widely used due to its simplicity and effectiveness in recovering original information from masked images. However, due to the high degree of structural locality in EM images, as well as the existence of considerable noise, many voxels contain little discriminative information, making MIM pretraining inefficient on the neuron segmentation task. To overcome this challenge, we propose a decision-based MIM that utilizes reinforcement learning (RL) to automatically search for optimal image masking ratio and masking strategy. Due to the vast exploration space, using single-agent RL for voxel prediction is impractical. Therefore, we treat each input patch as an agent with a shared behavior policy, allowing for multi-agent collaboration. Furthermore, this multi-agent model can capture dependencies between voxels, which is beneficial for the downstream segmentation task. Experiments conducted on representative EM datasets demonstrate that our approach has a significant advantage over alternative self-supervised methods on the task of neuron segmentation. Code is available at urlhttps://github.com/ydchen0806/dbMiM.