# BertsWin: Resolving Topological Sparsity in 3D Masked Autoencoders via Component-Balanced Structural Optimization

arXiv: https://arxiv.org/abs/2512.21769 · HTML: https://mlsanity.com/paper/2512.21769/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 25 Dec 2025.

## Builds on (most cited first, 13 of 13)

- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [MONAI: An open-source framework for deep learning in healthcare](https://mlsanity.com/paper/2211.02701/) (2022)
- [SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images](https://mlsanity.com/paper/2310.15161/) (2023)
- [Loss Functions and Metrics in Deep Learning](https://mlsanity.com/paper/2307.02694/) (2023)
- [Revisiting MAE pre-training for 3D medical image segmentation](https://mlsanity.com/paper/2410.23132/) (2024)
- [Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives](https://mlsanity.com/paper/2206.01136/) (2022)
- [Revisiting LARS for Large Batch Training Generalization of Neural Networks](https://mlsanity.com/paper/2309.14053/) (2023)
- [Minimum Noticeable Difference based Adversarial Privacy Preserving Image Generation](https://mlsanity.com/paper/2206.08638/) (2022)
- [Focus on Texture: Rethinking Pre-training in Masked Autoencoders for Medical Image Classification](https://mlsanity.com/paper/2507.10869/) (2025)
- [GCond: Gradient Conflict Resolution via Accumulation-based Stabilization for Large-Scale Multi-Task Learning](https://mlsanity.com/paper/2509.07252/) (2025)
- [LuminanceL1Loss: A loss function which measures percieved brightness and colour differences](https://mlsanity.com/paper/2311.04614/) (2023)
- [Depth Estimation using Weighted-loss and Transfer Learning](https://mlsanity.com/paper/2404.07686/) (2024)
- [Multimodal Contrastive Pretraining of CBCT and IOS for Enhanced Tooth Segmentation](https://mlsanity.com/paper/2509.07923/) (2025)

## Abstract (arXiv)

The application of self-supervised learning (SSL) and Vision Transformers (ViTs) approaches demonstrates promising results in the field of 2D medical imaging, but the use of these methods on 3D volumetric images is fraught with difficulties. Standard Masked Autoencoders (MAE), which are state-of-the-art solution for 2D, have a hard time capturing three-dimensional spatial relationships, especially when 75% of tokens are discarded during pre-training. We propose BertsWin, a hybrid architecture combining full BERT-style token masking using Swin Transformer windows, to enhance spatial context learning in 3D during SSL pre-training. Unlike the classic MAE, which processes only visible areas, BertsWin introduces a complete 3D grid of tokens (masked and visible), preserving the spatial topology. And to smooth out the quadratic complexity of ViT, single-level local Swin windows are used. We introduce a structural priority loss function and evaluate the results of cone beam computed tomography of the temporomandibular joints. The subsequent assessment includes TMJ segmentation on 3D CT scans. We demonstrate that the BertsWin architecture, by maintaining a complete three-dimensional spatial topology, inherently accelerates semantic convergence by a factor of 5.8x compared to standard ViT-MAE baselines. Furthermore, when coupled with our proposed GradientConductor optimizer, the full BertsWin framework achieves a 15-fold reduction in training epochs (44 vs 660) required to reach state-of-the-art reconstruction fidelity. Analysis reveals that BertsWin achieves this acceleration without the computational penalty typically associated with dense volumetric processing. At canonical input resolutions, the architecture maintains theoretical FLOP parity with sparse ViT baselines, resulting in a significant net reduction in total computational resources due to faster convergence.