mlsanityData to 14 Oct 2026

SHMoAReg: Spark Deformable Image Registration via Spatial Heterogeneous Mixture of Experts and Attention Heads

arXiv 2509.20073 ↗First posted 24 Sep 2025cs.CVcited by 0

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Encoder-Decoder architectures are widely used in deep learning-based Deformable Image Registration (DIR), where the encoder extracts multi-scale features and the decoder predicts deformation fields by recovering spatial locations. However, current methods lack specialized extraction of features (that are useful for registration) and predict deformation jointly and homogeneously in all three directions. In this paper, we propose a novel expert-guided DIR network with Mixture of Experts (MoE) mechanism applied in both encoder and decoder, named SHMoAReg. Specifically, we incorporate Mixture of Attention heads (MoA) into encoder layers, while Spatial Heterogeneous Mixture of Experts (SHMoE) into the decoder layers. The MoA enhances the specialization of feature extraction by dynamically selecting the optimal combination of attention heads for each image token. Meanwhile, the SHMoE predicts deformation fields heterogeneously in three directions for each voxel using experts with varying kernel sizes. Extensive experiments conducted on two publicly available datasets show consistent improvements over various methods, with a notable increase from 60.58% to 65.58% in Dice score for the abdominal CT dataset. Furthermore, SHMoAReg enhances model interpretability by differentiating experts' utilities across/within different resolution layers. To the best of our knowledge, we are the first to introduce MoE mechanism into DIR tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 33 of its 50 references that are in the map. First posted on arXiv 24 Sep 2025.

Builds on

33 of 50 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Robust Mixture-of-Expert Training for Convolutional Neural Networks19 Aug
  2. 2021Go Wider Instead of Deeper25 Jul
  3. 2021Scaling Vision with Sparse Mixture of Experts10 Jun
  4. 2020Biased Mixtures Of Experts: Enabling Computer Vision Inference Under Data Transfer Limitations21 Aug
  5. 2017Hard Mixtures of Experts for Large Scale Weakly Supervised Vision20 Apr

Other references · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  7. 2018VoxelMorph: A Learning Framework for Deformable Medical Image Registrationcited by 662
  8. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  9. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  10. 2020Large Deformation Diffeomorphic Image Registration with Laplacian Pyramid Networkscited by 136
  11. 2021ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registrationcited by 87
  12. 2022Mixture of Attention Heads: Selecting Attention Heads Per Tokencited by 60
16 more
  1. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 57
  2. 2016Network of Experts for Large-Scale Image Categorizationcited by 54
  3. 2019Dual-Stream Pyramid Registration Networkcited by 52
  4. 2022XMorpher: Full Transformer for Deformable Medical Image Registration via Cross Attentioncited by 46
  5. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Modelscited by 42
  6. 2023Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networkscited by 35
  7. 2023How Does Attention Work in Vision Transformers? A Visual Analytics Attemptcited by 30
  8. 2024M4oE: A Foundation Model for Medical Multimodal Image Segmentation with Mixture of Expertscited by 26
  9. 2023ModeT: Learning Deformable Image Registration via Motion Decomposition Transformercited by 23
  10. 2021Joint Progressive and Coarse-to-fine Registration of Brain MRI via Deformation Field Integration and Non-Rigid Feature Fusioncited by 22
  11. 2023Non-iterative Coarse-to-fine Transformer Networks for Joint Affine and Deformable Image Registrationcited by 21
  12. 2022Non-iterative Coarse-to-fine Registration based on Single-pass Deep Cumulative Learningcited by 20
  13. 2024Low-Rank Mixture-of-Experts for Continual Medical Image Segmentationcited by 16
  14. 2022Spatial Mixture-of-Expertscited by 14
  15. 2024SAM-Med3D-MoE: Towards a Non-Forgetting Segment Anything Model via Mixture of Experts for 3D Medical Image Segmentationcited by 12
  16. 2022Patcher: Patch Transformers with Mixture of Experts for Precise Medical Image Segmentationcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024LDM-Morph: Latent diffusion model guided deformable image registration
  2. 2025MorphSeek: Fine-grained Latent Representation-Level Policy Optimization for Deformable Image Registration
  3. 2026Interpretable Unsupervised Deformable Image Registration via Confidence-bound Multi-Hop Visual Reasoning
  4. 2021DiffuseMorph: Unsupervised Deformable Image Registration Using Diffusion Model
  5. 2024VMambaMorph: a Multi-Modality Deformable Image Registration Framework based on Visual State Space Model with Cross-Scan Module
  6. 2025Disentangling Progress in Medical Image Registration: Beyond Trend-Driven Architectures towards Domain-Specific Strategies
  7. 2023FSDiffReg: Feature-wise and Score-wise Diffusion-guided Unsupervised Deformable Image Registration for Cardiac Images
  8. 2024MambaReg: Mamba-Based Disentangled Convolutional Sparse Coding for Unsupervised Deformable Multi-Modal Image Registration