# Latent Space Is Not Flat: Rethinking Latent Structure for 3D Medical Image Synthesis

arXiv: https://arxiv.org/abs/2609.32794 · HTML: https://mlsanity.com/paper/2609.32794/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 27 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 25 of 25)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [Learning to (Learn at Test Time): RNNs with Expressive Hidden States](https://mlsanity.com/paper/2407.04620/) (2024)
- [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019)
- [MAISI: Medical AI for Synthetic Imaging](https://mlsanity.com/paper/2409.11169/) (2024)
- [Distilling Diffusion Models into Conditional GANs](https://mlsanity.com/paper/2405.05967/) (2024)
- [Towards Generalizable Tumor Synthesis](https://mlsanity.com/paper/2402.19470/) (2024)
- [Flow Matching for Medical Image Synthesis: Bridging the Gap Between Speed and Quality](https://mlsanity.com/paper/2503.00266/) (2025)
- [Boosting Latent Diffusion with Perceptual Objectives](https://mlsanity.com/paper/2411.04873/) (2024)
- [cWDM: Conditional Wavelet Diffusion Models for Cross-Modality 3D Medical Image Synthesis](https://mlsanity.com/paper/2411.17203/) (2024)
- [ViT3: Unlocking Test-Time Training in Vision](https://mlsanity.com/paper/2512.01643/) (2025)
- [LeFusion: Controllable Pathology Synthesis via Lesion-Focused Diffusion Models](https://mlsanity.com/paper/2403.14066/) (2024)
- [Learning neuroimaging models from health system-scale data](https://mlsanity.com/paper/2509.18638/) (2025)
- [Latent Diffusion Autoencoders: Toward Efficient and Meaningful Unsupervised Representation Learning in Medical Imaging](https://mlsanity.com/paper/2504.08635/) (2025)
- [WFM: 3D Wavelet Flow Matching for Ultrafast Multi-Modal MRI Synthesis](https://mlsanity.com/paper/2604.21146/) (2026)
- [Latent Diffusion Inversion Requires Understanding the Latent Space](https://mlsanity.com/paper/2511.20592/) (2025)
- [Aligning Latent Geometry for Spherical Flow Matching in Image Generation](https://mlsanity.com/paper/2605.15193/) (2026)
- [CrossFlow: One-Step Generation Across Latent and Pixel Spaces](https://mlsanity.com/paper/2606.19970/) (2026)
- [Synthesizing Accurate and Realistic T1-weighted Contrast-Enhanced MR Images using Posterior-Mean Rectified Flow](https://mlsanity.com/paper/2508.12640/) (2025)
- [The Learnability Gap in Medical Latent Diffusion](https://mlsanity.com/paper/2605.17087/) (2026)
- [MoRAE: Flow-Friendly Self-Supervised Latents for Text-to-Motion Generation](https://mlsanity.com/paper/2607.29180/) (2026)
- [Tokenizer-Generator Coupling in Medical Image Generation](https://mlsanity.com/paper/2608.07713/) (2026)
- [Flow Straight to Reality: Perceptually Consistent Flow Matching for Efficient Image Restoration](https://mlsanity.com/paper/2608.10544/) (2026)

## Abstract (arXiv)

Latent generative models make 3D medical image synthesis computationally practical by generating in a compressed space. However, we show that the common flat Euclidean assumption induced by ℓ2 objectives is imprecise: latent-space geometry is so strongly anisotropic that equal-magnitude errors can produce drastically different decoded distortions. We further find that this anisotropy has a clear feature: sensitive variation concentrates in a low-rank subspace. The dominant low-rank components capture the overall structure, encoding long-range, spatially coordinated variation while remaining resistant to local noise. Its orthogonal residual, in contrast, mainly captures local and image-specific variation. Motivated by this asymmetry, we introduce Latent Structure Flow (LSF). At each block, LSF decomposes the latent state into structure and residual, models structural changes with global context, and predicts residual variation locally while preserving a direct path for the input structure. LSF changes only the generator, leaving the frozen codec and pointwise training objective unchanged. Across cross-modality synthesis and tumor inpainting tasks, LSF outperforms all compared baselines on both global and tumor-specific metrics, demonstrating the benefit of explicitly modeling latent-space structure for 3D medical image synthesis.