# Controllable Histopathology Image Synthesis with Training-free Structural Initialization and Textural Modulation

arXiv: https://arxiv.org/abs/2606.27935 · HTML: https://mlsanity.com/paper/2606.27935/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 26 Jun 2026.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Universal Guidance for Diffusion Models](https://mlsanity.com/paper/2302.07121/) (2023)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [Unsupervised Medical Image Translation with Adversarial Diffusion Models](https://mlsanity.com/paper/2207.08208/) (2022)
- [Multimodal Whole Slide Foundation Model for Pathology](https://mlsanity.com/paper/2411.19666/) (2024)
- [Semantic Image Synthesis via Diffusion Models](https://mlsanity.com/paper/2207.00050/) (2022)
- [Artificial Intelligence for Digital and Computational Pathology](https://mlsanity.com/paper/2401.06148/) (2023)
- [Zero-shot Medical Image Translation via Frequency-Guided Diffusion Models](https://mlsanity.com/paper/2304.02742/) (2023)
- [Dense Steerable Filter CNNs for Exploiting Rotational Symmetry in Histology Images](https://mlsanity.com/paper/2004.03037/) (2020)
- [Diffusion-based Data Augmentation for Nuclei Image Segmentation](https://mlsanity.com/paper/2310.14197/) (2023)
- [Decoupling Training-Free Guided Diffusion by ADMM](https://mlsanity.com/paper/2411.12773/) (2024)
- [PixCell: A generative foundation model for digital histopathology images](https://mlsanity.com/paper/2506.05127/) (2025)
- [Sharp-GAN: Sharpness Loss Regularized GAN for Histopathology Image Synthesis](https://mlsanity.com/paper/2110.14709/) (2021)
- [PathDiff: Histopathology Image Synthesis with Unpaired Text and Mask Conditions](https://mlsanity.com/paper/2506.23440/) (2025)
- [CycleDiff: Cycle Diffusion Models for Unpaired Image-to-image Translation](https://mlsanity.com/paper/2508.06625/) (2025)
- [NeuralRemaster: Phase-Preserving Diffusion for Structure-Aligned Generation](https://mlsanity.com/paper/2512.05106/) (2025)

## Abstract (arXiv)

Deep learning has demonstrated remarkable success in high-throughput histopathology image analysis. However, the performance of learning-based models critically depends on the quality and size of annotations by expert pathologists, which is a resource-intensive and time-consuming process. To address the limitations of data scarcity and annotation burden, several methods have been proposed to synthesize paired histopathology data. Nevertheless, these frameworks typically still require annotation data, albeit in reduced quantities, to impose structural constraints during training. In this work, we present CHIS, a plug-in framework that guides the sampling trajectory of a pretrained diffusion model through two key stages: structural initialization at the start and textural modulation during generation. The initial noise state is refined by fusing the phase information from a prior mask with the amplitude of Gaussian noise in the frequency domain, yielding a structurally informed starting point. During the reverse diffusion process, we adaptively modulate both coarse-grained and fine-grained textures at different wavelet decomposition levels. This enables a diffusion model pretrained solely on unlabeled images to generate outputs that align with prior structural masks while preserving the reference tissue style. We conducted extensive experiments demonstrating the superiority of CHIS in generation fidelity and its substantial benefits for downstream segmentation tasks. Code is available at https://github.com/IBIL-Code/CHIS.