# HEMIT: H&E to Multiplex-immunohistochemistry Image Translation with Dual-Branch Pix2pix Generator

arXiv: https://arxiv.org/abs/2403.18501 · HTML: https://mlsanity.com/paper/2403.18501/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 37 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Builds on (most cited first, 14 of 14)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [DRIT++: Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1905.01270/) (2019)
- [Cell Detection with Star-convex Polygons](https://mlsanity.com/paper/1806.03535/) (2018)
- [Breaking the Dilemma of Medical Image-to-image Translation](https://mlsanity.com/paper/2110.06465/) (2021)
- [Deep learning-based virtual histology staining using auto-fluorescence of label-free tissue](https://mlsanity.com/paper/1803.11293/) (2018)
- [BCI: Breast Cancer Immunohistochemical Image Generation through Pyramid Pix2pix](https://mlsanity.com/paper/2204.11425/) (2022)
- [Adaptive Supervised PatchNCE Loss for Learning H&E-to-IHC Stain Translation with Inconsistent Groundtruth Image Pairs](https://mlsanity.com/paper/2303.06193/) (2023)
- [GANs N' Roses: Stable, Controllable, Diverse Image to Image Translation (works for videos too!)](https://mlsanity.com/paper/2106.06561/) (2021)
- [Simultaneous Nuclear Instance and Layer Segmentation in Oral Epithelial Dysplasia](https://mlsanity.com/paper/2108.13904/) (2021)
- [HEROHE Challenge: assessing HER2 status in breast cancer without immunohistochemistry or in situ hybridization](https://mlsanity.com/paper/2111.04738/) (2021)

## Built on by (most cited first, 7 of 7)

- [MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models](https://mlsanity.com/paper/2505.10294/) (2025)
- [Generative AI for Misalignment-Resistant Virtual Staining to Accelerate Histopathology Workflows](https://mlsanity.com/paper/2509.14119/) (2025)
- [IMPLICITSTAINER: Resolution Agnostic Data-Efficient Virtual Staining Using Neural Implicit Functions](https://mlsanity.com/paper/2505.09831/) (2025)
- [Deep histological synthesis from mass spectrometry imaging for multimodal registration](https://mlsanity.com/paper/2506.05441/) (2025)
- [DPC-QA Net: A No-Reference Dual-Stream Perceptual and Cellular Quality Assessment Network for Histopathology Images](https://mlsanity.com/paper/2509.15802/) (2025)
- [Building Trust in Virtual Immunohistochemistry: Automated Assessment of Image Quality](https://mlsanity.com/paper/2511.04615/) (2025)
- [Predicting Immune Biomarkers with MultiModal Mixture-of-Expert Pathology Foundation Models Empowers Precision Oncology](https://mlsanity.com/paper/2606.18123/) (2026)

## Abstract (arXiv)

Computational analysis of multiplexed immunofluorescence histology data is emerging as an important method for understanding the tumour micro-environment in cancer. This work presents HEMIT, a dataset designed for translating Hematoxylin and Eosin (H&E) sections to multiplex-immunohistochemistry (mIHC) images, featuring DAPI, CD3, and panCK markers. Distinctively, HEMIT's mIHC images are multi-component and cellular-level aligned with H&E, enriching supervised stain translation tasks. To our knowledge, HEMIT is the first publicly available cellular-level aligned dataset that enables H&E to multi-target mIHC image translation. This dataset provides the computer vision community with a valuable resource to develop novel computational methods which have the potential to gain new insights from H&E slide archives. We also propose a new dual-branch generator architecture, using residual Convolutional Neural Networks (CNNs) and Swin Transformers which achieves better translation outcomes than other popular algorithms. When evaluated on HEMIT, it outperforms pix2pixHD, pix2pix, U-Net, and ResNet, achieving the highest overall score on key metrics including the Structural Similarity Index Measure (SSIM), Pearson correlation score (R), and Peak signal-to-noise Ratio (PSNR). Additionally, downstream analysis has been used to further validate the quality of the generated mIHC images. These results set a new benchmark in the field of stain translation tasks.