# Multiscale Vector-Quantized Variational Autoencoder for Endoscopic Image Synthesis

arXiv: https://arxiv.org/abs/2511.19578 · HTML: https://mlsanity.com/paper/2511.19578/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 24 Nov 2025.

## Builds on (most cited first, 21 of 21)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Efficient Active Learning for Image Classification and Segmentation using a Sample Selection and Conditional Generative Adversarial Network](https://mlsanity.com/paper/1806.05473/) (2018)
- [Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training](https://mlsanity.com/paper/1711.06606/) (2017)
- [CT-Realistic Lung Nodule Simulation from 3D Conditional Generative Adversarial Networks for Robust Lung Segmentation](https://mlsanity.com/paper/1806.04051/) (2018)
- [Unpaired Brain MR-to-CT Synthesis using a Structure-Constrained CycleGAN](https://mlsanity.com/paper/1809.04536/) (2018)
- [Generating Highly Realistic Images of Skin Lesions with GANs](https://mlsanity.com/paper/1809.01410/) (2018)
- [Mask-conditioned latent diffusion for generating gastrointestinal polyp images](https://mlsanity.com/paper/2304.05233/) (2023)
- [Mutually improved endoscopic image synthesis and landmark detection in unpaired image-to-image translation](https://mlsanity.com/paper/2107.06941/) (2021)
- [This Intestine Does Not Exist: Multiscale Residual Variational Autoencoder for Realistic Wireless Capsule Endoscopy Image Generation](https://mlsanity.com/paper/2302.02150/) (2023)
- [PolypConnect: Image inpainting for generating realistic gastrointestinal tract images with polyps](https://mlsanity.com/paper/2205.15413/) (2022)
- [Evaluating clinical diversity and plausibility of synthetic capsule endoscopic images](https://mlsanity.com/paper/2301.06366/) (2023)
- [Simple U-net Based Synthetic Polyp Image Generation: Polyp to Negative and Negative to Polyp](https://mlsanity.com/paper/2302.09835/) (2023)
- [Roadmap on Signal Processing for Next Generation Measurement Systems](https://mlsanity.com/paper/2111.02493/) (2021)
- [Time-based Self-supervised Learning for Wireless Capsule Endoscopy](https://mlsanity.com/paper/2204.09773/) (2022)

## Built on by (most cited first, 2 of 2)

- [Gaussian Process Prior Variational Autoencoder for Endoscopic Videos](https://mlsanity.com/paper/2606.19908/) (2026)
- [Representation-driven Endoscopic Visual Embedding Alignment for Latent Generation](https://mlsanity.com/paper/2608.07176/) (2026)

## Abstract (arXiv)

Gastrointestinal (GI) imaging via Wireless Capsule Endoscopy (WCE) generates a large number of images requiring manual screening. Deep learning-based Clinical Decision Support (CDS) systems can assist screening, yet their performance relies on the existence of large, diverse, training medical datasets. However, the scarcity of such data, due to privacy constraints and annotation costs, hinders CDS development. Generative machine learning offers a viable solution to combat this limitation. While current Synthetic Data Generation (SDG) methods, such as Generative Adversarial Networks and Variational Autoencoders have been explored, they often face challenges with training stability and capturing sufficient visual diversity, especially when synthesizing abnormal findings. This work introduces a novel VAE-based methodology for medical image synthesis and presents its application for the generation of WCE images. The novel contributions of this work include a) multiscale extension of the Vector Quantized VAE model, named as Multiscale Vector Quantized Variational Autoencoder (MSVQ-VAE); b) unlike other VAE-based SDG models for WCE image generation, MSVQ-VAE is used to seamlessly introduce abnormalities into normal WCE images; c) it enables conditional generation of synthetic images, enabling the introduction of different types of abnormalities into the normal WCE images; d) it performs experiments with a variety of abnormality types, including polyps, vascular and inflammatory conditions. The utility of the generated images for CDS is assessed via image classification. Comparative experiments demonstrate that training a CDS classifier using the abnormal images generated by the proposed methodology yield comparable results with a classifier trained with only real data. The generality of the proposed methodology promises its applicability to various domains related to medical multimedia.