# DocVCE: Diffusion-based Visual Counterfactual Explanations for Document Image Classification

arXiv: https://arxiv.org/abs/2508.04233 · HTML: https://mlsanity.com/paper/2508.04233/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 41 of its 57 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [OCR-free Document Understanding Transformer](https://mlsanity.com/paper/2111.15664/) (2021)
- [Not Just a Black Box: Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1605.01713/) (2016)
- [Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval](https://mlsanity.com/paper/1502.07058/) (2015)
- [Characterising Bias in Compressed Models](https://mlsanity.com/paper/2010.03058/) (2020)
- [DiT: Self-supervised Pre-training for Document Image Transformer](https://mlsanity.com/paper/2203.02378/) (2022)
- [DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis](https://mlsanity.com/paper/2206.01062/) (2022)
- [Unifying Vision, Text, and Layout for Universal Document Processing](https://mlsanity.com/paper/2212.02623/) (2022)
- [Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer](https://mlsanity.com/paper/2102.09550/) (2021)
- [Explainable Deep Learning: A Field Guide for the Uninitiated](https://mlsanity.com/paper/2004.14545/) (2020)
- [Explaining in Style: Training a GAN to explain a classifier in StyleSpace](https://mlsanity.com/paper/2104.13369/) (2021)

## Abstract (arXiv)

As black-box AI-driven decision-making systems become increasingly widespread in modern document processing workflows, improving their transparency and reliability has become critical, especially in high-stakes applications where biases or spurious correlations in decision-making could lead to serious consequences. One vital component often found in such document processing workflows is document image classification, which, despite its widespread use, remains difficult to explain. While some recent works have attempted to explain the decisions of document image classification models through feature-importance maps, these maps are often difficult to interpret and fail to provide insights into the global features learned by the model. In this paper, we aim to bridge this research gap by introducing generative document counterfactuals that provide meaningful insights into the model's decision-making through actionable explanations. In particular, we propose DocVCE, a novel approach that leverages latent diffusion models in combination with classifier guidance to first generate plausible in-distribution visual counterfactual explanations, and then performs hierarchical patch-wise refinement to search for a refined counterfactual that is closest to the target factual image. We demonstrate the effectiveness of our approach through a rigorous qualitative and quantitative assessment on 3 different document classification datasets -- RVL-CDIP, Tobacco3482, and DocLayNet -- and 3 different models -- ResNet, ConvNeXt, and DiT -- using well-established evaluation criteria such as validity, closeness, and realism. To the best of the authors' knowledge, this is the first work to explore generative counterfactual explanations in document image analysis.