# Interpreting Low-level Vision Models with Causal Effect Maps

arXiv: https://arxiv.org/abs/2407.19789 · HTML: https://mlsanity.com/paper/2407.19789/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 62 references that are in the map.
- First posted on arXiv 29 Jul 2024.

## Builds on (most cited first, 30 of 42)

- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [Multi-Stage Progressive Image Restoration](https://mlsanity.com/paper/2102.02808/) (2021)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising](https://mlsanity.com/paper/1710.04026/) (2017)
- [Activating More Pixels in Image Super-Resolution Transformer](https://mlsanity.com/paper/2205.04437/) (2022)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [Progressive Image Deraining Networks: A Better and Simpler Baseline](https://mlsanity.com/paper/1901.09221/) (2019)
- [Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network](https://mlsanity.com/paper/1803.08664/) (2018)
- [Understanding Deep Networks via Extremal Perturbations and Smooth Masks](https://mlsanity.com/paper/1910.08485/) (2019)
- [DiffIR: Efficient Diffusion Model for Image Restoration](https://mlsanity.com/paper/2303.09472/) (2023)
- [HINet: Half Instance Normalization Network for Image Restoration](https://mlsanity.com/paper/2105.06086/) (2021)
- [Residual Non-local Attention Networks for Image Restoration](https://mlsanity.com/paper/1903.10082/) (2019)
- [A Survey on Causal Inference](https://mlsanity.com/paper/2002.02770/) (2020)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [Real Image Denoising with Feature Attention](https://mlsanity.com/paper/1904.07396/) (2019)
- [Interpreting Super-Resolution Networks with Local Attribution Maps](https://mlsanity.com/paper/2011.11036/) (2020)
- [Interventional Few-Shot Learning](https://mlsanity.com/paper/2009.13000/) (2020)
- [CXPlain: Causal Explanations for Model Interpretation under Uncertainty](https://mlsanity.com/paper/1910.12336/) (2019)
- [VmambaIR: Visual State Space Model for Image Restoration](https://mlsanity.com/paper/2403.11423/) (2024)
- [Efficient Image Super-Resolution Using Pixel Attention](https://mlsanity.com/paper/2010.01073/) (2020)
- [Practical Blind Image Denoising via Swin-Conv-UNet and Data Synthesis](https://mlsanity.com/paper/2203.13278/) (2022)
- [External Validity: From Do-Calculus to Transportability Across Populations](https://mlsanity.com/paper/1503.01603/) (2015)
- [Two Causal Principles for Improving Visual Dialog](https://mlsanity.com/paper/1911.10496/) (2019)

## Built on by (most cited first, 8 of 8)

- [Revisiting the Generalization Problem of Low-level Vision Models Through the Lens of Image Deraining](https://mlsanity.com/paper/2502.12600/) (2025)
- [Position: Agentic Systems Constitute a Key Component of Next-Generation Intelligent Image Processing](https://mlsanity.com/paper/2505.16007/) (2025)
- [Knowledge Transfer from Interaction Learning](https://mlsanity.com/paper/2509.18733/) (2025)
- [How far have we gone in Generative Image Restoration? A study on its capability, limitations and evaluation practices](https://mlsanity.com/paper/2603.05010/) (2026)
- [CIVQLLIE: Causal Intervention with Vector Quantization for Low-Light Image Enhancement](https://mlsanity.com/paper/2508.03338/) (2025)
- [RAM++: Robust Representation Learning via Adaptive Mask for All-in-One Image Restoration](https://mlsanity.com/paper/2509.12039/) (2025)
- [LogicIR: Logic Gate Networks for Image Restoration](https://mlsanity.com/paper/2606.26609/) (2026)
- [HIMOSA: Efficient Remote Sensing Image Super-Resolution with Hierarchical Mixture of Sparse Attention](https://mlsanity.com/paper/2512.00275/) (2025)

## Abstract (arXiv)

Deep neural networks have significantly improved the performance of low-level vision tasks but also increased the difficulty of interpretability. A deep understanding of deep models is beneficial for both network design and practical reliability. To take up this challenge, we introduce causality theory to interpret low-level vision models and propose a model-/task-agnostic method called Causal Effect Map (CEM). With CEM, we can visualize and quantify the input-output relationships on either positive or negative effects. After analyzing various low-level vision tasks with CEM, we have reached several interesting insights, such as: (1) Using more information of input images (e.g., larger receptive field) does NOT always yield positive outcomes. (2) Attempting to incorporate mechanisms with a global receptive field (e.g., channel attention) into image denoising may prove futile. (3) Integrating multiple tasks to train a general model could encourage the network to prioritize local information over global context. Based on the causal effect theory, the proposed diagnostic tool can refresh our common knowledge and bring a deeper understanding of low-level vision models. Codes are available at https://github.com/J-FHu/CEM.