# Learning Mutual Modulation for Self-Supervised Cross-Modal Super-Resolution

arXiv: https://arxiv.org/abs/2207.09156 · HTML: https://mlsanity.com/paper/2207.09156/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 71 references that are in the map.
- First posted on arXiv 19 Jul 2022.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Plug-and-Play Image Restoration with Deep Denoiser Prior](https://mlsanity.com/paper/2008.13751/) (2020)
- [Towards Real-World Blind Face Restoration with Generative Facial Prior](https://mlsanity.com/paper/2101.04061/) (2021)
- [Residual Non-local Attention Networks for Image Restoration](https://mlsanity.com/paper/1903.10082/) (2019)
- [TDAN: Temporally Deformable Alignment Network for Video Super-Resolution](https://mlsanity.com/paper/1812.02898/) (2018)
- [Unsupervised Degradation Representation Learning for Blind Super-Resolution](https://mlsanity.com/paper/2104.00416/) (2021)
- [Non-Local Recurrent Network for Image Restoration](https://mlsanity.com/paper/1806.02919/) (2018)
- [Cross-domain Correspondence Learning for Exemplar-based Image Translation](https://mlsanity.com/paper/2004.05571/) (2020)
- [Domain Adaptation for Image Dehazing](https://mlsanity.com/paper/2005.04668/) (2020)
- [Efficient Video Object Segmentation via Network Modulation](https://mlsanity.com/paper/1802.01218/) (2018)
- [Learning Dense Correspondence via 3D-guided Cycle Consistency](https://mlsanity.com/paper/1604.05383/) (2016)
- [Unpaired Image Super-Resolution using Pseudo-Supervision](https://mlsanity.com/paper/2002.11397/) (2020)
- [Unsupervised Multi-Modal Image Registration via Geometry Preserving Image-to-Image Translation](https://mlsanity.com/paper/2003.08073/) (2020)
- [Towards Fast and Accurate Real-World Depth Super-Resolution: Benchmark Dataset and Baseline](https://mlsanity.com/paper/2104.06174/) (2021)
- [Deformable Kernel Networks for Joint Image Filtering](https://mlsanity.com/paper/1910.08373/) (2019)
- [Temporal Modulation Network for Controllable Space-Time Video Super-Resolution](https://mlsanity.com/paper/2104.10642/) (2021)
- [Joint Image Filtering with Deep Convolutional Networks](https://mlsanity.com/paper/1710.04200/) (2017)
- [Deep Network Interpolation for Continuous Imagery Effect Transition](https://mlsanity.com/paper/1811.10515/) (2018)
- [Disentangling Light Fields for Super-Resolution and Disparity Estimation](https://mlsanity.com/paper/2202.10603/) (2022)
- [Modulating Image Restoration with Continual Levels via Adaptive Feature Modification Layers](https://mlsanity.com/paper/1904.08118/) (2019)
- [Joint Implicit Image Function for Guided Depth Super-Resolution](https://mlsanity.com/paper/2107.08717/) (2021)
- [Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation](https://mlsanity.com/paper/2104.02775/) (2021)
- [Learning Graph Regularisation for Guided Super-Resolution](https://mlsanity.com/paper/2203.14297/) (2022)
- [CFSNet: Toward a Controllable Feature Space for Image Restoration](https://mlsanity.com/paper/1904.00634/) (2019)
- [Occlusion-Aware Cost Constructor for Light Field Depth Estimation](https://mlsanity.com/paper/2203.01576/) (2022)
- [Learning from Multimodal and Multitemporal Earth Observation Data for Building Damage Mapping](https://mlsanity.com/paper/2009.06200/) (2020)
- [Guided Deep Decoder: Unsupervised Image Pair Fusion](https://mlsanity.com/paper/2007.11766/) (2020)

## Built on by (most cited first, 11 of 11)

- [Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution](https://mlsanity.com/paper/2303.08942/) (2023)
- [Guided Depth Map Super-resolution: A Survey](https://mlsanity.com/paper/2302.09598/) (2023)
- [RGB Guided ToF Imaging System: A Survey of Deep Learning-based Methods](https://mlsanity.com/paper/2405.10357/) (2024)
- [Spatial and Modal Optimal Transport for Fast Cross-Modal MRI Reconstruction](https://mlsanity.com/paper/2305.02774/) (2023)
- [Scene Prior Filtering for Depth Super-Resolution](https://mlsanity.com/paper/2402.13876/) (2024)
- [Advancing 3D Gaussian Splatting Editing with Complementary and Consensus Information](https://mlsanity.com/paper/2503.11601/) (2025)
- [Scene Structure Guidance Network: Unfolding Graph Partitioning into Pixel-Wise Feature Learning](https://mlsanity.com/paper/2301.00555/) (2023)
- [LATIS: Lambda Abstraction-based Thermal Image Super-resolution](https://mlsanity.com/paper/2311.12046/) (2023)
- [Topological GCN for Improving Detection of Hip Landmarks from B-Mode Ultrasound Images](https://mlsanity.com/paper/2408.13495/) (2024)
- [LapGSR: Laplacian Reconstructive Network for Guided Thermal Super-Resolution](https://mlsanity.com/paper/2411.07750/) (2024)
- [Robust Self-Supervised Cross-Modal Super-Resolution against Real-World Misaligned Observations](https://mlsanity.com/paper/2602.18822/) (2026)

## Abstract (arXiv)

Self-supervised cross-modal super-resolution (SR) can overcome the difficulty of acquiring paired training data, but is challenging because only low-resolution (LR) source and high-resolution (HR) guide images from different modalities are available. Existing methods utilize pseudo or weak supervision in LR space and thus deliver results that are blurry or not faithful to the source modality. To address this issue, we present a mutual modulation SR (MMSR) model, which tackles the task by a mutual modulation strategy, including a source-to-guide modulation and a guide-to-source modulation. In these modulations, we develop cross-domain adaptive filters to fully exploit cross-modal spatial dependency and help induce the source to emulate the resolution of the guide and induce the guide to mimic the modality characteristics of the source. Moreover, we adopt a cycle consistency constraint to train MMSR in a fully self-supervised manner. Experiments on various tasks demonstrate the state-of-the-art performance of our MMSR.