mlsanityData to 7 Oct 2026

Learning Mutual Modulation for Self-Supervised Cross-Modal Super-Resolution

arXiv 2207.09156 ↗First posted 19 Jul 2022cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

Self-supervised cross-modal super-resolution (SR) can overcome the difficulty of acquiring paired training data, but is challenging because only low-resolution (LR) source and high-resolution (HR) guide images from different modalities are available. Existing methods utilize pseudo or weak supervision in LR space and thus deliver results that are blurry or not faithful to the source modality. To address this issue, we present a mutual modulation SR (MMSR) model, which tackles the task by a mutual modulation strategy, including a source-to-guide modulation and a guide-to-source modulation. In these modulations, we develop cross-domain adaptive filters to fully exploit cross-modal spatial dependency and help induce the source to emulate the resolution of the guide and induce the guide to mimic the modality characteristics of the source. Moreover, we adopt a cycle consistency constraint to train MMSR in a fully self-supervised manner. Experiments on various tasks demonstrate the state-of-the-art performance of our MMSR.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 32 of its 71 references that are in the map. First posted on arXiv 19 Jul 2022.

Builds on

32 of 71 references.

References · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  3. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  4. 2017Enhanced Deep Residual Networks for Single Image Super-Resolutioncited by 1,543
  5. 2020Plug-and-Play Image Restoration with Deep Denoiser Priorcited by 467
  6. 2021Towards Real-World Blind Face Restoration with Generative Facial Priorcited by 347
  7. 2019Residual Non-local Attention Networks for Image Restorationcited by 236
  8. 2018TDAN: Temporally Deformable Alignment Network for Video Super-Resolutioncited by 225
  9. 2021Unsupervised Degradation Representation Learning for Blind Super-Resolutioncited by 177
  10. 2018Non-Local Recurrent Network for Image Restorationcited by 160
  11. 2020Cross-domain Correspondence Learning for Exemplar-based Image Translationcited by 116
  12. 2020Domain Adaptation for Image Dehazingcited by 108
20 more
  1. 2018Efficient Video Object Segmentation via Network Modulationcited by 98
  2. 2016Learning Dense Correspondence via 3D-guided Cycle Consistencycited by 91
  3. 2020Unpaired Image Super-Resolution using Pseudo-Supervisioncited by 67
  4. 2020Unsupervised Multi-Modal Image Registration via Geometry Preserving Image-to-Image Translationcited by 66
  5. 2021Towards Fast and Accurate Real-World Depth Super-Resolution: Benchmark Dataset and Baselinecited by 61
  6. 2019Deformable Kernel Networks for Joint Image Filteringcited by 59
  7. 2021Temporal Modulation Network for Controllable Space-Time Video Super-Resolutioncited by 54
  8. 2017Joint Image Filtering with Deep Convolutional Networkscited by 52
  9. 2018Deep Network Interpolation for Continuous Imagery Effect Transitioncited by 47
  10. 2022Disentangling Light Fields for Super-Resolution and Disparity Estimationcited by 42
  11. 2019Modulating Image Restoration with Continual Levels via Adaptive Feature Modification Layerscited by 39
  12. 2021Joint Implicit Image Function for Guided Depth Super-Resolutioncited by 31
  13. 2021Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separationcited by 27
  14. 2022Learning Graph Regularisation for Guided Super-Resolutioncited by 27
  15. 2019CFSNet: Toward a Controllable Feature Space for Image Restorationcited by 25
  16. 2022Occlusion-Aware Cost Constructor for Light Field Depth Estimationcited by 23
  17. 2020Learning from Multimodal and Multitemporal Earth Observation Data for Building Damage Mappingcited by 21
  18. 2020Guided Deep Decoder: Unsupervised Image Pair Fusioncited by 20
  19. 2021Overfitting the Data: Compact Neural Video Delivery via Content-aware Feature Modulationcited by 15
  20. 2018Multimodal Sensor Fusion In Single Thermal image Super-Resolutioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 4; 2025: 1; 2026: 1.

11 papers.

  1. 2023Spherical Space Feature Decomposition for Guided Depth Map Super-Resolutioncited by 28
  2. 2023Guided Depth Map Super-resolution: A Surveycited by 23
  3. 2024RGB Guided ToF Imaging System: A Survey of Deep Learning-based Methodscited by 8
  4. 2023Spatial and Modal Optimal Transport for Fast Cross-Modal MRI Reconstructioncited by 5
  5. 2024Scene Prior Filtering for Depth Super-Resolutioncited by 5
  6. 2025Advancing 3D Gaussian Splatting Editing with Complementary and Consensus Informationcited by 2
  7. 2023Scene Structure Guidance Network: Unfolding Graph Partitioning into Pixel-Wise Feature Learningcited by 1
  8. 2023LATIS: Lambda Abstraction-based Thermal Image Super-resolutioncited by 1
  9. 2024Topological GCN for Improving Detection of Hip Landmarks from B-Mode Ultrasound Imagescited by 1
  10. 2024LapGSR: Laplacian Reconstructive Network for Guided Thermal Super-Resolutioncited by 1
  11. 2026Robust Self-Supervised Cross-Modal Super-Resolution against Real-World Misaligned Observationscited by 0