mlsanityData to 14 Oct 2026

TC-PDM: Temporally Consistent Patch Diffusion Models for Infrared-to-Visible Video Translation

arXiv 2408.14227 ↗First posted 26 Aug 2024cs.CVcited by 0

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Infrared imaging offers resilience against changing lighting conditions by capturing object temperatures. Yet, in few scenarios, its lack of visual details compared to daytime visible images, poses a significant challenge for human and machine interpretation. This paper proposes a novel diffusion method, dubbed Temporally Consistent Patch Diffusion Models (TC-DPM), for infrared-to-visible video translation. Our method, extending the Patch Diffusion Model, consists of two key components. Firstly, we propose a semantic-guided denoising, leveraging the strong representations of foundational models. As such, our method faithfully preserves the semantic structure of generated visible images. Secondly, we propose a novel temporal blending module to guide the denoising trajectory, ensuring the temporal consistency between consecutive frames. Experiment shows that TC-PDM outperforms state-of-the-art methods by 35.3% in FVD for infrared-to-visible video translation and by 6.1% in AP50 for day-to-night object detection. Our code is publicly available at https://github.com/dzungdoan6/tc-pdm

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 49 of its 66 references that are in the map. First posted on arXiv 26 Aug 2024.

Builds on

49 of 66 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Any-Size-Diffusion: Toward Efficient Text-Driven Synthesis for Any-Size HD Images31 Aug
  2. 2023DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion13 Mar
  3. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation16 Feb
  4. 2022Shape-Guided Diffusion with Inside-Outside Attention1 Dec
  5. 2022T2V-DDPM: Thermal to Visible Face Translation using Denoising Diffusion Probabilistic Models19 Sep

Other references · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  10. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  12. 2016Wide Residual Networkscited by 3,070
32 more
  1. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  2. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  3. 2018Group Normalizationcited by 1,524
  4. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  5. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  6. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  7. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  8. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 982
  9. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  10. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  11. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 431
  12. 2018Video-to-Video Synthesiscited by 390
  13. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 278
  14. 2021TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditionscited by 244
  15. 2023Implicit Diffusion Models for Continuous Super-Resolutioncited by 218
  16. 2022Restoring Vision in Adverse Weather Conditions with Patch-Based Denoising Diffusion Modelscited by 215
  17. 2019Few-shot Video-to-Video Synthesiscited by 197
  18. 2018The Contextual Loss for Image Transformation with Non-Aligned Datacited by 135
  19. 2023Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Modelscited by 133
  20. 2021The Norm Must Go On: Dynamic Unsupervised Domain Adaptation by Normalizationcited by 131
  21. 2018Geometry-Consistent Generative Adversarial Networks for One-Sided Unsupervised Domain Mappingcited by 113
  22. 2023Equivariant Multi-Modality Image Fusioncited by 99
  23. 2018Recycle-GAN: Unsupervised Video Retargetingcited by 89
  24. 2023VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimationcited by 82
  25. 2024Towards Generalizable Tumor Synthesiscited by 65
  26. 2024On the Content Bias in Fr'echet Video Distancecited by 52
  27. 2019Mocycle-GAN: Unpaired Video-to-Video Translationcited by 37
  28. 2024Hierarchical Patch Diffusion Models for High-Resolution Video Generationcited by 24
  29. 2018DA-GAN: Instance-level Image Translation by Deep Attention Generative Adversarial Networks (with Supplementary Materials)cited by 21
  30. 2021I2V-GAN: Unpaired Infrared-to-Visible Video Translationcited by 10
  31. 2022Learning Temporally and Semantically Consistent Unpaired Video-to-video Translation Through Pseudo-Supervision From Synthetic Optical Flowcited by 5
  32. 2022ROMA: Cross-Domain Region Similarity Matching for Unpaired Nighttime Infrared to Daytime Visible Video Translationcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SC-Diff: Semantically Calibrated Diffusion for Visible-to-Infrared Image Translation
  2. 2024PID: Physics-Informed Diffusion Model for Infrared Image Generation
  3. 2023MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance
  4. 2025DiffV2IR: Visible-to-Infrared Diffusion Model via Vision-Language Understanding
  5. 2024Multi-Sensor Diffusion-Driven Optical Image Translation for Large-Scale Applications
  6. 2025Inference-Time Scaling of Diffusion Models for Infrared Data Generation
  7. 2025TIR-Diffusion: Diffusion-based Thermal Infrared Image Denoising via Latent and Wavelet Domain Optimization
  8. 2024Taming Diffusion Models for Image Restoration: A Review