mlsanityData to 7 Oct 2026

Reconstructing the Invisible: Video Frame Restoration through Siamese Masked Conditional Variational Autoencoder

arXiv 2401.10402 ↗First posted 18 Jan 2024cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

In the domain of computer vision, the restoration of missing information in video frames is a critical challenge, particularly in applications such as autonomous driving and surveillance systems. This paper introduces the Siamese Masked Conditional Variational Autoencoder (SiamMCVAE), leveraging a siamese architecture with twin encoders based on vision transformers. This innovative design enhances the model's ability to comprehend lost content by capturing intrinsic similarities between paired frames. SiamMCVAE proficiently reconstructs missing elements in masked frames, effectively addressing issues arising from camera malfunctions through variational inferences. Experimental results robustly demonstrate the model's effectiveness in restoring missing information, thus enhancing the resilience of computer vision systems. The incorporation of Siamese Vision Transformer (SiamViT) encoders in SiamMCVAE exemplifies promising potential for addressing real-world challenges in computer vision, reinforcing the adaptability of autonomous systems in dynamic environments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 18 Jan 2024.

Builds on

30 of 43 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  12. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
18 more
  1. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,618
  2. 2016Variational Inference: A Review for Statisticianscited by 1,466
  3. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  4. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  5. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  6. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  7. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  8. 2016Colorful Image Colorizationcited by 1,113
  9. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  10. 2022Simple Baselines for Image Restorationcited by 701
  11. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  12. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  13. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  14. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  15. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  16. 2022Recurrent Video Restoration Transformer with Guided Deformable Attentioncited by 168
  17. 2019Text Classification Algorithms: A Surveycited by 163
  18. 2021DG-Labeler and DGL-MOTS Dataset: Boost the Autonomous Driving Perceptioncited by 10

Built on by

Not cited yet.