# HSTR-Net: Reference Based Video Super-resolution with Dual Cameras

arXiv: https://arxiv.org/abs/2310.12092 · HTML: https://mlsanity.com/paper/2310.12092/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 58 references that are in the map.
- First posted on arXiv 18 Oct 2023.

## Builds on (most cited first, 29 of 29)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Accurate Image Super-Resolution Using Very Deep Convolutional Networks](https://mlsanity.com/paper/1511.04587/) (2015)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks](https://mlsanity.com/paper/1506.05751/) (2015)
- [Real-Time Intermediate Flow Estimation for Video Frame Interpolation](https://mlsanity.com/paper/2011.06294/) (2020)
- [AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation](https://mlsanity.com/paper/2304.09790/) (2023)
- [Image Super-Resolution by Neural Texture Transfer](https://mlsanity.com/paper/1903.00834/) (2019)
- [CrossNet: An End-to-end Reference-based Super Resolution Network using Cross-scale Warping](https://mlsanity.com/paper/1807.10547/) (2018)
- [MASA-SR: Matching Acceleration and Spatial Adaptation for Reference-Based Image Super-Resolution](https://mlsanity.com/paper/2106.02299/) (2021)
- [Robust Reference-based Super-Resolution via C2-Matching](https://mlsanity.com/paper/2106.01863/) (2021)
- [Reference-based Image Super-Resolution with Deformable Attention Transformer](https://mlsanity.com/paper/2207.11938/) (2022)
- [A Unified Pyramid Recurrent Network for Video Frame Interpolation](https://mlsanity.com/paper/2211.03456/) (2022)
- [Dual-Camera Super-Resolution with Aligned Attention Modules](https://mlsanity.com/paper/2109.01349/) (2021)
- [Reference-based Video Super-Resolution Using Multi-Camera Video Triplets](https://mlsanity.com/paper/2203.14537/) (2022)
- [Coarse-to-Fine Embedded PatchMatch and Multi-Scale Dynamic Aggregation for Reference-based Super-Resolution](https://mlsanity.com/paper/2201.04358/) (2022)
- [TTVFI: Learning Trajectory-Aware Transformer for Video Frame Interpolation](https://mlsanity.com/paper/2207.09048/) (2022)
- [Optical Flow Reusing for High-Efficiency Space-Time Video Super Resolution](https://mlsanity.com/paper/2110.06786/) (2021)
- [Video Frame Interpolation with Densely Queried Bilateral Correlation](https://mlsanity.com/paper/2304.13596/) (2023)
- [A Dual Camera System for High Spatiotemporal Resolution Video Acquisition](https://mlsanity.com/paper/1909.13051/) (2019)
- [H-VFI: Hierarchical Frame Interpolation for Videos with Large Motions](https://mlsanity.com/paper/2211.11309/) (2022)
- [Flow Guidance Deformable Compensation Network for Video Frame Interpolation](https://mlsanity.com/paper/2211.12117/) (2022)
- [A Feature Reuse Framework with Texture-adaptive Aggregation for Reference-based Super-Resolution](https://mlsanity.com/paper/2306.01500/) (2023)

## Abstract (arXiv)

High-spatio-temporal resolution (HSTR) video recording plays a crucial role in enhancing various imagery tasks that require fine-detailed information. State-of-the-art cameras provide this required high frame-rate and high spatial resolution together, albeit at a high cost. To alleviate this issue, this paper proposes a dual camera system for the generation of HSTR video using reference-based super-resolution (RefSR). One camera captures high spatial resolution low frame rate (HSLF) video while the other captures low spatial resolution high frame rate (LSHF) video simultaneously for the same scene. A novel deep learning architecture is proposed to fuse HSLF and LSHF video feeds and synthesize HSTR video frames. The proposed model combines optical flow estimation and (channel-wise and spatial) attention mechanisms to capture the fine motion and complex dependencies between frames of the two video feeds. Simulations show that the proposed model provides significant improvement over existing reference-based SR techniques in terms of PSNR and SSIM metrics. The method also exhibits sufficient frames per second (FPS) for aerial monitoring when deployed on a power-constrained drone equipped with dual cameras.