# Horizontal-to-Vertical Video Conversion

arXiv: https://arxiv.org/abs/2101.04051 · HTML: https://mlsanity.com/paper/2101.04051/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 89 references that are in the map.
- First posted on arXiv 11 Jan 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [Fast Online Object Tracking and Segmentation: A Unifying Approach](https://mlsanity.com/paper/1812.05050/) (2018)
- [Cascaded Partial Decoder for Fast and Accurate Salient Object Detection](https://mlsanity.com/paper/1904.08739/) (2019)
- [The Secrets of Salient Object Segmentation](https://mlsanity.com/paper/1406.2807/) (2014)
- [Deeply supervised salient object detection with short connections](https://mlsanity.com/paper/1611.04849/) (2016)
- [Salient Object Detection in the Deep Learning Era: An In-Depth Survey](https://mlsanity.com/paper/1904.09146/) (2019)
- [Photo Aesthetics Ranking Network with Attributes and Content Adaptation](https://mlsanity.com/paper/1606.01621/) (2016)
- [DSFD: Dual Shot Face Detector](https://mlsanity.com/paper/1810.10220/) (2018)
- [SegFlow: Joint Learning for Video Object Segmentation and Optical Flow](https://mlsanity.com/paper/1709.06750/) (2017)
- [FreeAnchor: Learning to Match Anchors for Visual Object Detection](https://mlsanity.com/paper/1909.02466/) (2019)
- [FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos](https://mlsanity.com/paper/1701.05384/) (2017)
- [Learning Video Object Segmentation with Visual Memory](https://mlsanity.com/paper/1704.05737/) (2017)
- [Effective Aesthetics Prediction with Multi-level Spatially Pooled Features](https://mlsanity.com/paper/1904.01382/) (2019)
- [Anchor Diffusion for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/1910.10895/) (2019)
- [Instance-Level Salient Object Segmentation](https://mlsanity.com/paper/1704.03604/) (2017)
- [Reliable and Efficient Image Cropping: A Grid Anchor based Approach](https://mlsanity.com/paper/1904.04441/) (2019)
- [Revisiting Salient Object Detection: Simultaneous Detection, Ranking, and Subitizing of Multiple Salient Objects](https://mlsanity.com/paper/1803.05082/) (2018)
- [Grid Anchor based Image Cropping: A New Benchmark and An Efficient Model](https://mlsanity.com/paper/1909.08989/) (2019)
- [Image Cropping with Composition and Saliency Aware Aesthetic Score Map](https://mlsanity.com/paper/1911.10492/) (2019)
- [TransNet: A deep network for fast detection of common shot transitions](https://mlsanity.com/paper/1906.03363/) (2019)
- [Salient Object Subitizing](https://mlsanity.com/paper/1607.07525/) (2016)
- [Understanding and Visualizing Deep Visual Saliency Models](https://mlsanity.com/paper/1903.02501/) (2019)
- [Learning Gaze Transitions from Depth to Improve Video Saliency Estimation](https://mlsanity.com/paper/1603.03669/) (2016)
- [An End-to-End Neural Network for Image Cropping by Learning Composition from Aesthetic Photos](https://mlsanity.com/paper/1907.01432/) (2019)

## Built on by (most cited first, 2 of 2)

- [Salient Object Ranking with Position-Preserved Attention](https://mlsanity.com/paper/2106.05047/) (2021)
- [Partitioned Saliency Ranking with Dense Pyramid Transformers](https://mlsanity.com/paper/2308.00236/) (2023)

## Abstract (arXiv)

Alongside the prevalence of mobile videos, the general public leans towards consuming vertical videos on hand-held devices. To revitalize the exposure of horizontal contents, we hereby set forth the exploration of automated horizontal-to-vertical (abbreviated as H2V) video conversion with our proposed H2V framework, accompanied by an accurately annotated H2V-142K dataset. Concretely, H2V framework integrates video shot boundary detection, subject selection and multi-object tracking to facilitate the subject-preserving conversion, wherein the key is subject selection. To achieve so, we propose a Rank-SS module that detects human objects, then selects the subject-to-preserve via exploiting location, appearance, and salient cues. Afterward, the framework automatically crops the video around the subject to produce vertical contents from horizontal sources. To build and evaluate our H2V framework, H2V-142K dataset is densely annotated with subject bounding boxes for 125 videos with 132K frames and 9,500 video covers, upon which we demonstrate superior subject selection performance comparing to traditional salient approaches, and exhibit promising horizontal-to-vertical conversion performance overall. By publicizing this dataset as well as our approach, we wish to pave the way for more valuable endeavors on the horizontal-to-vertical video conversion task.