# Deep Single Image Camera Calibration by Heatmap Regression to Recover Fisheye Images Under Manhattan World Assumption

arXiv: https://arxiv.org/abs/2303.17166 · HTML: https://mlsanity.com/paper/2303.17166/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 83 references that are in the map.
- First posted on arXiv 30 Mar 2023.

## Builds on (most cited first, 30 of 40)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [Convolutional Pose Machines](https://mlsanity.com/paper/1602.00134/) (2016)
- [Distribution-Aware Coordinate Representation for Human Pose Estimation](https://mlsanity.com/paper/1910.06278/) (2019)
- [Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression](https://mlsanity.com/paper/1904.07399/) (2019)
- [HoHoNet: 360 Indoor Holistic Understanding with Latent Horizontal Features](https://mlsanity.com/paper/2011.11498/) (2020)
- [CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving](https://mlsanity.com/paper/2203.07724/) (2022)
- [PanoFormer: Panorama Transformer for Indoor 360 Depth Estimation](https://mlsanity.com/paper/2203.09283/) (2022)
- [Deep Hierarchical Semantic Segmentation](https://mlsanity.com/paper/2203.14335/) (2022)
- [Pointly-Supervised Instance Segmentation](https://mlsanity.com/paper/2104.06404/) (2021)
- [CLRNet: Cross Layer Refinement Network for Lane Detection](https://mlsanity.com/paper/2203.10350/) (2022)
- [Tangent Images for Mitigating Spherical Distortion](https://mlsanity.com/paper/1912.09390/) (2019)
- [Rethinking the Heatmap Regression for Bottom-up Human Pose Estimation](https://mlsanity.com/paper/2012.15175/) (2020)
- [TLIO: Tight Learned Inertial Odometry](https://mlsanity.com/paper/2007.01867/) (2020)
- [Anchor3DLane: Learning to Regress 3D Anchors for Monocular 3D Lane Detection](https://mlsanity.com/paper/2301.02371/) (2023)
- [The StreetLearn Environment and Dataset](https://mlsanity.com/paper/1903.01292/) (2019)
- [CTRL-C: Camera calibration TRansformer with Line-Classification](https://mlsanity.com/paper/2109.02259/) (2021)
- [DirecFormer: A Directed Attention in Transformer Approach to Robust Action Recognition](https://mlsanity.com/paper/2203.10233/) (2022)
- [CONSAC: Robust Multi-Model Fitting by Conditional Sample Consensus](https://mlsanity.com/paper/2001.02643/) (2020)
- [HoliCity: A City-Scale Data Platform for Learning Holistic 3D Structures](https://mlsanity.com/paper/2008.03286/) (2020)
- [Detecting Vanishing Points using Global Image Context in a Non-Manhattan World](https://mlsanity.com/paper/1608.05684/) (2016)
- [NeurVPS: Neural Vanishing Point Scanning via Conic Convolution](https://mlsanity.com/paper/1910.06316/) (2019)
- [Multi-Modal Masked Pre-Training for Monocular Panoramic Depth Completion](https://mlsanity.com/paper/2203.09855/) (2022)

## Built on by (most cited first, 3 of 3)

- [AnyCalib: On-Manifold Learning for Model-Agnostic Single-View Camera Calibration](https://mlsanity.com/paper/2503.12701/) (2025)
- [Enhancing Soccer Camera Calibration Through Keypoint Exploitation](https://mlsanity.com/paper/2410.07401/) (2024)
- [Panoramic Distortion-Aware Tokenization for Person Detection and Localization in Overhead Fisheye Images](https://mlsanity.com/paper/2503.14228/) (2025)

## Abstract (arXiv)

A Manhattan world lying along cuboid buildings is useful for camera angle estimation. However, accurate and robust angle estimation from fisheye images in the Manhattan world has remained an open challenge because general scene images tend to lack constraints such as lines, arcs, and vanishing points. To achieve higher accuracy and robustness, we propose a learning-based calibration method that uses heatmap regression, which is similar to pose estimation using keypoints, to detect the directions of labeled image coordinates. Simultaneously, our two estimators recover the rotation and remove fisheye distortion by remapping from a general scene image. Without considering vanishing-point constraints, we find that additional points for learning-based methods can be defined. To compensate for the lack of vanishing points in images, we introduce auxiliary diagonal points that have the optimal 3D arrangement of spatial uniformity. Extensive experiments demonstrated that our method outperforms conventional methods on large-scale datasets and with off-the-shelf cameras.