# SimFIR: A Simple Framework for Fisheye Image Rectification with Self-supervised Representation Learning

arXiv: https://arxiv.org/abs/2308.09040 · HTML: https://mlsanity.com/paper/2308.09040/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 63 references that are in the map.
- First posted on arXiv 17 Aug 2023.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [SynDistNet: Self-Supervised Monocular Fisheye Camera Distance Estimation Synergized with Semantic Segmentation for Autonomous Driving](https://mlsanity.com/paper/2008.04017/) (2020)
- [Document Rectification and Illumination Correction using a Patch-based CNN](https://mlsanity.com/paper/1909.09470/) (2019)
- [FishEyeRecNet: A Multi-Context Collaborative Deep Network for Fisheye Image Rectification](https://mlsanity.com/paper/1804.04784/) (2018)
- [Blind Geometric Distortion Correction on Images Through Deep Learning](https://mlsanity.com/paper/1909.03459/) (2019)
- [Geometric Representation Learning for Document Image Rectification](https://mlsanity.com/paper/2210.08161/) (2022)
- [DocScanner: Robust Document Image Rectification with Progressive Learning](https://mlsanity.com/paper/2110.14968/) (2021)
- [Learning to Calibrate Straight Lines for Fisheye Image Rectification](https://mlsanity.com/paper/1904.09856/) (2019)
- [Progressively Complementary Network for Fisheye Image Rectification Using Appearance Flow](https://mlsanity.com/paper/2103.16026/) (2021)
- [Dewarping Document Image By Displacement Flow Estimation with Fully Convolutional Network](https://mlsanity.com/paper/2104.06815/) (2021)
- [Deep Unrestricted Document Image Rectification](https://mlsanity.com/paper/2304.08796/) (2023)
- [3D Object Detection from a Single Fisheye Image Without a Single Fisheye Training Image](https://mlsanity.com/paper/2003.03759/) (2020)
- [Wide-angle Image Rectification: A Survey](https://mlsanity.com/paper/2011.12108/) (2020)
- [Integrating Graph Partitioning and Matching for Trajectory Analysis in Video Surveillance](https://mlsanity.com/paper/1502.00377/) (2015)
- [SIR: Self-supervised Image Rectification via Seeing the Same Scene from Multiple Different Lenses](https://mlsanity.com/paper/2011.14611/) (2020)

## Built on by (most cited first, 13 of 13)

- [Depth Any Camera: Zero-Shot Metric Depth Estimation from Any Camera](https://mlsanity.com/paper/2501.02464/) (2025)
- [Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokens](https://mlsanity.com/paper/2508.04928/) (2025)
- [MOWA: Multiple-in-One Image Warping Model](https://mlsanity.com/paper/2404.10716/) (2024)
- [UniDAC: Universal Metric Depth Estimation for Any Camera](https://mlsanity.com/paper/2603.27105/) (2026)
- [Generalized Lightness Adaptation with Channel Selective Normalization](https://mlsanity.com/paper/2308.13783/) (2023)
- [A Comprehensive Overview of Fish-Eye Camera Distortion Correction Methods](https://mlsanity.com/paper/2401.00442/) (2023)
- [FishBEV: Distortion-Resilient Bird's Eye View Segmentation with Surround-View Fisheye Cameras](https://mlsanity.com/paper/2509.13681/) (2025)
- [AlignDiff: Learning Physically-Grounded Camera Alignment via Diffusion](https://mlsanity.com/paper/2503.21581/) (2025)
- [DarSwin-Unet: Distortion Aware Encoder-Decoder Architecture](https://mlsanity.com/paper/2407.17328/) (2024)
- [QueryCDR: Query-Based Controllable Distortion Rectification Network for Fisheye Images](https://mlsanity.com/paper/2412.13496/) (2024)
- [Simulating Refractive Distortions and Weather-Induced Artifacts for Resource-Constrained Autonomous Perception](https://mlsanity.com/paper/2507.05536/) (2025)
- [DepthAgent: Towards Better Universal Depth Estimation via Sample-wise Expert Selection](https://mlsanity.com/paper/2605.23281/) (2026)
- [OmniPoint: Universal Monocular Metric Pointcloud from Any Camera](https://mlsanity.com/paper/2609.09394/) (2026)

## Abstract (arXiv)

In fisheye images, rich distinct distortion patterns are regularly distributed in the image plane. These distortion patterns are independent of the visual content and provide informative cues for rectification. To make the best of such rectification cues, we introduce SimFIR, a simple framework for fisheye image rectification based on self-supervised representation learning. Technically, we first split a fisheye image into multiple patches and extract their representations with a Vision Transformer (ViT). To learn fine-grained distortion representations, we then associate different image patches with their specific distortion patterns based on the fisheye model, and further subtly design an innovative unified distortion-aware pretext task for their learning. The transfer performance on the downstream rectification task is remarkably boosted, which verifies the effectiveness of the learned representations. Extensive experiments are conducted, and the quantitative and qualitative results demonstrate the superiority of our method over the state-of-the-art algorithms as well as its strong generalization ability on real-world fisheye images.