# DIME-Net: Neural Network-Based Dynamic Intrinsic Parameter Rectification for Cameras with Optical Image Stabilization System

arXiv: https://arxiv.org/abs/2303.11307 · HTML: https://mlsanity.com/paper/2303.11307/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 35 references that are in the map.
- First posted on arXiv 20 Mar 2023.

## Builds on (most cited first, 12 of 12)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Generative Visual Manipulation on the Natural Image Manifold](https://mlsanity.com/paper/1609.03552/) (2016)
- [Geometric Loss Functions for Camera Pose Regression with Deep Learning](https://mlsanity.com/paper/1704.00390/) (2017)
- [Geometry-Aware Learning of Maps for Camera Localization](https://mlsanity.com/paper/1712.03342/) (2017)
- [TransMVSNet: Global Context-aware Multi-view Stereo Network with Transformers](https://mlsanity.com/paper/2111.14600/) (2021)
- [POCO: Point Convolution for Surface Reconstruction](https://mlsanity.com/paper/2201.01831/) (2022)
- [Pose-NDF: Modeling Human Pose Manifolds with Neural Distance Fields](https://mlsanity.com/paper/2207.13807/) (2022)
- [Moser Flow: Divergence-based Generative Modeling on Manifolds](https://mlsanity.com/paper/2108.08052/) (2021)
- [End-to-End Learnable Geometric Vision by Backpropagating PnP Optimization](https://mlsanity.com/paper/1909.06043/) (2019)
- [Solving the Blind Perspective-n-Point Problem End-To-End With Robust Differentiable Geometric Optimization](https://mlsanity.com/paper/2007.14628/) (2020)
- [DiffPoseNet: Direct Differentiable Camera Pose Estimation](https://mlsanity.com/paper/2203.11174/) (2022)
- [DeepFusion: Real-Time Dense 3D Reconstruction for Monocular SLAM using Single-View Depth and Gradient Predictions](https://mlsanity.com/paper/2207.12244/) (2022)

## Abstract (arXiv)

Optical Image Stabilization (OIS) system in mobile devices reduces image blurring by steering lens to compensate for hand jitters. However, OIS changes intrinsic camera parameters (i.e. K matrix) dynamically which hinders accurate camera pose estimation or 3D reconstruction. Here we propose a novel neural network-based approach that estimates K matrix in real-time so that pose estimation or scene reconstruction can be run at camera native resolution for the highest accuracy on mobile devices. Our network design takes gratified projection model discrepancy feature and 3D point positions as inputs and employs a Multi-Layer Perceptron (MLP) to approximate fK manifold. We also design a unique training scheme for this network by introducing a Back propagated PnP (BPnP) layer so that reprojection error can be adopted as the loss function. The training process utilizes precise calibration patterns for capturing accurate fK manifold but the trained network can be used anywhere. We name the proposed Dynamic Intrinsic Manifold Estimation network as DIME-Net and have it implemented and tested on three different mobile devices. In all cases, DIME-Net can reduce reprojection error by at least 64% indicating that our design is successful.