# Learning to Estimate Without Bias

arXiv: https://arxiv.org/abs/2110.12403 · HTML: https://mlsanity.com/paper/2110.12403/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 55 references that are in the map.
- First posted on arXiv 24 Oct 2021.

## Builds on (most cited first, 17 of 17)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [Environment Inference for Invariant Learning](https://mlsanity.com/paper/2010.07249/) (2020)
- [Deep Learning Techniques for Inverse Problems in Imaging](https://mlsanity.com/paper/2005.06001/) (2020)
- [Fair Regression: Quantitative Definitions and Reduction-based Algorithms](https://mlsanity.com/paper/1905.12843/) (2019)
- [Model-Based Deep Learning](https://mlsanity.com/paper/2012.08405/) (2020)
- [On Calibration and Out-of-domain Generalization](https://mlsanity.com/paper/2102.10395/) (2021)
- [Parameter Estimation with Dense and Convolutional Neural Networks Applied to the FitzHugh-Nagumo ODE](https://mlsanity.com/paper/2012.06691/) (2020)
- [Semi-supervised Deep Reinforcement Learning in Support of IoT and Smart City Services](https://mlsanity.com/paper/1810.04118/) (2018)
- [Data-driven Estimation of Sinusoid Frequencies](https://mlsanity.com/paper/1906.00823/) (2019)
- [Stochastic Deep Compressive Sensing for the Reconstruction of Diffusion Tensor Cardiac MRI](https://mlsanity.com/paper/1805.12064/) (2018)
- [Unfolded Algorithms for Deep Phase Retrieval](https://mlsanity.com/paper/2012.11102/) (2020)
- [SignalNet: A Low Resolution Sinusoid Decomposition and Estimation Network](https://mlsanity.com/paper/2106.05490/) (2021)

## Built on by (most cited first, 9 of 9)

- [Malign Overfitting: Interpolation Can Provably Preclude Invariance](https://mlsanity.com/paper/2211.15724/) (2022)
- [Leveraging Variational Autoencoders for Parameterized MMSE Estimation](https://mlsanity.com/paper/2307.05352/) (2023)
- [CFARnet: deep learning for target detection with constant false alarm rate](https://mlsanity.com/paper/2208.02474/) (2022)
- [Improving Fairness and Mitigating MADness in Generative Models](https://mlsanity.com/paper/2405.13977/) (2024)
- [Bayesian Cram'er-Rao Bound Estimation with Score-Based Models](https://mlsanity.com/paper/2309.16076/) (2023)
- [Learning minimal volume uncertainty ellipsoids](https://mlsanity.com/paper/2405.02441/) (2024)
- [Learning to Detect with Constant False Alarm Rate](https://mlsanity.com/paper/2206.05747/) (2022)
- [Fine Tuning a Simulation-Driven Estimator](https://mlsanity.com/paper/2504.04480/) (2025)
- [Bias-Reduced Neural Networks for Parameter Estimation in Quantitative MRI](https://mlsanity.com/paper/2312.11468/) (2023)

## Abstract (arXiv)

The Gauss Markov theorem states that the weighted least squares estimator is a linear minimum variance unbiased estimation (MVUE) in linear models. In this paper, we take a first step towards extending this result to non linear settings via deep learning with bias constraints. The classical approach to designing non-linear MVUEs is through maximum likelihood estimation (MLE) which often involves computationally challenging optimizations. On the other hand, deep learning methods allow for non-linear estimators with fixed computational complexity. Learning based estimators perform optimally on average with respect to their training set but may suffer from significant bias in other parameters. To avoid this, we propose to add a simple bias constraint to the loss function, resulting in an estimator we refer to as Bias Constrained Estimator (BCE). We prove that this yields asymptotic MVUEs that behave similarly to the classical MLEs and asymptotically attain the Cramer Rao bound. We demonstrate the advantages of our approach in the context of signal to noise ratio estimation as well as covariance estimation. A second motivation to BCE is in applications where multiple estimates of the same unknown are averaged for improved performance. Examples include distributed sensor networks and data augmentation in test-time. In such applications, we show that BCE leads to asymptotically consistent estimators.