# An information-theoretic learning model based on importance sampling

arXiv: https://arxiv.org/abs/2302.04438 · HTML: https://mlsanity.com/paper/2302.04438/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 33 references that are in the map.
- First posted on arXiv 9 Feb 2023.

## Builds on (most cited first, 10 of 10)

- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [Additive Margin Softmax for Face Verification](https://mlsanity.com/paper/1801.05599/) (2018)
- [Cross-Age LFW: A Database for Studying Cross-Age Face Recognition in Unconstrained Environments](https://mlsanity.com/paper/1708.08197/) (2017)
- [Mis-classified Vector Guided Softmax Loss for Face Recognition](https://mlsanity.com/paper/1912.00833/) (2019)
- [Meta Balanced Network for Fair Face Recognition](https://mlsanity.com/paper/2205.06548/) (2022)
- [A Minimax Approach to Supervised Learning](https://mlsanity.com/paper/1606.02206/) (2016)
- [SphereFace Revived: Unifying Hyperspherical Face Recognition](https://mlsanity.com/paper/2109.05565/) (2021)
- [Large age-gap face verification by feature injection in deep networks](https://mlsanity.com/paper/1602.06149/) (2016)
- [An Information-theoretic On-line Learning Principle for Specialization in Hierarchical Decision-Making Systems](https://mlsanity.com/paper/1907.11452/) (2019)

## Abstract (arXiv)

A crucial assumption underlying the most current theory of machine learning is that the training distribution is identical to the test distribution. However, this assumption may not hold in some real-world applications. In this paper, we develop a learning model based on principles of information theory by minimizing the worst-case loss at prescribed levels of uncertainty. We reformulate the empirical estimation of the risk functional and the distribution deviation constraint based on the importance sampling method. The objective of the proposed approach is to minimize the loss under maximum degradation and hence the resulting problem is a minimax problem which can be converted to an unconstrained minimum problem using the Lagrange method with the Lagrange multiplier T. We reveal that the minimization of the objective function under logarithmic transformation is equivalent to the minimization of the p-norm loss with p=frac1T. We applied the proposed model to the face verification task on Racial Faces in the Wild datasets and showed that the proposed model performs better under large distribution deviations.