# Uncertainty-sensitive Activity Recognition: a Reliability Benchmark and the CARING Models

arXiv: https://arxiv.org/abs/2101.00468 · HTML: https://mlsanity.com/paper/2101.00468/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 37 references that are in the map.
- First posted on arXiv 2 Jan 2021.

## Builds on (most cited first, 16 of 16)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Beyond Short Snippets: Deep Networks for Video Classification](https://mlsanity.com/paper/1503.08909/) (2015)
- [Analyzing Uncertainty in Neural Machine Translation](https://mlsanity.com/paper/1803.00047/) (2018)
- [The Limits and Potentials of Deep Learning for Robotics](https://mlsanity.com/paper/1804.06557/) (2018)
- [Informed Democracy: Voting-based Novelty Detection for Action Recognition](https://mlsanity.com/paper/1810.12819/) (2018)

## Built on by (most cited first, 5 of 5)

- [Confidence estimation of classification based on the distribution of the neural network output layer](https://mlsanity.com/paper/2210.07745/) (2022)
- [Is my Driver Observation Model Overconfident? Input-guided Calibration Networks for Reliable and Interpretable Confidence Estimates](https://mlsanity.com/paper/2204.04674/) (2022)
- [Driver Activity Classification Using Generalizable Representations from Vision-Language Models](https://mlsanity.com/paper/2404.14906/) (2024)
- [Should I take a walk? Estimating Energy Expenditure from Video Data](https://mlsanity.com/paper/2202.00712/) (2022)
- [Quantized Distillation: Optimizing Driver Activity Recognition Models for Resource-Constrained Environments](https://mlsanity.com/paper/2311.05970/) (2023)

## Abstract (arXiv)

Beyond assigning the correct class, an activity recognition model should also be able to determine, how certain it is in its predictions. We present the first study of how welthe confidence values of modern action recognition architectures indeed reflect the probability of the correct outcome and propose a learning-based approach for improving it. First, we extend two popular action recognition datasets with a reliability benchmark in form of the expected calibration error and reliability diagrams. Since our evaluation highlights that confidence values of standard action recognition architectures do not represent the uncertainty well, we introduce a new approach which learns to transform the model output into realistic confidence estimates through an additional calibration network. The main idea of our Calibrated Action Recognition with Input Guidance (CARING) model is to learn an optimal scaling parameter depending on the video representation. We compare our model with the native action recognition networks and the temperature scaling approach - a wide spread calibration method utilized in image classification. While temperature scaling alone drastically improves the reliability of the confidence values, our CARING method consistently leads to the best uncertainty estimates in all benchmark settings.