# RTify: Aligning Deep Neural Networks with Human Behavioral Decisions

arXiv: https://arxiv.org/abs/2411.03630 · HTML: https://mlsanity.com/paper/2411.03630/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 55 references that are in the map.
- First posted on arXiv 6 Nov 2024.

## Builds on (most cited first, 11 of 11)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Trustworthy AI: From Principles to Practices](https://mlsanity.com/paper/2110.01167/) (2021)
- [Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs](https://mlsanity.com/paper/1909.06161/) (2019)
- [Harmonizing the object recognition strategies of deep neural networks with humans](https://mlsanity.com/paper/2211.04533/) (2022)
- [Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex](https://mlsanity.com/paper/2306.03779/) (2023)
- [Deep Networks Can Resemble Human Feed-forward Vision in Invariant Object Recognition](https://mlsanity.com/paper/1508.03929/) (2015)
- [Are Deep Neural Networks Adequate Behavioural Models of Human Visual Perception?](https://mlsanity.com/paper/2305.17023/) (2023)
- [How Deep is the Feature Analysis underlying Rapid Visual Categorization?](https://mlsanity.com/paper/1606.01167/) (2016)
- [Computing a human-like reaction time metric from stable recurrent vision models](https://mlsanity.com/paper/2306.11582/) (2023)
- [CORNET: Learning Table Formatting Rules By Example](https://mlsanity.com/paper/2208.06032/) (2022)

## Built on by (most cited first, 3 of 3)

- [Better artificial intelligence does not mean better models of biology](https://mlsanity.com/paper/2504.16940/) (2025)
- [Alignment and Adversarial Robustness: Are More Human-Like Models More Secure?](https://mlsanity.com/paper/2502.12377/) (2025)
- [Inferring response times of perceptual decisions with Poisson variational autoencoders](https://mlsanity.com/paper/2511.11480/) (2025)

## Abstract (arXiv)

Current neural network models of primate vision focus on replicating overall levels of behavioral accuracy, often neglecting perceptual decisions' rich, dynamic nature. Here, we introduce a novel computational framework to model the dynamics of human behavioral choices by learning to align the temporal dynamics of a recurrent neural network (RNN) to human reaction times (RTs). We describe an approximation that allows us to constrain the number of time steps an RNN takes to solve a task with human RTs. The approach is extensively evaluated against various psychophysics experiments. We also show that the approximation can be used to optimize an "ideal-observer" RNN model to achieve an optimal tradeoff between speed and accuracy without human data. The resulting model is found to account well for human RT data. Finally, we use the approximation to train a deep learning implementation of the popular Wong-Wang decision-making model. The model is integrated with a convolutional neural network (CNN) model of visual processing and evaluated using both artificial and natural image stimuli. Overall, we present a novel framework that helps align current vision models with human behavior, bringing us closer to an integrated model of human vision.