# Visual Knowledge Tracing

arXiv: https://arxiv.org/abs/2207.10157 · HTML: https://mlsanity.com/paper/2207.10157/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 57 references that are in the map.
- First posted on arXiv 20 Jul 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Deep Knowledge Tracing](https://mlsanity.com/paper/1506.05908/) (2015)
- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [Learning Fine-grained Image Similarity with Deep Ranking](https://mlsanity.com/paper/1404.4661/) (2014)
- [A Self-Attentive model for Knowledge Tracing](https://mlsanity.com/paper/1907.06837/) (2019)
- [Revisiting Training Strategies and Generalization Performance in Deep Metric Learning](https://mlsanity.com/paper/2002.08473/) (2020)
- [When Does Contrastive Visual Representation Learning Work?](https://mlsanity.com/paper/2105.05837/) (2021)
- [Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing](https://mlsanity.com/paper/2002.07033/) (2020)
- [An Overview of Machine Teaching](https://mlsanity.com/paper/1801.05927/) (2018)
- [Learning Similarity Conditions Without Explicit Supervision](https://mlsanity.com/paper/1908.08589/) (2019)
- [Near-Optimally Teaching the Crowd to Classify](https://mlsanity.com/paper/1402.2092/) (2014)
- [Teaching Categories to Human Learners with Visual Explanations](https://mlsanity.com/paper/1802.06924/) (2018)
- [Enriching ImageNet with Human Similarity Judgments and Psychological Embeddings](https://mlsanity.com/paper/2011.11015/) (2020)
- [Teaching Multiple Concepts to a Forgetful Learner](https://mlsanity.com/paper/1805.08322/) (2018)
- [Transforming Neural Network Visual Representations to Predict Human Judgments of Similarity](https://mlsanity.com/paper/2010.06512/) (2020)
- [Cost-Effective HITs for Relative Similarity Comparisons](https://mlsanity.com/paper/1404.3291/) (2014)
- [Becoming the Expert - Interactive Multi-Class Machine Teaching](https://mlsanity.com/paper/1504.07575/) (2015)
- [Context Embedding Networks](https://mlsanity.com/paper/1710.01691/) (2017)

## Built on by (most cited first, 1 of 1)

- [CleverBirds: A Multiple-Choice Benchmark for Fine-grained Human Knowledge Tracing](https://mlsanity.com/paper/2511.08512/) (2025)

## Abstract (arXiv)

Each year, thousands of people learn new visual categorization tasks -- radiologists learn to recognize tumors, birdwatchers learn to distinguish similar species, and crowd workers learn how to annotate valuable data for applications like autonomous driving. As humans learn, their brain updates the visual features it extracts and attend to, which ultimately informs their final classification decisions. In this work, we propose a novel task of tracing the evolving classification behavior of human learners as they engage in challenging visual classification tasks. We propose models that jointly extract the visual features used by learners as well as predicting the classification functions they utilize. We collect three challenging new datasets from real human learners in order to evaluate the performance of different visual knowledge tracing methods. Our results show that our recurrent models are able to predict the classification behavior of human learners on three challenging medical image and species identification tasks.