# Interpreting Expert Annotation Differences in Animal Behavior

arXiv: https://arxiv.org/abs/2106.06114 · HTML: https://mlsanity.com/paper/2106.06114/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 24 references that are in the map.
- First posted on arXiv 11 Jun 2021.

## Builds on (most cited first, 9 of 9)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [Learning to Infer Graphics Programs from Hand-Drawn Images](https://mlsanity.com/paper/1707.09627/) (2017)
- [HOUDINI: Lifelong Learning as Program Synthesis](https://mlsanity.com/paper/1804.00218/) (2018)
- [Learning Differentiable Programs with Admissible Neural Heuristics](https://mlsanity.com/paper/2007.12101/) (2020)
- [An Empirical Study into Annotator Agreement, Ground Truth Estimation, and Algorithm Evaluation](https://mlsanity.com/paper/1307.0426/) (2013)
- [Simple rules for complex decisions](https://mlsanity.com/paper/1702.04690/) (2017)
- [Learning Neurosymbolic Generative Models via Program Synthesis](https://mlsanity.com/paper/1901.08565/) (2019)

## Built on by (most cited first, 1 of 1)

- [Neurosymbolic Programming for Science](https://mlsanity.com/paper/2210.05050/) (2022)

## Abstract (arXiv)

Hand-annotated data can vary due to factors such as subjective differences, intra-rater variability, and differing annotator expertise. We study annotations from different experts who labelled the same behavior classes on a set of animal behavior videos, and observe a variation in annotation styles. We propose a new method using program synthesis to help interpret annotation differences for behavior analysis. Our model selects relevant trajectory features and learns a temporal filter as part of a program, which corresponds to estimated importance an annotator places on that feature at each timestamp. Our experiments on a dataset from behavioral neuroscience demonstrate that compared to baseline approaches, our method is more accurate at capturing annotator labels and learns interpretable temporal filters. We believe that our method can lead to greater reproducibility of behavior annotations used in scientific studies. We plan to release our code.