# Task Programming: Learning Data Efficient Behavior Representations

arXiv: https://arxiv.org/abs/2011.13917 · HTML: https://mlsanity.com/paper/2011.13917/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2020.

## Built on by (most cited first, 27 of 27)

- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [VideoPrism: A Foundational Visual Encoder for Video Understanding](https://mlsanity.com/paper/2402.13217/) (2024)
- [The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions](https://mlsanity.com/paper/2104.02710/) (2021)
- [Self-Supervised Keypoint Discovery in Behavioral Videos](https://mlsanity.com/paper/2112.05121/) (2021)
- [Utilizing Expert Features for Contrastive Learning of Time-Series Representations](https://mlsanity.com/paper/2206.11517/) (2022)
- [Neurosymbolic Grounding for Compositional World Models](https://mlsanity.com/paper/2310.12690/) (2023)
- [MABe22: A Multi-Species Multi-Task Benchmark for Learned Representations of Behavior](https://mlsanity.com/paper/2207.10553/) (2022)
- [AmadeusGPT: a natural language interface for interactive animal behavioral analysis](https://mlsanity.com/paper/2307.04858/) (2023)
- [Unsupervised Learning of Neurosymbolic Encoders](https://mlsanity.com/paper/2107.13132/) (2021)
- [Neurosymbolic Programming for Science](https://mlsanity.com/paper/2210.05050/) (2022)
- [VideoPro: A Visual Analytics Approach for Interactive Video Programming](https://mlsanity.com/paper/2308.00401/) (2023)
- [Automatic Synthesis of Diverse Weak Supervision Sources for Behavior Analysis](https://mlsanity.com/paper/2111.15186/) (2021)
- [SciVid: Cross-Domain Evaluation of Video Models in Scientific Applications](https://mlsanity.com/paper/2507.03578/) (2025)
- [Animal behavioral analysis and neural encoding with transformer-based self-supervised pretraining](https://mlsanity.com/paper/2507.09513/) (2025)
- [Open-Source Tools for Behavioral Video Analysis: Setup, Methods, and Development](https://mlsanity.com/paper/2204.02842/) (2022)
- [Learning Behavior Representations Through Multi-Timescale Bootstrapping](https://mlsanity.com/paper/2206.07041/) (2022)
- [Relax, it doesn't matter how you get there: A new self-supervised approach for multi-timescale behavior analysis](https://mlsanity.com/paper/2303.08811/) (2023)
- [Pre-trained Large Language Models Learn Hidden Markov Models In-context](https://mlsanity.com/paper/2506.07298/) (2025)
- [Learning Keypoints for Multi-Agent Behavior Analysis using Self-Supervision](https://mlsanity.com/paper/2409.09455/) (2024)
- [NeuroQL: A Neuro-Symbolic Language and Dataset for Inter-Subjective Reasoning](https://mlsanity.com/paper/2303.07146/) (2023)
- [A study of animal action segmentation algorithms across supervised, unsupervised, and semi-supervised learning paradigms](https://mlsanity.com/paper/2407.16727/) (2024)
- [Overcoming the Domain Gap in Contrastive Learning of Neural Action Representations](https://mlsanity.com/paper/2111.14595/) (2021)
- [Overcoming the Domain Gap in Neural Action Representations](https://mlsanity.com/paper/2112.01176/) (2021)
- [Learning signatures of decision making from many individuals playing the same game](https://mlsanity.com/paper/2302.11023/) (2023)
- [Building Socially-Equitable Public Models](https://mlsanity.com/paper/2406.02790/) (2024)
- [Mouse Lockbox Dataset: Behavior Recognition for Mice Solving Lockboxes](https://mlsanity.com/paper/2505.15408/) (2025)
- [From eye to AI: studying rodent social behavior in the era of machine Learning](https://mlsanity.com/paper/2508.04255/) (2025)

## Abstract (arXiv)

Specialized domain knowledge is often necessary to accurately annotate training sets for in-depth analysis, but can be burdensome and time-consuming to acquire from domain experts. This issue arises prominently in automated behavior analysis, in which agent movements or actions of interest are detected from video tracking data. To reduce annotation effort, we present TREBA: a method to learn annotation-sample efficient trajectory embedding for behavior analysis, based on multi-task self-supervised learning. The tasks in our method can be efficiently engineered by domain experts through a process we call "task programming", which uses programs to explicitly encode structured knowledge from domain experts. Total domain expert effort can be reduced by exchanging data annotation time for the construction of a small number of programmed tasks. We evaluate this trade-off using data from behavioral neuroscience, in which specialized domain knowledge is used to identify behaviors. We present experimental results in three datasets across two domains: mice and fruit flies. Using embeddings from TREBA, we reduce annotation burden by up to a factor of 10 without compromising accuracy compared to state-of-the-art features. Our results thus suggest that task programming and self-supervision can be an effective way to reduce annotation effort for domain experts.