# Who Let The Dogs Out? Modeling Dog Behavior From Visual Data

arXiv: https://arxiv.org/abs/1803.10827 · HTML: https://mlsanity.com/paper/1803.10827/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2018.

## Built on by (most cited first, 20 of 20)

- [P3IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervision](https://mlsanity.com/paper/2205.02300/) (2022)
- [PDPP: Projected Diffusion for Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2303.14676/) (2023)
- [EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understanding](https://mlsanity.com/paper/2301.02217/) (2023)
- [Animal Avatars: Reconstructing Animatable 3D Animals from Casual Videos](https://mlsanity.com/paper/2403.17103/) (2024)
- [SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2403.01599/) (2024)
- [Event-Guided Procedure Planning from Instructional Videos with Text Supervision](https://mlsanity.com/paper/2308.08885/) (2023)
- [Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videos](https://mlsanity.com/paper/2403.02782/) (2024)
- [EgoPet: Egomotion and Interaction Data from an Animal's Perspective](https://mlsanity.com/paper/2404.09991/) (2024)
- [PlanLLM: Video Procedure Planning with Refinable Large Language Models](https://mlsanity.com/paper/2412.19139/) (2024)
- [Better artificial intelligence does not mean better models of biology](https://mlsanity.com/paper/2504.16940/) (2025)
- [RAP: Retrieval-Augmented Planner for Adaptive Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2403.18600/) (2024)
- [GeoWorld: Geometric World Models](https://mlsanity.com/paper/2602.23058/) (2026)
- [Masked Temporal Interpolation Diffusion for Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2507.03393/) (2025)
- [CI w/o TN: Context Injection without Task Name for Procedure Planning](https://mlsanity.com/paper/2402.15579/) (2024)
- [Phonetic and Lexical Discovery of a Canine Language using HuBERT](https://mlsanity.com/paper/2402.15985/) (2024)
- [Who's a Good Boy? Reinforcing Canine Behavior in Real-Time using Machine Learning](https://mlsanity.com/paper/2101.02380/) (2021)
- [Understanding in Artificial Intelligence](https://mlsanity.com/paper/2101.06573/) (2021)
- [Does My Dog ''Speak'' Like Me? The Acoustic Correlation between Pet Dogs and Their Human Owners](https://mlsanity.com/paper/2309.13085/) (2023)
- [Contrastive Energy Fields for Inference-Time Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2608.16457/) (2026)
- [Beyond Tasks: A Vision for Reproducing an Animal-like Behavioral Substrate Using Modern Robot Learning Techniques](https://mlsanity.com/paper/2609.33165/) (2026)

## Abstract (arXiv)

We introduce the task of directly modeling a visually intelligent agent. Computer vision typically focuses on solving various subtasks related to visual intelligence. We depart from this standard approach to computer vision; instead we directly model a visually intelligent agent. Our model takes visual information as input and directly predicts the actions of the agent. Toward this end we introduce DECADE, a large-scale dataset of ego-centric videos from a dog's perspective as well as her corresponding movements. Using this data we model how the dog acts and how the dog plans her movements. We show under a variety of metrics that given just visual input we can successfully model this intelligent agent in many situations. Moreover, the representation learned by our model encodes distinct information compared to representations trained on image classification, and our learned representation can generalize to other domains. In particular, we show strong results on the task of walkable surface estimation by using this dog modeling task as representation learning.