mlsanityData to 7 Oct 2026

Relax, it doesn't matter how you get there: A new self-supervised approach for multi-timescale behavior analysis

arXiv 2303.08811 ↗First posted 15 Mar 2023cs.LG cs.ROcited by 4

Not on the map. Why?

Abstract (arXiv)

Natural behavior consists of dynamics that are complex and unpredictable, especially when trying to predict many steps into the future. While some success has been found in building representations of behavior under constrained or simplified task-based conditions, many of these models cannot be applied to free and naturalistic settings where behavior becomes increasingly hard to model. In this work, we develop a multi-task representation learning model for behavior that combines two novel components: (i) An action prediction objective that aims to predict the distribution of actions over future timesteps, and (ii) A multi-scale architecture that builds separate latent spaces to accommodate short- and long-term dynamics. After demonstrating the ability of the method to build representations of both local and global dynamics in realistic robots in varying environments and terrains, we apply our method to the MABe 2022 Multi-agent behavior challenge, where our model ranks 1st overall and on all global tasks, and 1st or 2nd on 7 out of 9 frame-level tasks. In all of these cases, we show that our model can build representations that capture the many different factors that drive behavior and solve a wide range of downstream tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 36 of its 70 references that are in the map. First posted on arXiv 15 Mar 2023.

Builds on

36 of 70 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  7. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  8. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
  9. 2019Contrastive Multiview Codingcited by 1,297
  10. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  11. 2019An Introduction to Variational Autoencoderscited by 997
  12. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
24 more
  1. 2020Contrastive Multi-View Representation Learning on Graphscited by 651
  2. 2021TS2Vec: Towards Universal Representation of Time Seriescited by 463
  3. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  4. 2018Markerless tracking of user-defined features with deep learningcited by 203
  5. 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 168
  6. 2018Learning to Decompose and Disentangle Representations for Video Predictioncited by 120
  7. 2018Disentangled Sequential Autoencodercited by 117
  8. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  9. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  10. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 101
  11. 2019Real or Fake? Learning to Discriminate Machine from Human Generated Textcited by 98
  12. 2018GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learningcited by 77
  13. 2018Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speechcited by 52
  14. 2018Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddingscited by 52
  15. 2016Squared Earth Mover's Distance-based Loss for Training Deep Neural Networkscited by 45
  16. 2022SimPer: Simple Self-Supervised Learning of Periodic Targetscited by 44
  17. 2021Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Predictioncited by 36
  18. 2020Task Programming: Learning Data Efficient Behavior Representationscited by 27
  19. 2021Self-Supervised Keypoint Discovery in Behavioral Videoscited by 23
  20. 2017Learning Features for Offline Handwritten Signature Verification using Deep Convolutional Neural Networkscited by 20
  21. 2022Detecting Individual Decision-Making Style: Exploring Behavioral Stylometry in Chesscited by 18
  22. 2022The no-free-lunch theorems of supervised learningcited by 16
  23. 2021Leaving Flatland: Advances in 3D behavioral measurementcited by 10
  24. 2021Learning interaction rules from multi-animal trajectories via augmented behavioral modelscited by 5

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025BaRISTA: Brain Scale Informed Spatiotemporal Representation of Human Intracranial Neural Activitycited by 7
  2. 2025Animal behavioral analysis and neural encoding with transformer-based self-supervised pretrainingcited by 5
  3. 2025Grounding Intelligence in Movementcited by 1
  4. 2025From eye to AI: studying rodent social behavior in the era of machine Learningcited by 0