mlsanityData to 7 Oct 2026

Overcoming the Domain Gap in Neural Action Representations

arXiv 2112.01176 ↗First posted 2 Dec 2021cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Relating animal behaviors to brain activity is a fundamental goal in neuroscience, with practical applications in building robust brain-machine interfaces. However, the domain gap between individuals is a major issue that prevents the training of general models that work on unlabeled subjects. Since 3D pose data can now be reliably extracted from multi-view video sequences without manual intervention, we propose to use it to guide the encoding of neural action representations together with a set of neural and behavioral augmentations exploiting the properties of microscopy imaging. To reduce the domain gap, during training, we swap neural and behavioral data across animals that seem to be performing similar actions. To demonstrate this, we test our methods on three very different multimodal datasets; one that features flies and their neural activity, one that contains human neural Electrocorticography (ECoG) data, and lastly the RGB video data of human activities from different viewpoints.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 44 of its 113 references that are in the map. First posted on arXiv 2 Dec 2021.

Builds on

44 of 113 references.

References · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  9. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  10. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  11. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  12. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
32 more
  1. 2021BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG datacited by 194
  2. 2021VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examplescited by 154
  3. 2021TCLR: Temporal Contrastive Learning for Video Representationcited by 136
  4. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 122
  5. 2020Multi-Modal Domain Adaptation for Fine-Grained Action Recognitioncited by 115
  6. 2019Temporal Attentive Alignment for Large-Scale Video Domain Adaptationcited by 100
  7. 2020MS2L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognitioncited by 92
  8. 2019PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognitioncited by 90
  9. 2020Articulation-aware Canonical Surface Mappingcited by 80
  10. 2016Deep Learning Human Mind for Automated Visual Classificationcited by 76
  11. 2020Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learningcited by 62
  12. 2020Social NCE: Contrastive Learning of Socially-aware Motion Representationscited by 58
  13. 2021Neural Latents Benchmark '21: Evaluating latent variable models of neural population activitycited by 57
  14. 2018Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Featurescited by 52
  15. 2021Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixingcited by 49
  16. 2017Machine learning for neural decodingcited by 45
  17. 2016Linear dynamical neural population models through nonlinear embeddingscited by 36
  18. 2021The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactionscited by 31
  19. 2021Aligning Correlation Information for Domain Adaptation in Action Recognitioncited by 30
  20. 2021Large-scale neural recordings call for new insights to link brain and behaviorcited by 28
  21. 2018Adversarial Domain Adaptation for Stable Brain-Machine Interfacescited by 27
  22. 2018Tree-Structured Recurrent Switching Linear Dynamical Systems for Multi-Scale Modelingcited by 27
  23. 2020Task Programming: Learning Data Efficient Behavior Representationscited by 27
  24. 20203D-ZeF: A 3D Zebrafish Tracking Benchmark Datasetcited by 24
  25. 2020Transferring Dense Pose to Proximal Animal Classescited by 23
  26. 2019Virtual Mixup Training for Unsupervised Domain Adaptationcited by 21
  27. 2020Object classification from randomized EEG trialscited by 19
  28. 2016Learning recurrent representations for hierarchical behavior modelingcited by 16
  29. 2020Select, Label, and Mix: Learning Discriminative Invariant Feature Representations for Partial Domain Adaptationcited by 16
  30. 2020Deformation-aware Unpaired Image Translation for Pose Estimation on Laboratory Animalscited by 14
  31. 2013Bayesian spike inference from calcium imaging datacited by 4
  32. 2017AJILE Movement Prediction: Multimodal Deep Learning for Natural Human Neural Recordings and Videocited by 3

Built on by

Not cited yet.