# What Makes a Latent Space Shared? Direct and Recoverable Compatibility in Motion VAEs

arXiv: https://arxiv.org/abs/2610.17376 · HTML: https://mlsanity.com/paper/2610.17376/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 12 Oct 2026.

## Builds on (most cited first, 25 of 25)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style](https://mlsanity.com/paper/2106.04619/) (2021)
- [Relative representations enable zero-shot latent space communication](https://mlsanity.com/paper/2209.15430/) (2022)
- [Skeleton-Aware Networks for Deep Motion Retargeting](https://mlsanity.com/paper/2005.05732/) (2020)
- [On Linear Identifiability of Learned Representations](https://mlsanity.com/paper/2007.00810/) (2020)
- [Neural Kinematic Networks for Unsupervised Motion Retargetting](https://mlsanity.com/paper/1804.05653/) (2018)
- [Variational Autoencoders Pursue PCA Directions (by Accident)](https://mlsanity.com/paper/1812.06775/) (2018)
- [CycleGAN, a Master of Steganography](https://mlsanity.com/paper/1712.02950/) (2017)
- [Identifiability Results for Multimodal Contrastive Learning](https://mlsanity.com/paper/2303.09166/) (2023)
- [Latent Space Translation via Semantic Alignment](https://mlsanity.com/paper/2311.00664/) (2023)
- [Pose-aware Attention Network for Flexible Motion Retargeting by Body Part](https://mlsanity.com/paper/2306.08006/) (2023)
- [Multi-Way Representation Alignment](https://mlsanity.com/paper/2602.06205/) (2026)
- [Bridging Functional and Representational Similarity via Usable Information](https://mlsanity.com/paper/2601.21568/) (2026)

## Abstract (arXiv)

A shared latent space is often assumed to make representations from different schemas directly interchangeable, yet sharing parameters, dimensionality, or a prior does not determine how corresponding instances are expressed in latent coordinates. We show that this assumption conflates recoverable correspondence, direct compatibility, and decoder reuse. In a controlled multi-schema motion VAE on HumanML3D, even identical inputs yield only 5.9--17.7% raw Recall@1 and 0.221--0.484 paired-token cosine, despite correspondence being almost perfectly recoverable after linear calibration. Cycle consistency strengthens recoverability but still leaves direct retrieval and token agreement near zero. In contrast, only 64 paired clips raise raw Recall@1 to 88.4--90.3% and token cosine to approximately 0.82, making most post-hoc alignment unnecessary, although crossed decoding remains substantially worse than native reconstruction. We further find that reducing measured marginal discrepancy does not imply schema invariance. These results show that a shared backbone or prior does not ensure directly compatible latent coordinates, while limited paired correspondence can substantially improve direct compatibility in the tested setting.