How good are variational autoencoders at transfer learning?
Not on the map. Why?
Abstract (arXiv)
Variational autoencoders (VAEs) are used for transfer learning across various research domains such as music generation or medical image analysis. However, there is no principled way to assess before transfer which components to retrain or whether transfer learning is likely to help on a target task. We propose to explore this question through the lens of representational similarity. Specifically, using Centred Kernel Alignment (CKA) to evaluate the similarity of VAEs trained on different datasets, we show that encoders' representations are generic but decoders' specific. Based on these insights, we discuss the implications for selecting which components of a VAE to retrain and propose a method to visually assess whether transfer learning is likely to help on classification tasks.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 31 of its 55 references that are in the map. First posted on arXiv 21 Apr 2023.
Builds on
31 of 55 references.
References · 31
- 2014Deep Learning Face Attributes in the Wildcited by 3,679
- 2015Variational Inference with Normalizing Flowscited by 2,161
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2014How transferable are features in deep neural networks?cited by 1,216
- 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
- 2015Generating Sentences from a Continuous Spacecited by 607
- 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
- 2016Tutorial on Variational Autoencoderscited by 404
- 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 366
- 2020What is being transferred in transfer learning?cited by 365
- 2015Understanding Neural Networks Through Deep Visualizationcited by 332
- 2018Understanding disentangling in β-VAEcited by 332
19 more
- 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
- 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
- 2017Variational Inference of Disentangled Latent Concepts from Unlabeled Observationscited by 240
- 2021Revisiting Model Stitching to Compare Neural Representationscited by 203
- 2019Diagnosing and Enhancing VAE Modelscited by 192
- 2019Lagging Inference Networks and Posterior Collapse in Variational Autoencoderscited by 124
- 2019Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapsecited by 120
- 2018Variational Autoencoders Pursue PCA Directions (by Accident)cited by 84
- 2021Similarity and Matching of Neural Network Representationscited by 77
- 2019Investigating Multilingual NMT Representations at Scalecited by 68
- 2018Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representationcited by 62
- 2019Universality and individuality in neural dynamics across large populations of recurrent networkscited by 61
- 2019The Usual Suspects? Reassessing Blame for VAE Posterior Collapsecited by 52
- 2021Variational AutoEncoder for Reference based Image Super-Resolutioncited by 18
- 2019Improving Automatic Jazz Melody Generation by Transfer Learning Techniquescited by 8
- 2021Be More Active! Understanding the Differences between Mean and Sampled Representations of Variational Autoencoderscited by 7
- 2018Transfer Learning From Synthetic To Real Images Using Variational Autoencoders For Precise Position Detectioncited by 6
- 2021Information flows of diverse autoencoderscited by 6
- 2022FONDUE: an algorithm to find the optimal dimensionality of the latent representations of variational autoencoderscited by 5
Built on by
Citing papers per year: 2024: 2; 2025: 0; 2026: 1.
3 papers.