Self-supervised learning of multi-omics embeddings in the low-label, high-data regime
Not on the map. Why?
Abstract (arXiv)
Contrastive, self-supervised learning (SSL) is used to train a model that predicts cancer type from miRNA, mRNA or RPPA expression data. This model, a pretrained FT-Transformer, is shown to outperform XGBoost and CatBoost, standard benchmarks for tabular data, when labelled samples are scarce but the number of unlabelled samples is high. This is despite the fact that the datasets we use have O(101) classes and O(102)-O(104) features. After demonstrating the efficacy of our chosen method of self-supervised pretraining, we investigate SSL for multi-modal models. A late-fusion model is proposed, where each omics is passed through its own sub-network, the outputs of which are averaged and passed to the pretraining or downstream objective function. Multi-modal pretraining is shown to improve predictions from a single omics, and we argue that this is useful for datasets with many unlabelled multi-modal samples, but few labelled unimodal samples. Additionally, we show that pretraining each omics-specific module individually is highly effective. This enables the application of the proposed model in a variety of contexts where a large amount of unlabelled data is available from each omics, but only a few labelled samples.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 16 Nov 2023.
Builds on
29 of 38 references.
References · 29
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
- 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
- 2021Zero-Shot Text-to-Image Generationcited by 4,288
- 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
17 more
- 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
- 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
- 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
- 2021Revisiting Deep Learning Models for Tabular Datacited by 740
- 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
- 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 319
- 2023A Cookbook of Self-Supervised Learningcited by 250
- 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 156
- 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 109
- 2021SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learningcited by 104
- 2023Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Datacited by 69
- 2021Reducing Label Effort: Self-Supervised meets Active Learningcited by 42
- 2021A Variational Information Bottleneck Approach to Multi-Omics Data Integrationcited by 40
- 2021OmiEmbed: a unified multi-task deep learning framework for multi-omics datacited by 12
- 2023Rethinking Data Augmentation for Tabular Data in Deep Learningcited by 10
- 2023CLCLSA: Cross-omics Linked embedding with Contrastive Learning and Self Attention for multi-omics integration with incomplete multi-omics datacited by 2
- 2022Self-omics: A Self-supervised Learning Framework for Multi-omics Cancer Datacited by 1
Built on by
Not cited yet.