# DNABERT-S: Pioneering Species Differentiation with Species-Aware DNA Embeddings

arXiv: https://arxiv.org/abs/2402.08777 · HTML: https://mlsanity.com/paper/2402.08777/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 13 Feb 2024.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [On The Power of Curriculum Learning in Training Deep Networks](https://mlsanity.com/paper/1904.03626/) (2019)
- [HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution](https://mlsanity.com/paper/2306.15794/) (2023)
- [DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome](https://mlsanity.com/paper/2306.15006/) (2023)
- [Pairwise Supervised Contrastive Learning of Sentence Representations](https://mlsanity.com/paper/2109.05424/) (2021)
- [i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning](https://mlsanity.com/paper/2010.08887/) (2020)
- [Learning Dialogue Representations from Consecutive Utterances](https://mlsanity.com/paper/2205.13568/) (2022)
- [dna2vec: Consistent vector representations of variable-length k-mers](https://mlsanity.com/paper/1701.06279/) (2017)
- [Manifold Mixup: Better Representations by Interpolating Hidden States](https://mlsanity.com/paper/1806.05236/) (2018)
- [Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning](https://mlsanity.com/paper/2109.05941/) (2021)

## Built on by (most cited first, 1 of 1)

- [GENEB: Why Genomic Models Are Hard to Compare](https://mlsanity.com/paper/2606.04525/) (2026)

## Abstract (arXiv)

We introduce DNABERT-S, a tailored genome model that develops species-aware embeddings to naturally cluster and segregate DNA sequences of different species in the embedding space. Differentiating species from genomic sequences (i.e., DNA and RNA) is vital yet challenging, since many real-world species remain uncharacterized, lacking known genomes for reference. Embedding-based methods are therefore used to differentiate species in an unsupervised manner. DNABERT-S builds upon a pre-trained genome foundation model named DNABERT-2. To encourage effective embeddings to error-prone long-read DNA sequences, we introduce Manifold Instance Mixup (MI-Mix), a contrastive objective that mixes the hidden representations of DNA sequences at randomly selected layers and trains the model to recognize and differentiate these mixed proportions at the output layer. We further enhance it with the proposed Curriculum Contrastive Learning (C2LR) strategy. Empirical results on 23 diverse datasets show DNABERT-S's effectiveness, especially in realistic label-scarce scenarios. For example, it identifies twice more species from a mixture of unlabeled genomic sequences, doubles the Adjusted Rand Index (ARI) in species clustering, and outperforms the top baseline's performance in 10-shot species classification with just a 2-shot training. Model, codes, and data are publicly available at urlhttps://github.com/MAGICS-LAB/DNABERTS.