# PeptideBERT: A Language Model based on Transformers for Peptide Property Prediction

arXiv: https://arxiv.org/abs/2309.03099 · HTML: https://mlsanity.com/paper/2309.03099/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 54 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction](https://mlsanity.com/paper/1811.02633/) (2018)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [Mapping the Space of Chemical Reactions Using Attention-Based Neural Networks](https://mlsanity.com/paper/2012.06051/) (2020)
- [Classifying Antimicrobial and Multifunctional Peptides with Bayesian Network Models](https://mlsanity.com/paper/1804.06327/) (2018)

## Built on by (most cited first, 21 of 21)

- [Protein Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2502.17504/) (2025)
- [Multi-Objective-Guided Discrete Flow Matching for Controllable Biological Sequence Design](https://mlsanity.com/paper/2505.07086/) (2025)
- [GPT-MolBERTa: GPT Molecular Features Language Model for molecular property prediction](https://mlsanity.com/paper/2310.03030/) (2023)
- [Large Language Model Agent for Modular Task Execution in Drug Discovery](https://mlsanity.com/paper/2507.02925/) (2025)
- [TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusion](https://mlsanity.com/paper/2509.25171/) (2025)
- [MOFGPT: Generative Design of Metal-Organic Frameworks using Language Models](https://mlsanity.com/paper/2506.00198/) (2025)
- [Multi-Peptide: Multimodality Leveraged Language-Graph Learning of Peptide Properties](https://mlsanity.com/paper/2407.03380/) (2024)
- [Computational Protein Science in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2501.10282/) (2025)
- [AlloyBERT: Alloy Property Prediction with Large Language Models](https://mlsanity.com/paper/2403.19783/) (2024)
- [Peptide-GPT: Generative Design of Peptides using Generative Pre-trained Transformers and Bio-informatic Supervision](https://mlsanity.com/paper/2410.19222/) (2024)
- [AReUReDi: Annealed Rectified Updates for Refining Discrete Flows with Multi-Objective Guidance](https://mlsanity.com/paper/2510.00352/) (2025)
- [GPCR-BERT: Interpreting Sequential Design of G Protein Coupled Receptors Using Protein Language Models](https://mlsanity.com/paper/2310.19915/) (2023)
- [PepHarmony: A Multi-View Contrastive Learning Framework for Integrated Sequence and Structure-Based Peptide Encoding](https://mlsanity.com/paper/2401.11360/) (2024)
- [Beyond Force Metrics: Pre-Training MLFFs for Stable MD Simulations](https://mlsanity.com/paper/2506.14850/) (2025)
- [Protein Structure-Function Relationship: A Kernel-PCA Approach for Reaction Coordinate Identification](https://mlsanity.com/paper/2503.19186/) (2025)
- [Tensor-DTI: Enhancing Biomolecular Interaction Prediction with Contrastive Embedding Learning](https://mlsanity.com/paper/2601.05792/) (2026)
- [Pepti-Agent: An AI Agent for Peptide Design and Optimization](https://mlsanity.com/paper/2606.15422/) (2026)
- [Text to Band Gap: Pre-trained Language Models as Encoders for Semiconductor Band Gap Prediction](https://mlsanity.com/paper/2501.03456/) (2025)
- [BioPars: A Pretrained Biomedical Large Language Model for Persian Biomedical Text Mining](https://mlsanity.com/paper/2506.21567/) (2025)
- [Meta-Learning for Cross-Task Generalization in Protein Mutation Property Prediction](https://mlsanity.com/paper/2510.20943/) (2025)
- [Reflection Pretraining Enables Token-Level Self-Correction in Biological Sequence Models](https://mlsanity.com/paper/2512.20954/) (2025)

## Abstract (arXiv)

Recent advances in Language Models have enabled the protein modeling community with a powerful tool since protein sequences can be represented as text. Specifically, by taking advantage of Transformers, sequence-to-property prediction will be amenable without the need for explicit structural data. In this work, inspired by recent progress in Large Language Models (LLMs), we introduce PeptideBERT, a protein language model for predicting three key properties of peptides (hemolysis, solubility, and non-fouling). The PeptideBert utilizes the ProtBERT pretrained transformer model with 12 attention heads and 12 hidden layers. We then finetuned the pretrained model for the three downstream tasks. Our model has achieved state of the art (SOTA) for predicting Hemolysis, which is a task for determining peptide's potential to induce red blood cell lysis. Our PeptideBert non-fouling model also achieved remarkable accuracy in predicting peptide's capacity to resist non-specific interactions. This model, trained predominantly on shorter sequences, benefits from the dataset where negative examples are largely associated with insoluble peptides. Codes, models, and data used in this study are freely available at: https://github.com/ChakradharG/PeptideBERT