# Analysis of Quantized and Efficiently Adapted Protein Language Models

arXiv: https://arxiv.org/abs/2610.00665 · HTML: https://mlsanity.com/paper/2610.00665/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [ProGen2: Exploring the Boundaries of Protein Language Models](https://mlsanity.com/paper/2206.13517/) (2022)
- [Open Sesame: Getting Inside BERT's Linguistic Knowledge](https://mlsanity.com/paper/1906.01698/) (2019)
- [ProLLaMA: A Protein Large Language Model for Multi-Task Protein Language Processing](https://mlsanity.com/paper/2402.16445/) (2024)
- [TorchDrug: A Powerful and Flexible Machine Learning Platform for Drug Discovery](https://mlsanity.com/paper/2202.08320/) (2022)
- [Ankh: Optimized Protein Language Model Unlocks General-Purpose Modelling](https://mlsanity.com/paper/2301.06568/) (2023)
- [Explaining and Improving Model Behavior with k Nearest Neighbor Representations](https://mlsanity.com/paper/2010.09030/) (2020)
- [What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation](https://mlsanity.com/paper/2403.06408/) (2024)
- [Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective](https://mlsanity.com/paper/2508.16712/) (2025)
- [Layer-Wise Evolution of Representations in Fine-Tuned Transformers: Insights from Sparse AutoEncoders](https://mlsanity.com/paper/2502.16722/) (2025)
- [Ankh3: Multi-Task Pretraining with Sequence Denoising and Completion Enhances Protein Representations](https://mlsanity.com/paper/2505.20052/) (2025)
- [Exploring Post-Training Quantization of Protein Language Models](https://mlsanity.com/paper/2310.19624/) (2023)

## Abstract (arXiv)

Background: Protein language models (PLMs) are increasingly used for sequence generation and property prediction, but their size makes fine-tuning and deployment expensive. The effects of quantization and parameter efficient fine-tuning on performance, representations and generation remain insufficiently characterized. Results: We evaluated 4-bit quantization and low-rank adapter fine-tuning (QLoRA) across ESM-2, ESMC, ProtBERT, ProtT5, Ankh, Ankh3 and Profluent-E1. Across protein prediction tasks, many model-task pairs retained more than 90% of full fine-tuning performance. Peak GPU memory savings approached 90% for the largest models, although performance and efficiency varied by model, dataset and training configuration. QLoRA often preserved early-layer representations while inducing task-specific adaptations in middle and late layers, resembling full fine-tuning with smaller representational changes. Training speed and power effects were more varied. For unconditional generation with ProLLaMA, ProtGPT2, ProGen2, ProteinGLM and ESM3, 4-bit quantization largely preserved predicted structural and sequence-level properties, but token-level analysis revealed model-dependent shifts in autoregressive output distributions. Conclusion: QLoRA and 4-bit quantization reduce PLM computational requirements, particularly GPU memory usage. Our results support QLoRA as a first-pass strategy for memory limited adaptation, reserving full fine-tuning for challenging tasks, unstable architectures or low validation recovery. For generative PLMs, sequence-level and structural metrics should be complemented with distributional analysis, since downstream predictions alone may miss quantization-induced shifts. These approaches can broaden access to large-scale protein modelling while requiring model- and task-specific validation.