# Freeze, Diffuse, Decode: Task-Aware Adaptation of Transformer Embeddings for Antimicrobial Peptide Design

arXiv: https://arxiv.org/abs/2511.23120 · HTML: https://mlsanity.com/paper/2511.23120/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 37 references that are in the map.
- First posted on arXiv 28 Nov 2025.

## Builds on (most cited first, 19 of 19)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [Learning Generative Models with Sinkhorn Divergences](https://mlsanity.com/paper/1706.00292/) (2017)
- [The geometry of hidden representations of large transformer models](https://mlsanity.com/paper/2302.00294/) (2023)
- [The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning](https://mlsanity.com/paper/2106.15831/) (2021)
- [A Non-Linear Structural Probe](https://mlsanity.com/paper/2105.10185/) (2021)
- [Geometry- and Accuracy-Preserving Random Forest Proximities](https://mlsanity.com/paper/2201.12682/) (2022)
- [How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy](https://mlsanity.com/paper/2109.04740/) (2021)
- [Extendable and invertible manifold learning with geometry regularized autoencoders](https://mlsanity.com/paper/2007.07142/) (2020)
- [Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learning](https://mlsanity.com/paper/2508.06199/) (2025)
- [Optimal Transport losses and Sinkhorn algorithm with general convex regularization](https://mlsanity.com/paper/2007.00976/) (2020)
- [Molecular Fingerprints Are Strong Models for Peptide Function Prediction](https://mlsanity.com/paper/2501.17901/) (2025)
- [Synergistic Benefits of Joint Molecule Generation and Property Prediction](https://mlsanity.com/paper/2504.16559/) (2025)
- [Random Forest Autoencoders for Guided Representation Learning](https://mlsanity.com/paper/2502.13257/) (2025)

## Abstract (arXiv)

Pretrained transformers provide rich, general-purpose embeddings, which are transferred to downstream tasks. However, current transfer strategies: fine-tuning and probing, either distort the pretrained geometric structure of the embeddings or lack sufficient expressivity to capture task-relevant signals. These issues become even more pronounced when supervised data are scarce. Here, we introduce Freeze, Diffuse, Decode (FDD), a novel diffusion-based framework that adapts pre-trained embeddings to downstream tasks while preserving their underlying geometric structure. FDD propagates supervised signal along the intrinsic manifold of frozen embeddings, enabling a geometry-aware adaptation of the embedding space. Applied to antimicrobial peptide design, FDD yields low-dimensional, predictive, and interpretable representations that support property prediction, retrieval, and latent-space interpolation.