# HD-Prot: A Protein Language Model for Joint Sequence-Structure Modeling with Continuous Structure Tokens

arXiv: https://arxiv.org/abs/2512.15133 · HTML: https://mlsanity.com/paper/2512.15133/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 58 references that are in the map.
- First posted on arXiv 17 Dec 2025.

## Builds on (most cited first, 28 of 28)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [Autoregressive Image Generation without Vector Quantization](https://mlsanity.com/paper/2406.11838/) (2024)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2410.13848/) (2024)
- [Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design](https://mlsanity.com/paper/2402.04997/) (2024)
- [Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens](https://mlsanity.com/paper/2410.13863/) (2024)
- [Diffusion Language Models Are Versatile Protein Learners](https://mlsanity.com/paper/2402.18567/) (2024)
- [Proteina: Scaling Flow-based Protein Structure Generative Models](https://mlsanity.com/paper/2503.00710/) (2025)
- [DPLM-2: A Multimodal Diffusion Protein Language Model](https://mlsanity.com/paper/2410.13782/) (2024)
- [Discrete Diffusion in Large Language and Multimodal Models: A Survey](https://mlsanity.com/paper/2506.13759/) (2025)
- [SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer](https://mlsanity.com/paper/2412.10958/) (2024)
- [Improved motif-scaffolding with SE(3) flow matching](https://mlsanity.com/paper/2401.04082/) (2024)
- [Out of Many, One: Designing and Scaffolding Proteins at the Scale of the Structural Universe with Genie 2](https://mlsanity.com/paper/2405.15489/) (2024)
- [La-Proteina: Atomistic Protein Generation via Partially Latent Flow Matching](https://mlsanity.com/paper/2507.09466/) (2025)
- [Speak-to-Structure: Evaluating LLMs in Open-domain Natural Language-Driven Molecule Generation](https://mlsanity.com/paper/2412.14642/) (2024)
- [Diffusion on language model encodings for protein sequence generation](https://mlsanity.com/paper/2403.03726/) (2024)
- [MolReFlect: Towards In-Context Fine-grained Alignments between Molecules and Texts](https://mlsanity.com/paper/2411.14721/) (2024)
- [Elucidating the Design Space of Multimodal Protein Language Models](https://mlsanity.com/paper/2504.11454/) (2025)
- [Computational Protein Science in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2501.10282/) (2025)
- [ProtInvTree: Deliberate Protein Inverse Folding with Reward-guided Tree Search](https://mlsanity.com/paper/2506.00925/) (2025)
- [GLProtein: Global-and-Local Structure Aware Protein Representation Learning](https://mlsanity.com/paper/2506.06294/) (2025)

## Built on by (most cited first, 4 of 4)

- [Enhancing Molecular Property Predictions by Learning from Bond Modelling and Interactions](https://mlsanity.com/paper/2603.00568/) (2026)
- [DeMa: Dual-Path Delay-Aware Mamba for Efficient Multivariate Time Series Analysis](https://mlsanity.com/paper/2601.05527/) (2026)
- [PriHA: A RAG-Enhanced LLM Framework for Primary Healthcare Assistant in Hong Kong](https://mlsanity.com/paper/2604.14215/) (2026)
- [Unlocking Multimodal Protein Language Models at Inference Time](https://mlsanity.com/paper/2608.25855/) (2026)

## Abstract (arXiv)

Proteins inherently possess a consistent sequence-structure duality. The abundance of protein sequence data, which can be readily represented as discrete tokens, has driven fruitful developments in protein language models (pLMs). A key remaining challenge, however, is how to effectively integrate continuous structural knowledge into pLMs. Current methods often discretize protein structures to accommodate the language modeling framework, which inevitably results in the loss of fine-grained information and limits the performance potential of multimodal pLMs. In this paper, we argue that such concerns can be circumvented: a sequence-based pLM can be extended to incorporate the structure modality through continuous tokens, i.e., high-fidelity protein structure latents that avoid vector quantization. Specifically, we propose a hybrid diffusion protein language model, HD-Prot, which embeds a continuous-valued diffusion head atop a discrete pLM, enabling seamless operation with both discrete and continuous tokens for joint sequence-structure modeling. It captures inter-token dependencies across modalities through a unified absorbing diffusion process, and estimates per-token distributions via categorical prediction for sequences and continuous diffusion for structures. Extensive results demonstrate that HD-Prot achieves competitive performance in unconditional sequence-structure co-generation, motif-scaffolding, protein structure prediction, and inverse folding tasks. Furthermore, our method can perform on par with state-of-the-art multimodal pLMs, despite being developed under limited computational resources (i.e., less than one-tenth the budget for modality extension fine-tuning). It highlights the viability of simultaneously estimating categorical and continuous distributions within a unified language model architecture, offering a promising alternative direction for multimodal pLMs.