# FoldToken: Learning Protein Language via Vector Quantization and Beyond

arXiv: https://arxiv.org/abs/2403.09673 · HTML: https://mlsanity.com/paper/2403.09673/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 50 references that are in the map.
- First posted on arXiv 4 Feb 2024.

## Builds on (most cited first, 30 of 30)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Autoregressive Image Generation using Residual Quantization](https://mlsanity.com/paper/2203.01941/) (2022)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [Finite Scalar Quantization: VQ-VAE Made Simple](https://mlsanity.com/paper/2309.15505/) (2023)
- [Learning from Protein Structure with Geometric Vector Perceptrons](https://mlsanity.com/paper/2009.01411/) (2020)
- [Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problem](https://mlsanity.com/paper/2206.04119/) (2022)
- [Protein Representation Learning by Geometric Structure Pretraining](https://mlsanity.com/paper/2203.06125/) (2022)
- [Protein structure generation via folding diffusion](https://mlsanity.com/paper/2209.15611/) (2022)
- [Iterative Refinement Graph Neural Network for Antibody Sequence-Structure Co-design](https://mlsanity.com/paper/2110.04624/) (2021)
- [PiFold: Toward effective and efficient protein inverse folding](https://mlsanity.com/paper/2209.12643/) (2022)
- [Conditional Antibody Design as 3D Equivariant Graph Translation](https://mlsanity.com/paper/2208.06073/) (2022)
- [AlphaDesign: A graph protein design method and benchmark on AlphaFoldDB](https://mlsanity.com/paper/2202.01079/) (2022)
- [Differentiable Product Quantization for End-to-End Embedding Compression](https://mlsanity.com/paper/1908.09756/) (2019)
- [Image Compression with Product Quantized Masked Image Modeling](https://mlsanity.com/paper/2212.07372/) (2022)
- [Protein Sequence and Structure Co-Design with Equivariant Translation](https://mlsanity.com/paper/2210.08761/) (2022)
- [Stacked Quantizers for Compositional Vector Compression](https://mlsanity.com/paper/1411.2173/) (2014)
- [Regression as Classification: Influence of Task Formulation on Neural Network Features](https://mlsanity.com/paper/2211.05641/) (2022)
- [A step towards understanding why classification helps regression](https://mlsanity.com/paper/2308.10603/) (2023)
- [Knowledge-Design: Pushing the Limit of Protein Design via Knowledge Refinement](https://mlsanity.com/paper/2305.15151/) (2023)
- [DiffSDS: A language diffusion model for protein backbone inpainting under geometric conditions and constraints](https://mlsanity.com/paper/2301.09642/) (2023)
- [Cross-Gate MLP with Protein Complex Invariant Embedding is A One-Shot Antibody Designer](https://mlsanity.com/paper/2305.09480/) (2023)
- [Progressive Multi-Modality Learning for Inverse Protein Folding](https://mlsanity.com/paper/2312.06297/) (2023)
- [Streaming Active Learning for Regression Problems Using Regression via Classification](https://mlsanity.com/paper/2309.01013/) (2023)
- [Functional Geometry Guided Protein Sequence and Backbone Structure Co-Design](https://mlsanity.com/paper/2310.04343/) (2023)
- [Efficiently Predicting Protein Stability Changes Upon Single-point Mutation with Large Language Models](https://mlsanity.com/paper/2312.04019/) (2023)

## Built on by (most cited first, 23 of 23)

- [Addressing Representation Collapse in Vector Quantized Models with One Linear Layer](https://mlsanity.com/paper/2411.02038/) (2024)
- [DPLM-2: A Multimodal Diffusion Protein Language Model](https://mlsanity.com/paper/2410.13782/) (2024)
- [Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality](https://mlsanity.com/paper/2505.18227/) (2025)
- [Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Survey](https://mlsanity.com/paper/2403.01528/) (2024)
- [Structure Language Models for Protein Conformation Generation](https://mlsanity.com/paper/2410.18403/) (2024)
- [Learning the Language of Protein Structure](https://mlsanity.com/paper/2405.15840/) (2024)
- [Bio2Token: All-atom tokenization of any biomolecular structure with Mamba](https://mlsanity.com/paper/2410.19110/) (2024)
- [Flow Autoencoders are Effective Protein Tokenizers](https://mlsanity.com/paper/2510.00351/) (2025)
- [Constraint Decoupled Latent Diffusion for Protein Backmapping](https://mlsanity.com/paper/2410.13264/) (2024)
- [Adaptive Protein Tokenization](https://mlsanity.com/paper/2602.06418/) (2026)
- [FoldToken2: Learning compact, invariant and generative protein structure language](https://mlsanity.com/paper/2407.00050/) (2024)
- [Tokenizing 3D Molecule Structure with Quantized Spherical Coordinates](https://mlsanity.com/paper/2412.01564/) (2024)
- [VecFormer: Towards Efficient and Generalizable Graph Transformer with Graph Token Attention](https://mlsanity.com/paper/2602.19622/) (2026)
- [Yeti: A compact protein structure tokenizer for reconstruction and multi-modal generation](https://mlsanity.com/paper/2605.09981/) (2026)
- [PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documents](https://mlsanity.com/paper/2605.10341/) (2026)
- [Flow-Matching-Based Protein Structure Tokenizer Made Efficient and Easy](https://mlsanity.com/paper/2609.33129/) (2026)
- [AlphaFold Database Debiasing for Robust Inverse Folding](https://mlsanity.com/paper/2506.08365/) (2025)
- [Reinforcement-Learned Unequal Error Protection for Quantized Semantic Embeddings](https://mlsanity.com/paper/2601.00186/) (2026)
- [RiboSphere: Learning Unified and Efficient Representations of RNA Structures](https://mlsanity.com/paper/2603.19636/) (2026)
- [Mitigating Premature Discretization with Progressive Quantization for Robust Vector Tokenization](https://mlsanity.com/paper/2603.22304/) (2026)
- [Discrete Diffusion Models: A Unified Framework from Tokenization to Generation](https://mlsanity.com/paper/2607.13431/) (2026)
- [Learning Latent Protein Languages for Autoregressive Generation](https://mlsanity.com/paper/2610.03978/) (2026)
- [ProTok: Unified Protein Tokenization via Modular Multi-Codebook Quantization](https://mlsanity.com/paper/2610.18481/) (2026)

## Abstract (arXiv)

Is there a foreign language describing protein sequences and structures simultaneously? Protein structures, represented by continuous 3D points, have long posed a challenge due to the contrasting modeling paradigms of discrete sequences. We introduce FoldTokenizer to represent protein sequence-structure as discrete symbols. This innovative approach involves projecting residue types and structures into a discrete space, guided by a reconstruction loss for information preservation. We refer to the learned discrete symbols as FoldToken, and the sequence of FoldTokens serves as a new protein language, transforming the protein sequence-structure into a unified modality. We apply the created protein language on general backbone inpainting and antibody design tasks, building the first GPT-style model (FoldGPT) for sequence-structure co-generation with promising results. Key to our success is the substantial enhancement of the vector quantization module, Soft Conditional Vector Quantization (SoftCVQ).