# Protein Structure Tokenization: Benchmarking and New Recipe

arXiv: https://arxiv.org/abs/2503.00089 · HTML: https://mlsanity.com/paper/2503.00089/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 71 references that are in the map.
- First posted on arXiv 28 Feb 2025.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Finite Scalar Quantization: VQ-VAE Made Simple](https://mlsanity.com/paper/2309.15505/) (2023)
- [Evaluating Protein Transfer Learning with TAPE](https://mlsanity.com/paper/1906.08230/) (2019)
- [PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding](https://mlsanity.com/paper/2206.02096/) (2022)
- [ATOM3D: Tasks On Molecules in Three Dimensions](https://mlsanity.com/paper/2012.04035/) (2020)
- [Vocabulary Learning via Optimal Transport for Neural Machine Translation](https://mlsanity.com/paper/2012.15671/) (2020)
- [DeepSF: deep convolutional neural network for mapping protein sequences to folds](https://mlsanity.com/paper/1706.01010/) (2017)
- [DASB - Discrete Audio and Speech Benchmark](https://mlsanity.com/paper/2406.14294/) (2024)
- [Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modeling](https://mlsanity.com/paper/2403.10071/) (2024)
- [Evaluating representation learning on the protein structure universe](https://mlsanity.com/paper/2406.13864/) (2024)
- [Learning the Language of Protein Structure](https://mlsanity.com/paper/2405.15840/) (2024)
- [Bio2Token: All-atom tokenization of any biomolecular structure with Mamba](https://mlsanity.com/paper/2410.19110/) (2024)

## Built on by (most cited first, 15 of 15)

- [Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality](https://mlsanity.com/paper/2505.18227/) (2025)
- [Protein Structure Tokenization via Geometric Byte Pair Encoding](https://mlsanity.com/paper/2511.11758/) (2025)
- [ProteinAE: Protein Diffusion Autoencoders for Structure Encoding](https://mlsanity.com/paper/2510.10634/) (2025)
- [Adaptive Protein Tokenization](https://mlsanity.com/paper/2602.06418/) (2026)
- [Yeti: A compact protein structure tokenizer for reconstruction and multi-modal generation](https://mlsanity.com/paper/2605.09981/) (2026)
- [ENSEMBITS: an alphabet of protein conformational ensembles](https://mlsanity.com/paper/2605.13789/) (2026)
- [Flow-Matching-Based Protein Structure Tokenizer Made Efficient and Easy](https://mlsanity.com/paper/2609.33129/) (2026)
- [Tokenizing Loops of Antibodies](https://mlsanity.com/paper/2509.08707/) (2025)
- [From Snapshots to Symphonies: The Evolution of Protein Prediction from Static Structures to Generative Dynamics and Multimodal Interactions](https://mlsanity.com/paper/2603.18505/) (2026)
- [Mitigating Premature Discretization with Progressive Quantization for Robust Vector Tokenization](https://mlsanity.com/paper/2603.22304/) (2026)
- [Atom-level Protein Representation Learning Improves Protein Structure Prediction](https://mlsanity.com/paper/2605.22133/) (2026)
- [CryoProt: A Protein Pretraining Framework with Cross-Box Interactions on Cryo-EM Density Maps](https://mlsanity.com/paper/2606.00955/) (2026)
- [Discrete Diffusion Models: A Unified Framework from Tokenization to Generation](https://mlsanity.com/paper/2607.13431/) (2026)
- [Learning Latent Protein Languages for Autoregressive Generation](https://mlsanity.com/paper/2610.03978/) (2026)
- [ProTok: Unified Protein Tokenization via Modular Multi-Codebook Quantization](https://mlsanity.com/paper/2610.18481/) (2026)

## Abstract (arXiv)

Recent years have witnessed a surge in the development of protein structural tokenization methods, which chunk protein 3D structures into discrete or continuous representations. Structure tokenization enables the direct application of powerful techniques like language modeling for protein structures, and large multimodal models to integrate structures with protein sequences and functional texts. Despite the progress, the capabilities and limitations of these methods remain poorly understood due to the lack of a unified evaluation framework. We first introduce StructTokenBench, a framework that comprehensively evaluates the quality and efficiency of structure tokenizers, focusing on fine-grained local substructures rather than global structures, as typical in existing benchmarks. Our evaluations reveal that no single model dominates all benchmarking perspectives. Observations of codebook under-utilization led us to develop AminoAseed, a simple yet effective strategy that enhances codebook gradient updates and optimally balances codebook size and dimension for improved tokenizer utilization and quality. Compared to the leading model ESM3, our method achieves an average of 6.31% performance improvement across 24 supervised tasks, with sensitivity and utilization rates increased by 12.83% and 124.03%, respectively. Source code and model weights are available at https://github.com/KatarinaYuan/StructTokenBench