# GeneFormer: Learned Gene Compression using Transformer-based Context Modeling

arXiv: https://arxiv.org/abs/2212.08379 · HTML: https://mlsanity.com/paper/2212.08379/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 60 references that are in the map.
- First posted on arXiv 16 Dec 2022.

## Builds on (most cited first, 6 of 6)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [General-purpose, long-context autoregressive modeling with Perceiver AR](https://mlsanity.com/paper/2202.07765/) (2022)
- [A Fast Transformer-based General-Purpose Lossless Compressor](https://mlsanity.com/paper/2203.16114/) (2022)

## Built on by (most cited first, 10 of 10)

- [The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4](https://mlsanity.com/paper/2311.07361/) (2023)
- [A Large-Scale Benchmark of Cross-Modal Learning for Histology and Gene Expression in Spatial Transcriptomics](https://mlsanity.com/paper/2508.01490/) (2025)
- [scCluBench: Comprehensive Benchmarking of Clustering Algorithms for Single-Cell RNA Sequencing](https://mlsanity.com/paper/2512.02471/) (2025)
- [Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities in Biomedicine](https://mlsanity.com/paper/2405.19950/) (2024)
- [Progressive Multi-Agent Reasoning for Biological Perturbation Prediction](https://mlsanity.com/paper/2602.07408/) (2026)
- [Best Practices for Biorisk Evaluations on Open-Weight Bio-Foundation Models](https://mlsanity.com/paper/2510.27629/) (2025)
- [AgentGC: Evolutionary Learning-based Lossless Compression for Genomics Data with LLM-driven Multiple Agent](https://mlsanity.com/paper/2601.13559/) (2026)
- [D2R2: Discrete Diffusion with Regulation Reinforcement for Single-Cell Perturbation Prediction](https://mlsanity.com/paper/2608.15288/) (2026)
- [GRIT: Graph-Regularized Logit Refinement for Zero-shot Cell Type Annotation](https://mlsanity.com/paper/2508.04747/) (2025)
- [PanFoMa: A Lightweight Foundation Model and Benchmark for Pan-Cancer](https://mlsanity.com/paper/2512.03111/) (2025)

## Abstract (arXiv)

With the development of gene sequencing technology, an explosive growth of gene data has been witnessed. And the storage of gene data has become an important issue. Traditional gene data compression methods rely on general software like G-zip, which fails to utilize the interrelation of nucleotide sequence. Recently, many researchers begin to investigate deep learning based gene data compression method. In this paper, we propose a transformer-based gene compression method named GeneFormer. Specifically, we first introduce a modified transformer structure to fully explore the nucleotide sequence dependency. Then, we propose fixed-length parallel grouping to accelerate the decoding speed of our autoregressive model. Experimental results on real-world datasets show that our method saves 29.7% bit rate compared with the state-of-the-art method, and the decoding speed is significantly faster than all existing learning-based gene compression methods.