# Beyond Conditional Computation: Retrieval-Augmented Genomic Foundation Models with Gengram

arXiv: https://arxiv.org/abs/2601.22203 · HTML: https://mlsanity.com/paper/2601.22203/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 13 of its 36 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Placed in

- Retrieval-augmented generation / [Medical and biomedical RAG](https://mlsanity.com/rag/medical-and-biomedical-rag/)

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [ST-MoE: Designing Stable and Transferable Sparse Expert Models](https://mlsanity.com/paper/2202.08906/) (2022)
- [Layer by Layer: Uncovering Hidden Representations in Language Models](https://mlsanity.com/paper/2502.02013/) (2025)
- [Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free](https://mlsanity.com/paper/2505.06708/) (2025)
- [Analyzing the Structure of Attention in a Transformer Language Model](https://mlsanity.com/paper/1906.04284/) (2019)
- [Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling](https://mlsanity.com/paper/2403.03234/) (2024)
- [Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models](https://mlsanity.com/paper/2601.07372/) (2026)
- [GENERator: A Long-Context Generative Genomic Foundation Model](https://mlsanity.com/paper/2502.07272/) (2025)
- [ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specialization](https://mlsanity.com/paper/2511.10971/) (2025)
- [Regulatory DNA sequence Design with Reinforcement Learning](https://mlsanity.com/paper/2503.07981/) (2025)

## Built on by (most cited first, 1 of 1)

- [Memory Is Not Always Needed: Characterizing Conditional Memory in Scientific Reasoning](https://mlsanity.com/paper/2608.23982/) (2026)

## Abstract (arXiv)

Current genomic foundation models (GFMs) rely on extensive neural computation to implicitly approximate conserved biological motifs from single-nucleotide inputs. We propose Gengram, a conditional memory module that introduces an explicit and highly efficient lookup primitive for multi-base motifs via a genomic-specific hashing scheme, establishing genomic "syntax". Integrated into the backbone of state-of-the-art GFMs, Gengram achieves substantial gains (up to 14%) across several functional genomics tasks. The module demonstrates robust architectural generalization, while further inspection of Gengram's latent space reveals the emergence of meaningful representations that align closely with fundamental biological knowledge. By establishing structured motif memory as a modeling primitive, Gengram simultaneously boosts empirical performance and mechanistic interpretability, providing a scalable and biology-aligned pathway for the next generation of GFMs. The code is available at https://github.com/zhejianglab/Genos, and the model checkpoint is available at https://huggingface.co/ZhejiangLab/Gengram.