# Bidirectional Representations Augmented Autoregressive Biological Sequence Generation

arXiv: https://arxiv.org/abs/2510.08169 · HTML: https://mlsanity.com/paper/2510.08169/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 49 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Bridging the Gap between Training and Inference for Neural Machine Translation](https://mlsanity.com/paper/1906.02448/) (2019)
- [If beam search is the answer, what was the question?](https://mlsanity.com/paper/2010.02650/) (2020)
- [Glancing Transformer for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2008.07905/) (2020)
- [FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow](https://mlsanity.com/paper/1909.02480/) (2019)
- [From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery](https://mlsanity.com/paper/2508.14111/) (2025)
- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoning](https://mlsanity.com/paper/2506.10521/) (2025)
- [Why Prompt Design Matters and Works: A Complexity Analysis of Prompt Search Space in LLMs](https://mlsanity.com/paper/2503.10084/) (2025)
- [Uncovering Memorization Effect in the Presence of Spurious Correlations](https://mlsanity.com/paper/2501.00961/) (2025)
- [A Character-Level Length-Control Algorithm for Non-Autoregressive Sentence Summarization](https://mlsanity.com/paper/2205.14522/) (2022)
- [ContraNovo: A Contrastive Learning Approach to Enhance De Novo Peptide Sequencing](https://mlsanity.com/paper/2312.11584/) (2023)
- [AdaNovo: Adaptive De Novo Peptide Sequencing with Conditional Mutual Information](https://mlsanity.com/paper/2403.07013/) (2024)
- [Universal Biological Sequence Reranking for Improved De Novo Peptide Sequencing](https://mlsanity.com/paper/2505.17552/) (2025)
- [Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing](https://mlsanity.com/paper/2506.13485/) (2025)

## Built on by (most cited first, 1 of 1)

- [Reflection Pretraining Enables Token-Level Self-Correction in Biological Sequence Models](https://mlsanity.com/paper/2512.20954/) (2025)

## Abstract (arXiv)

Autoregressive (AR) models, common in sequence generation, are limited in many biological tasks such as de novo peptide sequencing and protein modeling by their unidirectional nature, failing to capture crucial global bidirectional token dependencies. Non-Autoregressive (NAR) models offer holistic, bidirectional representations but face challenges with generative coherence and scalability. To transcend this, we propose a hybrid framework enhancing AR generation by dynamically integrating rich contextual information from non-autoregressive mechanisms. Our approach couples a shared input encoder with two decoders: a non-autoregressive one learning latent bidirectional biological features, and an AR decoder synthesizing the biological sequence by leveraging these bidirectional features. A novel cross-decoder attention module enables the AR decoder to iteratively query and integrate these bidirectional features, enriching its predictions. This synergy is cultivated via a tailored training strategy with importance annealing for balanced objectives and cross-decoder gradient blocking for stable, focused learning. Evaluations on a demanding nine-species benchmark of de novo peptide sequencing show that our model substantially surpasses AR and NAR baselines. It uniquely harmonizes AR stability with NAR contextual awareness, delivering robust, superior performance on diverse downstream data. This research advances biological sequence modeling techniques and contributes a novel architectural paradigm for augmenting AR models with enhanced bidirectional understanding for complex sequence generation. Code is available at https://github.com/BEAM-Labs/denovo.