# Correcting the Dropout-LayerNorm Expectation Gap Improves Protein Structure Models

arXiv: https://arxiv.org/abs/2609.32062 · HTML: https://mlsanity.com/paper/2609.32062/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 33 references that are in the map.
- First posted on arXiv 25 Sep 2026.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Generating Novel, Designable, and Diverse Protein Structures by Equivariantly Diffusing Oriented Residue Clouds](https://mlsanity.com/paper/2301.12485/) (2023)
- [Improved motif-scaffolding with SE(3) flow matching](https://mlsanity.com/paper/2401.04082/) (2024)
- [Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift](https://mlsanity.com/paper/1801.05134/) (2018)
- [Out of Many, One: Designing and Scaffolding Proteins at the Scale of the Structural Universe with Genie 2](https://mlsanity.com/paper/2405.15489/) (2024)
- [ABodyBuilder3: Improved and scalable antibody structure predictions](https://mlsanity.com/paper/2405.20863/) (2024)
- [Conformation-Aware Structure Prediction of Antigen-Recognizing Immune Proteins](https://mlsanity.com/paper/2507.09054/) (2025)
- [QuickBind: A Light-Weight And Interpretable Molecular Docking Model](https://mlsanity.com/paper/2410.16474/) (2024)

## Abstract (arXiv)

Although mathbbE[dropout(x)] = x, here we show that mathbbE[LayerNorm(dropout(x))] is not equal to LayerNorm(x). Accordingly, the pattern of a Dropout layer followed by a LayerNorm, which is common to many AlphaFold2-based protein structure predictors, produces a systematic bias at evaluation time that can hamper performance. To address this, we derive a closed-form, first-order correction for this gap, which we call a Dropout-LayerNorm Correction (DLC). DLC empirically matches the performance boost of large Monte Carlo dropout ensembles. We evaluate its effect across nine protein structure models (ESMFold, OpenFold, ABB3, FlashABB, Ibex, NbForge, Genie1, Genie2, Genie3) on both paired and single-chain antibody structures as well as one protein-ligand docking model (QuickBind). The correction is computationally negligible and improves accuracy in all ten models tested (~0.3%-13%), with a modest but consistent improvement in ESMFold and OpenFold and a substantial improvement in antibody-specific models. This work identifies the mathematical consequence of chaining together Dropout and LayerNorm and provides a free, principled adjustment to improve the evaluation performance of many pretrained models. Code to reproduce the experiments is available at https://github.com/oxpig/DLC.