# ORBIT-FMIB: Tracking Order-Resolved Epistatic Information Through ESM-2

arXiv: https://arxiv.org/abs/2610.00672 · HTML: https://mlsanity.com/paper/2610.00672/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 24 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Builds on (most cited first, 4 of 4)

- [A distributional simplicity bias in the learning dynamics of transformers](https://mlsanity.com/paper/2410.19637/) (2024)
- [On Recovering Higher-order Interactions from Protein Language Models](https://mlsanity.com/paper/2405.06645/) (2024)
- [BRIDGE: Bottleneck-Aware Regulator-Set Inference and Diagnosis for Cooperative Gene Regulatory Recovery](https://mlsanity.com/paper/2607.18602/) (2026)
- [Beyond Tokens: Probing Higher-Order Epistasis in Learned Protein Representations](https://mlsanity.com/paper/2608.24953/) (2026)

## Abstract (arXiv)

Protein foundation models support mutation-effect and structural prediction, but predictive performance alone does not reveal which forms of biological interaction information remain accessible through model depth. We ask whether ESM-2 retains higher-order epistatic information as strongly as first- and second-order information across its representation hierarchy, introducing ORBIT-FMIB, a diagnostic framework combining Walsh-based interaction decomposition with subset-conditioned neural dependence estimation. The method is validated on synthetic landscapes with known interaction structure before being applied to the dense four-site GB1 fitness landscape using frozen ESM-2 representations. An initial production run suggested ESM-2 retains higher-order epistatic information less well than lower-order information (DeltaHO-LO=-0.107). An independent replication of the complete measurement grid, under matched GPU hardware and identical critic seeds, substantially reduced this contrast (DeltaHO-LO=-0.017), and its sign was unstable across otherwise-defensible evaluation-pairing choices applied to the same trained critics (-0.011 to +0.015). We therefore do not currently have robust evidence that ESM-2 selectively loses higher-order epistatic information, nor that retention is equal across orders; the directional question remains open. The measurement protocol itself, including its documented removal of a positional-subset shortcut in pooled critics, remains validated and is unaffected by this finding. ORBIT-FMIB is offered as a diagnostic framework for probing interaction structure in protein foundation models; this study's own replication result illustrates why such probing requires adequately-powered reproducibility checks before its output is treated as a biological finding.