# Beyond Tokens: Probing Higher-Order Epistasis in Learned Protein Representations

arXiv: https://arxiv.org/abs/2608.24953 · HTML: https://mlsanity.com/paper/2608.24953/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 14 references that are in the map.
- First posted on arXiv 24 Aug 2026.

## Builds on (most cited first, 4 of 4)

- [A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree Spectral Bias of Neural Networks](https://mlsanity.com/paper/2305.09779/) (2023)
- [On Recovering Higher-order Interactions from Protein Language Models](https://mlsanity.com/paper/2405.06645/) (2024)
- [Augmenting Biological Fitness Prediction Benchmarks with Landscapes Features from GraphFLA](https://mlsanity.com/paper/2510.24826/) (2025)
- [BRIDGE: Bottleneck-Aware Regulator-Set Inference and Diagnosis for Cooperative Gene Regulatory Recovery](https://mlsanity.com/paper/2607.18602/) (2026)

## Built on by (most cited first, 1 of 1)

- [ORBIT-FMIB: Tracking Order-Resolved Epistatic Information Through ESM-2](https://mlsanity.com/paper/2610.00672/) (2026)

## Abstract (arXiv)

Protein fitness landscapes contain nonlinear interactions in which mutation effects depend on other residues. We introduce ORBIT, an Order-Resolved Benchmarking of Interaction Transformations framework that separates interaction presence, representation accessibility, and functional recovery. ORBIT first validates Walsh-based diagnostics on synthetic landscapes with known interaction order, then analyzes the experimentally measured GB1 fitness landscape under the FLIP 2-vs-rest setting. We compare ridge regression, a standard MLP, independent tokens, nonlinear independent tokens, and Residual Interaction Tokenization (RIT). Across 20 paired training seeds, the primary two-hidden-layer comparison found no significant architecture differences in FLIP test R^2, third- or fourth-order functional recovery, or final-layer third- or fourth-order accessibility. However, RIT significantly increased pairwise accessibility at the token stage relative to both independent-token controls (Delta A_tok,2 = 0.2468, d_z = 1.67, Holm-adjusted p = 1.14 x 10^-5), without a detectable downstream higher-order advantage. A pre-specified depth/capacity analysis showed that deeper MLPs improved FLIP prediction, third-order functional recovery, and final-layer third-order accessibility; fourth-order accessibility also improved relative to the shallow MLP but remained below zero in absolute held-out R^2. ORBIT therefore reveals representation-level changes hidden by conventional prediction metrics and distinguishes early interaction-aware encoding from higher-order structure constructed by downstream nonlinear capacity.