# Entangled by Design: Spurious Intra-Variable Signal Routing in Tabular In-Context Learners

arXiv: https://arxiv.org/abs/2607.25532 · HTML: https://mlsanity.com/paper/2607.25532/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 28 Jul 2026.

## Builds on (most cited first, 12 of 12)

- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Improve Unsupervised Domain Adaptation with Mixup Training](https://mlsanity.com/paper/2001.00677/) (2020)
- [Contrastive representations of high-dimensional, structured treatments](https://mlsanity.com/paper/2411.19245/) (2024)

## Abstract (arXiv)

Consider a model trained at a single hospital to predict patient recovery, where the measured feature X bundles the patient's true health signal (C) with a systematic artefact from that hospital's equipment (S). Within that hospital, the artefact correlates with outcomes through unmeasured confounders such as patient demographics; an in-context learner rationally routes predictions through S, not C, and fails silently when deployed at a new hospital with different equipment. We formalise this as spurious routing in composite representations: when a feature X = [C;,α S;,η] encodes a causal signal C and a spurious signal S in distinct subspaces, the ICL cannot determine which drives predictions. We prove that under ridge ICL, a linear in-context learner, this routing is unavoidable regardless of context size; TabPFN, a state-of-the-art pretrained tabular ICL model, shows qualitatively consistent behaviour empirically. We derive a closed-form characterisation, CSR propto ρS/ρC, confirmed at r = 0.997 for linear ICL and r = 0.979 for TabPFN. Contrary to intuition, larger context sharpens commitment to the dominant in-context signal, amplifying spurious routing by up to 1.74×; in the high-spurious corner, more expressive models show greater vulnerability empirically (+2.22 CSR gap at high entanglement). We introduce two lightweight mitigations: environment-stratified context construction and S-swap augmentation, that require only weak environment labels and no knowledge of the causal partition. S-swap reduces spurious routing by 74% for linear ICL and 98.8% for TabPFN, with TabPFN's causal sensitivity increasing 8.4× simultaneously: the model does not become agnostic, it reroutes through the causal signal.