# A Shared Encoder Is Not a Shared Task: Conditional Comparison for Deep Expert Pools

arXiv: https://arxiv.org/abs/2609.27866 · HTML: https://mlsanity.com/paper/2609.27866/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 8 references that are in the map.
- First posted on arXiv 21 Aug 2026.

## Builds on (most cited first, 8 of 8)

- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Out-of-Distribution Detection with Deep Nearest Neighbors](https://mlsanity.com/paper/2204.06507/) (2022)
- [Generalized Category Discovery](https://mlsanity.com/paper/2201.02609/) (2022)
- [Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning](https://mlsanity.com/paper/2403.18886/) (2024)
- [Quantifying Data Similarity Using Cross Learning](https://mlsanity.com/paper/2510.10866/) (2025)
- [Separating Covariate Shift from Mechanism Change with Two Discriminators: CJSD, a Conditional Discrepancy with an Exact Covariate-Concept Decomposition](https://mlsanity.com/paper/2608.19885/) (2026)

## Abstract (arXiv)

Sharing a deep encoder does not, by itself, fix the central confound of task-comparison scores. We show that cross-evaluated heads on a frozen shared representation inherit the extrapolation confound of shallow exchange scores: pure input rotations with fixed labels inflate a deep exchange score from about 0 to 0.80, while representation-novelty scores are blind in the complementary direction (flat under label permutations that change the task completely). Transplanting a conditional two-discriminator discrepancy into the embedding space resolves both blind spots: the functional axis stays within +-0.001 under rotations and tracks label-permutation drift mass monotonically. Built into a mixture-of-heads lifecycle, the two-axis gate attains better decision quality with fewer heads than exchange or novelty triggers at a matched training budget. On generalized category discovery, the same chunk-level functional axis separates semantic novelty from photometric shift with AUROC 0.98-0.99 where per-input OOD scores (MSP, Energy, Mahalanobis, KNN) sit near chance for that distinction. All findings replicate across frozen ImageNet-21k ViT-B/16 and self-supervised DINOv2 backbones on CIFAR-100, and extend to residual adapter pools with recurrence, where a null-calibrated novelty trigger never fires on mechanism changes while the two-axis gate handles them with full recurrence reuse. We state explicitly the common-factoring condition under which embedding-space conclusions transfer to the original mechanism.