# The Arbitrary-Placement Problem in Entropy-Minimizing Selection, and a Residual-Entropy Formulation

arXiv: https://arxiv.org/abs/2610.05925 · HTML: https://mlsanity.com/paper/2610.05925/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 18 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Builds on (most cited first, 15 of 15)

- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Understanding and Robustifying Differentiable Architecture Search](https://mlsanity.com/paper/1909.09656/) (2019)
- [Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy](https://mlsanity.com/paper/2310.01334/) (2023)
- [Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions](https://mlsanity.com/paper/1903.03088/) (2019)
- [Self-Supervised Classification Network](https://mlsanity.com/paper/2103.10994/) (2021)
- [On Partial Prototype Collapse in the DINO Family of Self-Supervised Methods](https://mlsanity.com/paper/2410.14060/) (2024)

## Abstract (arXiv)

Entropy-based selection objectives suffer from a fundamental degeneracy: minimizing Shannon entropy H(pA) rewards confident selection regardless of whether the selected candidate is informative. We address this limitation with the residual entropy D = H(pA) - H(pβ), where pβ is induced by candidate trust weights. We prove the exact identity D = -KL(pAVert pβ) - Delta, where Delta measures whether the score-induced distribution and trust profile favor the same candidates. Boundary cases establish basic safety: under uniform trust, D≤0 automatically, so an equal-trust, non-starving state is never penalized, while at any one-hot limit, D→0 regardless of the selected candidate. For the intermediate regime where selection occurs, we prove that D≤0 when candidate ordering by trust agrees pairwise with ordering by informativeness, and derive a tighter certificate based on the leading candidate's margin over its competitors. These results are independent of the candidate-scoring function and apply to both stationary and dynamically changing information. Experiments with a gradient-based mixture-of-experts router confirm that the ordering conditions can hold during real optimization and show that correct ordering improves downstream performance when candidates are non-interchangeable and selections are used directly rather than averaged. Beyond routing, margin-based reweighting matches or outperforms fixed-strength baselines in a class-imbalance task, while informative selection in a production video-prediction system reduces MSE by approximately 20% and transfers to a related species. Residual entropy, therefore, provides a safety criterion for selection and a usable signal for deciding when that selection is informative.