Probing Routing-Conditional Calibration in Attention-Residual Transformers
Not on the map. Why?
Abstract (arXiv)
Post-hoc calibration is usually evaluated as a function of logits or softmax confidence alone, even as routing-augmented architectures increasingly accompany predictions with sample-specific internal routing traces and pair them with claims of calibration-relevant uncertainty. We ask a basic question: do these traces provide stable routing-specific evidence for post-hoc calibration beyond confidence? We study this in Attention-Residual transformers (Kimi Team, 2026) through a matched-confidence diagnostic suite that stratifies examples by routing-derived state, compares subgroup gaps against within-bin routing-permutation nulls, and evaluates matched post-hoc probes differing only in their auxiliary feature. Across our completed AR runs, scalar routing summaries do not provide stable evidence of routing-conditional miscalibration: weighted gaps remain small or seed-sensitive, and only 1 of 30 within-bin permutation tests rejects the conditional-null at α=0.05 (only on one seed; not stable across seeds in that cell). AR-CondCal, a minimal 2-D Nadaraya--Watson probe on confidence and routing-depth variance, lies within the seed-variance band of matched confidence-only and predictive-entropy controls and does not reliably improve worst-routing-tertile ECE; bandwidth-sensitivity checks (Scott multiples, CV-NLL, global-ECE oracle) do not change this. A full-vector MLP over (c, H1, ldots, HL) can appear to improve over a linear confidence baseline, but the apparent gain disappears once a capacity-matched confidence-only MLP is included as a control, and shuffled routing profiles achieve comparable performance. Apparent routing-aware calibration gains in this AR setting should not be read as internal-state calibration until matched-confidence, bandwidth, capacity, and permutation controls rule out common confounds.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 11 May 2026.
Builds on
27 of 37 references.
References · 27
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
- 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
- 2017Mixed Precision Trainingcited by 1,156
- 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
15 more
- 2019Measuring Calibration in Deep Learningcited by 446
- 2021Revisiting the Calibration of Modern Neural Networkscited by 390
- 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
- 2020Calibrating Deep Neural Networks using Focal Losscited by 357
- 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 332
- 2019Verified Uncertainty Calibrationcited by 309
- 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 305
- 2022Mitigating Neural Network Overconfidence with Logit Normalizationcited by 252
- 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
- 2021Efficient Training of Visual Transformers with Small Datasetscited by 108
- 2020Mitigating Bias in Calibration Error Estimationcited by 101
- 2021Soft Calibration Objectives for Neural Networkscited by 92
- 2026Attention Residualscited by 65
- 2023Smooth ECE: Principled Reliability Diagrams via Kernel Smoothingcited by 63
- 2021Parameterized Temperature Scaling for Boosting the Expressive Power in Post-Hoc Uncertainty Calibrationcited by 48
Built on by
Citing papers per year: 2026: 1.
1 paper.