Auditing Routing Entropy as an Uncertainty Signal in Attention-Residual Transformers
Not on the map. Why?
Abstract (arXiv)
Dynamic architectures leave a per-example routing trace beside each prediction, and diffuse routing is easy to read as a sign that the prediction is unreliable. We audit that reading for routing entropy in Attention-Residual (AR) variants of Swin-Tiny and DeiT-Small, trained from scratch on CIFAR-10/100 with a soft-binned calibration auxiliary loss, asking whether the trace carries information about correctness beyond what the model's own confidence already reveals. Three checks probe this increment: does a routing signal appear at fixed confidence, does it replicate across training seeds, and can a held-out predictor exploit it against output-only and shuffled-trace controls? A sensitivity audit then injects effects of known size and measures the fraction of each that the probes recover. No test in the fixed 30-test binned family survives multiplicity correction, and neither the nominal hit nor a borderline result recurs in its sibling seeds. Across 24 paired runs a scalar routing probe yields no pooled improvement in routing-stratified calibration, and an entropy-profile probe predicts correctness better than the same probe given shuffled profiles yet worse than a confidence-only predictor in both binary log-loss and Brier score: a gain over shuffled traces does not become a gain over the output. Conditioning on the complete logit vector leaves the corresponding comparison unresolved. The audit bounds how far these non-detections can be read: at an injected effect of 0.010 nats the profile probe recovers 24-59% of the oracle gain, and a reference-preserving correction probe recovers 8% and 23% in the two CIFAR-100 settings, below the threshold we fixed for applying it to real labels. The results establish control-dependent gains and incomplete estimator recovery, not the absence of conditional routing information.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 1 Oct 2026.
Builds on
27 of 36 references.
References · 27
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
- 2017Mixed Precision Trainingcited by 1,135
- 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
- 2023The Internal State of an LLM Knows When It's Lyingcited by 644
- 2019Designing and Interpreting Probes with Control Taskscited by 608
15 more
- 2019Measuring Calibration in Deep Learningcited by 443
- 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
- 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
- 2019Verified Uncertainty Calibrationcited by 307
- 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
- 2018To Trust Or Not To Trust A Classifiercited by 205
- 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
- 2019Addressing Failure Prediction by Learning Model Confidencecited by 183
- 2020A Theory of Usable Information Under Computational Constraintscited by 153
- 2020Mitigating Bias in Calibration Error Estimationcited by 100
- 2021Soft Calibration Objectives for Neural Networkscited by 89
- 2021Conditional probing: measuring usable information beyond a baselinecited by 61
- 2023Smooth ECE: Principled Reliability Diagrams via Kernel Smoothingcited by 59
- 2026Attention Residualscited by 57
- 2021Parameterized Temperature Scaling for Boosting the Expressive Power in Post-Hoc Uncertainty Calibrationcited by 48
Built on by
Not cited yet.