mlsanityData to 7 Oct 2026

Auditing Routing Entropy as an Uncertainty Signal in Attention-Residual Transformers

arXiv 2610.01495 ↗First posted 1 Oct 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Dynamic architectures leave a per-example routing trace beside each prediction, and diffuse routing is easy to read as a sign that the prediction is unreliable. We audit that reading for routing entropy in Attention-Residual (AR) variants of Swin-Tiny and DeiT-Small, trained from scratch on CIFAR-10/100 with a soft-binned calibration auxiliary loss, asking whether the trace carries information about correctness beyond what the model's own confidence already reveals. Three checks probe this increment: does a routing signal appear at fixed confidence, does it replicate across training seeds, and can a held-out predictor exploit it against output-only and shuffled-trace controls? A sensitivity audit then injects effects of known size and measures the fraction of each that the probes recover. No test in the fixed 30-test binned family survives multiplicity correction, and neither the nominal hit nor a borderline result recurs in its sibling seeds. Across 24 paired runs a scalar routing probe yields no pooled improvement in routing-stratified calibration, and an entropy-profile probe predicts correctness better than the same probe given shuffled profiles yet worse than a confidence-only predictor in both binary log-loss and Brier score: a gain over shuffled traces does not become a gain over the output. Conditioning on the complete logit vector leaves the corresponding comparison unresolved. The audit bounds how far these non-detections can be read: at an injected effect of 0.010 nats the profile probe recovers 24-59% of the oracle gain, and a reference-preserving correction probe recovers 8% and 23% in the two CIFAR-100 settings, below the threshold we fixed for applying it to real labels. The results establish control-dependent gains and incomplete estimator recovery, not the absence of conditional routing information.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

27 of 36 references.

References · 27

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  9. 2017Mixed Precision Trainingcited by 1,135
  10. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  11. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  12. 2019Designing and Interpreting Probes with Control Taskscited by 608
15 more
  1. 2019Measuring Calibration in Deep Learningcited by 443
  2. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
  3. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  4. 2019Verified Uncertainty Calibrationcited by 307
  5. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  6. 2018To Trust Or Not To Trust A Classifiercited by 205
  7. 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
  8. 2019Addressing Failure Prediction by Learning Model Confidencecited by 183
  9. 2020A Theory of Usable Information Under Computational Constraintscited by 153
  10. 2020Mitigating Bias in Calibration Error Estimationcited by 100
  11. 2021Soft Calibration Objectives for Neural Networkscited by 89
  12. 2021Conditional probing: measuring usable information beyond a baselinecited by 61
  13. 2023Smooth ECE: Principled Reliability Diagrams via Kernel Smoothingcited by 59
  14. 2026Attention Residualscited by 57
  15. 2021Parameterized Temperature Scaling for Boosting the Expressive Power in Post-Hoc Uncertainty Calibrationcited by 48

Built on by

Not cited yet.