mlsanityData to 7 Oct 2026

Auditing Invisible Weight Updates with Reference Traces

arXiv 2607.09800 ↗First posted 9 Jul 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Direct low-precision write-back can erase nonzero optimizer proposals. We ask what a high-precision reference trace establishes before a low-precision run. The exact target-code event is auditable coordinatewise on a realized target trajectory; pre-run aggregate projection also assumes the reference remains a useful counterfactual. In a controlled two-layer grid, 55/72 cells have measured and predicted post-initialization crossings: times span 384×, 52/55 are within 15%, and 4/72 differ in category. Matched decoder experiments show stochastic rather than nearest write-back recovers most of the loss gap. A prospective analytic-grid E4M3 audit reuses one fp32 trace across three unseen NeoX-style seeds. It passes absolute-accuracy and skill gates (macro RMSE 0.00858) but fails directional specificity. In a target-outcome-blind comparison, a historical template has lower descriptive RMSE (0.00360) than the predeclared source predictor (0.00438); a post-outcome decomposition assigns 99.65% of variation to common time, while a privileged matched-reference correction reaches 0.00283. Persistent-native Study~1 pairs three seeds across two schedules. Five cells are canonical; a manual sixth lacks canonical process identity, so the registered result remains inconclusive. A retrospective protocol-deviation analysis is negative because the complete constant-mid cohort is disjoint from the recovered cosine-restart cell. Study~2 reports mean full-SR/dead-zone-SR recoveries of 0.9766/0.9777 and a ratio of 1.0012, a policy contrast rather than causal mediation. Simulated-INT3 Study~3 replays six checkpoints and observes a 7.3071-nat (69.71%) validation-loss reduction in one fixed seed. Exact events and write-back effects are auditable, but aggregate forecasts can reflect shared time rather than source-specific transfer.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 51 references that are in the map. First posted on arXiv 9 Jul 2026.

Builds on

32 of 51 references.

References · 32

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2020GLU Variants Improve Transformercited by 1,724
  3. 2019Root Mean Square Layer Normalizationcited by 1,425
  4. 2017Mixed Precision Trainingcited by 1,135
  5. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  6. 2015Deep Learning with Limited Numerical Precisioncited by 368
  7. 2022FP8 Formats for Deep Learningcited by 187
  8. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
  9. 2023FP8-LM: Training FP8 Large Language Modelscited by 75
  10. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 37
  11. 2025Training LLMs with MXFP4cited by 36
  12. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 32
20 more
  1. 2020FracTrain: Fractionally Squeezing Bit Savings Both Temporally and Spatially for Efficient DNN Trainingcited by 23
  2. 2023Unit Scaling: Out-of-the-Box Low-Precision Trainingcited by 17
  3. 2026Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimationcited by 16
  4. 2020Revisiting BFloat16 Trainingcited by 15
  5. 2025Why Low-Precision Transformer Training Fails: An Analysis on Flash Attentioncited by 11
  6. 2025Stochastic Rounding for LLM Training: Theory and Practicecited by 10
  7. 2024Speeding up and reducing memory usage for scientific machine learning via mixed precisioncited by 8
  8. 2026Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attentioncited by 7
  9. 2026Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resetscited by 6
  10. 2024Direct Quantized Training of Language Models with Stochastic Roundingcited by 5
  11. 2026Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RLcited by 5
  12. 2026Pretraining large language models with MXFP4 on Native FP4 Hardwarecited by 5
  13. 2026STQuant: Spatio-Temporal Adaptive Framework for Optimizer Quantization in Large Multimodal Model Trainingcited by 4
  14. 2026ECO: Quantized Training without Full-Precision Master Weightscited by 3
  15. 2026Model Compression with Exact Budget Constraints via Riemannian Manifoldscited by 2
  16. 2026dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formatscited by 2
  17. 2026Quantizing Time-Series Models As Dynamical Systems: Trajectory-Based Quantization Sensitivity Scorecited by 2
  18. 2026FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Trainingcited by 2
  19. 2025Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Roundingcited by 1
  20. 2026Budget-aware Auto Optimizer Configuratorcited by 1

Built on by

Not cited yet.