mlsanityData to 14 Oct 2026

DP-Muon: Differentially Private Optimization via Matrix-Orthogonalized Momentum

arXiv 2605.12994 ↗First posted 13 May 2026cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

We study differentially private optimization with matrix-orthogonalized momentum. DP-Muon uses conventional global per-example clipping and one Gaussian gradient release per step; matrix updates and auxiliary updates are post-processing. Our main contribution concerns the additional mean distortion created when fresh Gaussian noise passes through a nonlinear matrix map. Conditioning on the actual adaptive history immediately before the current noise yields an exact Gaussian heat identity. For a smooth Newton-Schulz map, first-order DP-MuonBC reduces this conditional output bias from second to fourth order in the fresh noise scale, and an arbitrary-order extension has bias of order 2K+2. We prove matrix-block stationarity bounds under global clipping, retain finite-step orthogonalization error explicitly, and give an exact criterion for improvement of the resulting upper bound. A separate inequality exposes the effect of auxiliary Adam updates. GPT-2 experiments on E2E at four privacy targets favor the reported Muon configurations over Adam baselines in test NLL.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 18 of its 33 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

18 of 33 references.

References · 18

  1. 2016Deep Learning with Differential Privacycited by 2,728
  2. 2017Renyi Differential Privacycited by 650
  3. 2025Muon is Scalable for LLM Trainingcited by 428
  4. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  5. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  6. 2014RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Responsecited by 272
  7. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
  8. 2018Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergencescited by 242
  9. 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 236
  10. 2019Differentially Private Learning with Adaptive Clippingcited by 220
  11. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 204
  12. 2017Collecting Telemetry Data Privatelycited by 149
6 more
  1. 2018Differentially Private Empirical Risk Minimization Revisited: Faster and More Generalcited by 128
  2. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 90
  3. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 88
  4. 2019AdaCliP: Adaptive Clipping for Private SGDcited by 77
  5. 2026Convergence of Muon with Newton-Schulzcited by 36
  6. 2023DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction)cited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026The Spectral Dynamics and Noise Geometry of Muoncited by 3
  2. 2026Inner Momentum for Differentially Private Muoncited by 0