mlsanityData to 14 Oct 2026

Reparameterized LLM Training via Orthogonal Equivalence Transformation

arXiv 2506.08001 ↗First posted 9 Jun 2025cs.LG cs.AI cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

While large language models (LLMs) are driving the rapid advancement of artificial intelligence, effectively and reliably training these large models remains one of the field's most significant challenges. To address this challenge, we propose POET, a novel reParameterized training algorithm that uses Orthogonal Equivalence Transformation to optimize neurons. Specifically, POET reparameterizes each neuron with two learnable orthogonal matrices and a fixed random weight matrix. Because of its provable preservation of spectral properties of weight matrices, POET can stably optimize the objective function with improved generalization. We further develop efficient approximations that make POET flexible and scalable for training large-scale neural networks. Extensive experiments validate the effectiveness and scalability of POET in training LLMs.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 60 of its 83 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

60 of 83 references.

References · 60

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  9. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  10. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  11. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  12. 2017Deep Neural Networks as Gaussian Processescited by 622
38 more
  1. 2017Self-Normalizing Neural Networkscited by 598
  2. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  3. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  4. 2025Muon is Scalable for LLM Trainingcited by 428
  5. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  6. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  7. 2017Implicit Regularization in Matrix Factorizationcited by 322
  8. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  9. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 293
  10. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 208
  11. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  12. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  13. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 182
  14. 2019Complexity of Linear Regions in Deep Networkscited by 167
  15. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  16. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  17. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 130
  18. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 125
  19. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 114
  20. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 106
  21. 2017Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networkscited by 92
  22. 2025Practical Efficiency of Muon for Pretrainingcited by 80
  23. 2019Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizationscited by 76
  24. 2018Learning towards Minimum Hyperspherical Energycited by 75
  25. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 70
  26. 2017Deep Hyperspherical Learningcited by 55
  27. 2018Decoupled Networkscited by 45
  28. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 45
  29. 2020Orthogonal Over-Parameterized Trainingcited by 44
  30. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 44
  31. 2020A case for new neural network smoothness constraintscited by 43
  32. 2024Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?cited by 43
  33. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 39
  34. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 38
  35. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 38
  36. 2021Learning with Hyperspherical Uniformitycited by 35
  37. 2022The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancycited by 35
  38. 2016Diverse Neural Network Learns True Target Functionscited by 33

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 8.

9 papers.

  1. 2025Orthogonal Finetuning Made Scalablecited by 7
  2. 2026Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformationcited by 6
  3. 2026POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformationcited by 5
  4. 2026Orthogonal Model Mergingcited by 4
  5. 2026ISO: An RLVR-Native Optimization Stackcited by 2
  6. 2026CORA: Per-Slice Coherent Orthogonal Rotation for SVD-based Low-Rank Adaptationcited by 1
  7. 2026LoCO: Low-rank Compositional Rotation Fine-tuningcited by 0
  8. 2026Efficient Pre-Training of LLMs through Truncated SVD Representationscited by 0
  9. 2026ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Trainingcited by 0