# A Geometric Analysis of Sign-Magnitude Asymmetry in a ReLU + RMSNorm Block under Ternary Quantization

arXiv: https://arxiv.org/abs/2605.18933 · HTML: https://mlsanity.com/paper/2605.18933/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 18 May 2026.

## Builds on (most cited first, 15 of 15)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [BitNet: Scaling 1-bit Transformers for Large Language Models](https://mlsanity.com/paper/2310.11453/) (2023)
- [Find A Winning Sign: Sign Is All We Need to Win the Lottery](https://mlsanity.com/paper/2504.05357/) (2025)
- [Disentangling Direction and Magnitude in Transformer Representations: A Double Dissociation Through L2-Matched Perturbation Analysis](https://mlsanity.com/paper/2602.11169/) (2026)

## Abstract (arXiv)

Pre-norm Transformers with RMSNorm tolerate ternary -1,0,+1 weight quantization with surprisingly small loss (Ma et al., 2024). We give a geometric explanation via sign-magnitude decomposition of weight perturbations. In a two-layer ReLU + RMSNorm model with i.i.d. Gaussian weights, sign-flips produce π/(π-2) ≈ 2.75 times more transverse output energy than sign-preserving magnitude perturbations of equal Frobenius norm, as the flip rate p → 0 (Theorem 3). The mechanism: ReLU creates a hidden-space directional asymmetry between the two perturbation types, which RMSNorm's transverse-projection Fr'echet derivative selectively exposes. Sign-quantization error is itself a sign-preserving perturbation with angular alignment cos2 → 2/π (Theorem 4); its post-ReLU radial fraction (0.365) matches the pre-ReLU value 1-2/π within 0.4%, so ReLU is approximately transparent to ternary error. Multi-layer compounding of the 2.75× factor is not experimentally supported; the gap to real-model sign sensitivity arises from outlier features violating delocalization. For an input dimension with amplitude α, a single sign-flip produces post-ReLU energy amplified by R ≈ nα2 relative to a delocalized entry. On TinyLlama-1.1B, at linear response (p ≤ 0.5%), count-matched NLL leverage stabilizes at ~ 10× ≈ nmathbbE[α2], matching the per-entry theory; the all-column NLL ratio of 5.0× falls within Rcol ≤ 19 (67× PPL gap reflects metric nonlinearity). Measured outlier α at layer 12 (median 0.024, max 0.26) confirms heavy-tailed concentration. The Bussgang constant 2/π, RMSNorm geometry, and ReLU half-space structure together explain sign-magnitude asymmetry in pre-norm models, with R propto nα2 accounting for real-model deviations.