mlsanityData to 7 Oct 2026

IBNorm: Information-Bottleneck Inspired Normalization for Representation Learning

arXiv 2510.25262 ↗First posted 29 Oct 2025cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Normalization is fundamental to deep learning, but existing approaches such as BatchNorm, LayerNorm, and RMSNorm are variance-centric by enforcing zero mean and unit variance, stabilizing training without controlling how representations capture task-relevant information. We propose IB-Inspired Normalization (IBNorm), a simple yet powerful family of methods grounded in the Information Bottleneck principle. IBNorm introduces bounded compression operations that encourage embeddings to preserve predictive information while suppressing nuisance variability, yielding more informative representations while retaining the stability and compatibility of standard normalization. Theoretically, we prove that IBNorm achieves a higher IB value and tighter generalization bounds than variance-centric methods. Empirically, IBNorm consistently outperforms BatchNorm, LayerNorm, and RMSNorm across large-scale language models (LLaMA, GPT-2) and vision models (ResNet, ViT), with mutual information analysis confirming superior information bottleneck behavior. Code will be released publicly.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 54 of its 76 references that are in the map. First posted on arXiv 29 Oct 2025.

Builds on

54 of 76 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2024The Llama 3 Herd of Modelscited by 14,192
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  10. 2016Layer Normalizationcited by 5,579
  11. 2022A ConvNet for the 2020scited by 3,545
  12. 2023Qwen Technical Reportcited by 3,273
38 more
  1. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2018Group Normalizationcited by 1,514
  4. 2019Root Mean Square Layer Normalizationcited by 1,425
  5. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  6. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  7. 2016Deep Variational Information Bottleneckcited by 915
  8. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  9. 2018MINE: Mutual Information Neural Estimationcited by 680
  10. 2017Exploring Generalization in Deep Learningcited by 595
  11. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  12. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  13. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  14. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  15. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  16. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
  17. 2019Understanding and Improving Layer Normalizationcited by 213
  18. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  19. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 189
  20. 2017Modulating early visual processing by languagecited by 167
  21. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  22. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 131
  23. 2017Network Representation Learning: A Surveycited by 101
  24. 2019Iterative Normalization: Beyond Standardization towards Efficient Whiteningcited by 92
  25. 2023How Does Information Bottleneck Help Deep Learning?cited by 92
  26. 2020Normalization Techniques in Training DNNs: Methodology, Analysis and Applicationcited by 90
  27. 2019A Mutual Information Maximization Perspective of Language Representation Learningcited by 85
  28. 2018Decorrelated Batch Normalizationcited by 83
  29. 2017Nonlinear Information Bottleneckcited by 71
  30. 2020The Information Bottleneck Problem and Its Applications in Machine Learningcited by 66
  31. 2018Entropy and mutual information in models of deep neural networkscited by 50
  32. 2018Generalization Error in Deep Learningcited by 46
  33. 2015Natural Neural Networkscited by 44
  34. 2019Four Things Everyone Should Know to Improve Batch Normalizationcited by 28
  35. 2018Whitening and Coloring batch transform for GANscited by 27
  36. 2017Dynamic Layer Normalization for Adaptive Neural Acoustic Modeling in Speech Recognitioncited by 14
  37. 2023On the impact of activation and normalization in obtaining isometric embeddings at initializationcited by 13
  38. 2023Modulate Your Spectrum in Self-Supervised Learningcited by 9

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Raw-Routed Mixture of Adapters: A Causal Intervention for Routing Collapse in Time Series Foundation Modelscited by 0