mlsanityData to 7 Oct 2026

Rethinking the Harmonic Loss via Non-Euclidean Distance Layers

arXiv 2603.10225 ↗First posted 10 Mar 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Cross-entropy loss has long been the standard choice for training deep neural networks, yet it suffers from interpretability limitations, unbounded weight growth, and inefficiencies that can contribute to costly training dynamics. The harmonic loss is a distance-based alternative grounded in Euclidean geometry that improves interpretability and mitigates phenomena such as grokking, or delayed generalization on the test set. However, the study of harmonic loss remains narrow: only Euclidean distance is explored, and no systematic evaluation of computational efficiency or sustainability was conducted. We extend harmonic loss by systematically investigating a broad spectrum of distance metrics as replacements for the Euclidean distance. We comprehensively evaluate distance-tailored harmonic losses on both vision backbones and large language models. Our analysis is framed around a three-way evaluation of model performance, interpretability, and sustainability. On vision tasks, cosine distances provide the most favorable trade-off, consistently improving accuracy while lowering carbon emissions, whereas Bray-Curtis and Mahalanobis further enhance interpretability at varying efficiency costs. On language models, cosine-based harmonic losses improve gradient and learning stability, strengthen representation structure, and reduce emissions relative to cross-entropy and Euclidean heads. Our code is available at: https://anonymous.4open.science/r/rethinking-harmonic-loss-5BAB/.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 61 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

25 of 61 references.

References · 25

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  4. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  5. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  6. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  7. 2018CosFace: Large Margin Cosine Loss for Deep Face Recognitioncited by 968
  8. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  9. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  10. 2020A Survey on Neural Network Interpretabilitycited by 291
  11. 2017Interpretable Convolutional Neural Networkscited by 161
  12. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
13 more
  1. 2020Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descentcited by 118
  2. 2020A unifying mutual information view of metric learning: cross-entropy vs. pairwise lossescited by 100
  3. 2017On Loss Functions for Deep Neural Networks in Classificationcited by 68
  4. 2017Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networkscited by 58
  5. 2021Orthogonal Projection Losscited by 44
  6. 2019Improved Training Speed, Accuracy, and Data Utilization Through Loss Function Optimizationcited by 28
  7. 2020A Study of Gradient Variance in Deep Learningcited by 27
  8. 2018Max-Mahalanobis Linear Discriminant Analysis Networkscited by 18
  9. 2017Improving Interpretability of Deep Neural Networks with Semantic Informationcited by 14
  10. 2020AMC-Loss: Angular Margin Contrastive Loss for Improved Explainability in Image Classificationcited by 14
  11. 2023Energy and Carbon Considerations of Fine-Tuning BERTcited by 11
  12. 2025Harmonic Loss Trains Interpretable AI Modelscited by 4
  13. 2025Compressing Large Language Models with PCA Without Performance Losscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Loss Functions and Metrics in Deep Learning
  2. 2025Spectral Alignment as Predictor of Loss Explosion in Neural Network Training
  3. 2026Understanding Pruning Regimes in Vision-Language Models Through Domain-Aware Layer Selection
  4. 2026Beyond Compression: Quantifying Spectral Accessibility in Vision Representations
  5. 2026Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling
  6. 2025AME: Aligned Manifold Entropy for Robust Vision-Language Distillation
  7. 2026R'enyi Entropy: A New Token Pruning Metric for Vision Transformers
  8. 2025High-Entropy Tokens as Multimodal Failure Points in Vision-Language Models