# Heated-Up Softmax Embedding

arXiv: https://arxiv.org/abs/1809.04157 · HTML: https://mlsanity.com/paper/1809.04157/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2018.

## Built on by (most cited first, 15 of 15)

- [DER: Dynamically Expandable Representation for Class Incremental Learning](https://mlsanity.com/paper/2103.16788/) (2021)
- [Temperature as Uncertainty in Contrastive Learning](https://mlsanity.com/paper/2110.04403/) (2021)
- [Transformer-based Language Model Fine-tuning Methods for COVID-19 Fake News Detection](https://mlsanity.com/paper/2101.05509/) (2021)
- [Provable Lifelong Learning of Representations](https://mlsanity.com/paper/2110.14098/) (2021)
- [To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO](https://mlsanity.com/paper/2404.04575/) (2024)
- [On the Importance of Embedding Norms in Self-Supervised Learning](https://mlsanity.com/paper/2502.09252/) (2025)
- [Embed and Emulate: Learning to estimate parameters of dynamical systems with uncertainty quantification](https://mlsanity.com/paper/2211.01554/) (2022)
- [Hyperspherical Embedding for Point Cloud Completion](https://mlsanity.com/paper/2307.05634/) (2023)
- [Robust Interaction-Based Relevance Modeling for Online e-Commerce Search](https://mlsanity.com/paper/2406.02135/) (2024)
- [Learning Natural Language Generation from Scratch](https://mlsanity.com/paper/2109.09371/) (2021)
- [Text analysis and deep learning: A network approach](https://mlsanity.com/paper/2110.04151/) (2021)
- [Fine-tune your Classifier: Finding Correlations With Temperature](https://mlsanity.com/paper/2210.09715/) (2022)
- [Probability-Dependent Gradient Decay in Large Margin Softmax](https://mlsanity.com/paper/2210.17145/) (2022)
- [Analytical Softmax Temperature Setting from Feature Dimensions for Model- and Domain-Robust Classification](https://mlsanity.com/paper/2504.15594/) (2025)
- [Optimization Dynamics Imprint Semantic Specificity in Contrastive Embedding Norms](https://mlsanity.com/paper/2606.30625/) (2026)

## Abstract (arXiv)

Metric learning aims at learning a distance which is consistent with the semantic meaning of the samples. The problem is generally solved by learning an embedding for each sample such that the embeddings of samples of the same category are compact while the embeddings of samples of different categories are spread-out in the feature space. We study the features extracted from the second last layer of a deep neural network based classifier trained with the cross entropy loss on top of the softmax layer. We show that training classifiers with different temperature values of softmax function leads to features with different levels of compactness. Leveraging these insights, we propose a "heating-up" strategy to train a classifier with increasing temperatures, leading the corresponding embeddings to achieve state-of-the-art performance on a variety of metric learning benchmarks.