# One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective

arXiv: https://arxiv.org/abs/2109.14449 · HTML: https://mlsanity.com/paper/2109.14449/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 54 references that are in the map.
- First posted on arXiv 29 Sep 2021.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrieval](https://mlsanity.com/paper/2004.01804/) (2020)
- [Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking](https://mlsanity.com/paper/1803.11285/) (2018)
- [Return of the Devil in the Details: Delving Deep into Convolutional Nets](https://mlsanity.com/paper/1405.3531/) (2014)
- [HashNet: Deep Learning to Hash by Continuation](https://mlsanity.com/paper/1702.00758/) (2017)
- [A Survey on Learning to Hash](https://mlsanity.com/paper/1606.00185/) (2016)
- [Central Similarity Quantization for Efficient Image and Video Retrieval](https://mlsanity.com/paper/1908.00347/) (2019)
- [A Decade Survey of Content Based Image Retrieval using Deep Learning](https://mlsanity.com/paper/2012.00641/) (2020)
- [Feature Learning based Deep Supervised Hashing with Pairwise Labels](https://mlsanity.com/paper/1511.03855/) (2015)
- [Simultaneous Feature Learning and Hash Coding with Deep Neural Networks](https://mlsanity.com/paper/1504.03410/) (2015)
- [Learning to Hash for Indexing Big Data - A Survey](https://mlsanity.com/paper/1509.05472/) (2015)
- [Deep Supervised Discrete Hashing](https://mlsanity.com/paper/1705.10999/) (2017)
- [Deep Supervised Hashing with Triplet Labels](https://mlsanity.com/paper/1612.03900/) (2016)
- [Deep Semantic Ranking Based Hashing for Multi-Label Image Retrieval](https://mlsanity.com/paper/1501.06272/) (2015)
- [Deep Unsupervised Image Hashing by Maximizing Bit Entropy](https://mlsanity.com/paper/2012.12334/) (2020)
- [Deep Triplet Quantization](https://mlsanity.com/paper/1902.00153/) (2019)
- [Supervised Online Hashing via Hadamard Codebook Learning](https://mlsanity.com/paper/1905.03694/) (2019)
- [Embarrassingly Simple Binary Representation Learning](https://mlsanity.com/paper/1908.09573/) (2019)
- [Neural Networks Weights Quantization: Target None-retraining Ternary (TNT)](https://mlsanity.com/paper/1912.09236/) (2019)
- [A General Two-Step Approach to Learning-Based Hashing](https://mlsanity.com/paper/1309.1853/) (2013)

## Built on by (most cited first, 30 of 43)

- [FashionViL: Fashion-Focused Vision-and-Language Representation Learning](https://mlsanity.com/paper/2207.08150/) (2022)
- [Prototypical Hash Encoding for On-the-Fly Fine-Grained Category Discovery](https://mlsanity.com/paper/2410.19213/) (2024)
- [COMAE: COMprehensive Attribute Exploration for Zero-shot Hashing](https://mlsanity.com/paper/2402.16424/) (2024)
- [SEMICON: A Learning-to-hash Solution for Large-scale Fine-grained Image Retrieval](https://mlsanity.com/paper/2209.13833/) (2022)
- [HashVFL: Defending Against Data Reconstruction Attacks in Vertical Federated Learning](https://mlsanity.com/paper/2212.00325/) (2022)
- [Deep Hash Distillation for Image Retrieval](https://mlsanity.com/paper/2112.08816/) (2021)
- [One Loss for Quantization: Deep Hashing with Discrete Wasserstein Distributional Matching](https://mlsanity.com/paper/2205.15721/) (2022)
- [Attribute-Aware Deep Hashing with Self-Consistency for Large-Scale Fine-Grained Image Retrieval](https://mlsanity.com/paper/2311.12894/) (2023)
- [PMI Sampler: Patch Similarity Guided Frame Selection for Aerial Action Recognition](https://mlsanity.com/paper/2304.06866/) (2023)
- [Making a Bird AI Expert Work for You and Me](https://mlsanity.com/paper/2112.02747/) (2021)
- [Hijack Vertical Federated Learning Models As One Party](https://mlsanity.com/paper/2212.00322/) (2022)
- [Semantic-Aware Adversarial Training for Reliable Deep Hashing Retrieval](https://mlsanity.com/paper/2310.14637/) (2023)
- [Towards Cross-modal Backward-compatible Representation Learning for Vision-Language Models](https://mlsanity.com/paper/2405.14715/) (2024)
- [MITFAS: Mutual Information based Temporal Feature Alignment and Sampling for Aerial Video Action Recognition](https://mlsanity.com/paper/2303.02575/) (2023)
- [HyP2 Loss: Beyond Hypersphere Metric Space for Multi-label Image Retrieval](https://mlsanity.com/paper/2208.06866/) (2022)
- [Constrained Multiview Representation for Self-supervised Contrastive Learning](https://mlsanity.com/paper/2402.03456/) (2024)
- [Deep Hashing with Semantic Hash Centers for Image Retrieval](https://mlsanity.com/paper/2507.08404/) (2025)
- [Mutual Learning for Hashing: Unlocking Strong Hash Functions from Weak Supervision](https://mlsanity.com/paper/2510.07703/) (2025)
- [CgAT: Center-Guided Adversarial Training for Deep Hashing-Based Retrieval](https://mlsanity.com/paper/2204.10779/) (2022)
- [A Lower Bound of Hash Codes' Performance](https://mlsanity.com/paper/2210.05899/) (2022)
- [Unsupervised Hashing with Similarity Distribution Calibration](https://mlsanity.com/paper/2302.07669/) (2023)
- [Deep Hashing via Householder Quantization](https://mlsanity.com/paper/2311.04207/) (2023)
- [Sobolev neural network with residual weighting as a surrogate in linear and non-linear mechanics](https://mlsanity.com/paper/2407.16466/) (2024)
- [Nested Hash Layer: A Plug-and-play Module for Multiple-length Hash Code Learning](https://mlsanity.com/paper/2412.08922/) (2024)
- [Image Hashing via Cross-View Code Alignment in the Age of Foundation Models](https://mlsanity.com/paper/2510.27584/) (2025)
- [Spruce: Scalable Private Outsourced Retrieval Using Compact Embeddings](https://mlsanity.com/paper/2609.03376/) (2026)
- [CoopHash: Cooperative Learning of Multipurpose Descriptor and Contrastive Pair Generator via Variational MCMC Teaching for Supervised Image Hashing](https://mlsanity.com/paper/2210.04288/) (2022)
- [ConceptHash: Interpretable Fine-Grained Hashing via Concept Discovery](https://mlsanity.com/paper/2406.08457/) (2024)
- [Learning Attribute-Aware Hash Codes for Fine-Grained Image Retrieval via Query Optimization](https://mlsanity.com/paper/2509.17049/) (2025)
- [Codebook-Centric Deep Hashing: End-to-End Joint Learning of Semantic Hash Centers and Neural Hash Function](https://mlsanity.com/paper/2511.12162/) (2025)

## Abstract (arXiv)

A deep hashing model typically has two main learning objectives: to make the learned binary hash codes discriminative and to minimize a quantization error. With further constraints such as bit balance and code orthogonality, it is not uncommon for existing models to employ a large number (>4) of losses. This leads to difficulties in model training and subsequently impedes their effectiveness. In this work, we propose a novel deep hashing model with only a single learning objective. Specifically, we show that maximizing the cosine similarity between the continuous codes and their corresponding binary orthogonal codes can ensure both hash code discriminativeness and quantization error minimization. Further, with this learning objective, code balancing can be achieved by simply using a Batch Normalization (BN) layer and multi-label classification is also straightforward with label smoothing. The result is an one-loss deep hashing model that removes all the hassles of tuning the weights of various losses. Importantly, extensive experiments show that our model is highly effective, outperforming the state-of-the-art multi-loss hashing models on three large-scale instance retrieval benchmarks, often by significant margins. Code is available at https://github.com/kamwoh/orthohash