# Hamming Attention Distillation: Binarizing Keys and Queries for Efficient Long-Context Transformers

arXiv: https://arxiv.org/abs/2502.01770 · HTML: https://mlsanity.com/paper/2502.01770/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Feb 2025.

## Built on by (most cited first, 2 of 2)

- [CAMformer: Associative Memory is All You Need](https://mlsanity.com/paper/2511.19740/) (2025)
- [Latest Exact Match Attention](https://mlsanity.com/paper/2609.25802/) (2026)

## Abstract (arXiv)

Pre-trained transformer models with extended context windows are notoriously expensive to run at scale, often limiting real-world deployment due to their high computational and memory requirements. In this paper, we introduce Hamming Attention Distillation (HAD), a novel framework that binarizes keys and queries in the attention mechanism to achieve significant efficiency gains. By converting keys and queries into -1, +1 vectors and replacing dot-product operations with efficient Hamming distance computations, our method drastically reduces computational overhead. Additionally, we incorporate attention matrix sparsification to prune low-impact activations, which further reduces the cost of processing long-context sequences. par Despite these aggressive compression strategies, our distilled approach preserves a high degree of representational power, leading to substantially improved accuracy compared to prior transformer binarization methods. We evaluate HAD on a range of tasks and models, including the GLUE benchmark, ImageNet, and QuALITY, demonstrating state-of-the-art performance among binarized Transformers while drastically reducing the computational costs of long-context inference. par We implement HAD in custom hardware simulations, demonstrating superior performance characteristics compared to a custom hardware implementation of standard attention. HAD achieves just 1.78% performance losses on GLUE compared to 9.08% in state-of-the-art binarization work, and 2.5% performance losses on ImageNet compared to 12.14%, all while targeting custom hardware with a 79% area reduction and 87% power reduction compared to its standard attention counterpart.