Learned Queries and Keys Are All You Need: Replacing the Value Projection with Structured Transforms
Not on the map. Why?
Abstract (arXiv)
To reduce the number of parameters and cache memory requirements of transformers we introduce dual-headed transformers instead of three heads. We studied Walsh-Hadamard Transform (WHT), Discrete Cosine Transform (DCT), Discrete Fourier Transform, filterbank based Shearlet Transform, and Multiplication-Avoiding (MA) operators to construct dual heads. We combine spatial patches and their orthogonal transforms (or Shearlet and MA operators) in a structure similar to the attention block. We obtained better results than triple headed transformers in ImageNet. Extensive simulation examples are presented.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 7 of its 21 references that are in the map. First posted on arXiv 29 Sep 2026.
Builds on
7 of 21 references.
References · 7
- 2017Attention Is All You Needcited by 60,361
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021Transformers in Vision: A Surveycited by 1,013
- 2021MF-Net: Compute-In-Memory SRAM for Multibit Precision Inference using Memory-immersed Data Conversion and Multiplication-free Operatorscited by 12
- 2023A Hybrid Quantum-Classical Approach based on the Hadamard Transform for the Convolutional Layercited by 12
- 2024Discrete Cosine Transform Based Decorrelated Attention for Vision Transformerscited by 2
- 2025HTMA-Net: Towards Multiplication-Avoiding Neural Networks via Hadamard Transform and In-Memory Computingcited by 1
Built on by
Not cited yet.