# Compute-in-Memory Attention: A Time-Domain Analog Softmax Circuit with RC-Tunable Temperature

arXiv: https://arxiv.org/abs/2609.04266 · HTML: https://mlsanity.com/paper/2609.04266/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 25 references that are in the map.
- First posted on arXiv 2 Sep 2026.

## Builds on (most cited first, 5 of 5)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers](https://mlsanity.com/paper/2103.09301/) (2021)
- [Analog In-Memory Computing Attention Mechanism for Fast and Energy-Efficient Large Language Models](https://mlsanity.com/paper/2409.19315/) (2024)
- [Hyft: A Reconfigurable Softmax Accelerator with Hybrid Numeric Format for both Training and Inference](https://mlsanity.com/paper/2311.13290/) (2023)
- [GEM3D CIM General Purpose Matrix Computation Using 3D Integrated SRAM eDRAM Hybrid Compute In Memory on Memory Architecture](https://mlsanity.com/paper/2604.13969/) (2026)

## Abstract (arXiv)

Softmax is a key operation in Transformer attention, but its exponentiation and normalization add significant overhead in compute-in-memory (CIM) accelerators, especially when analog attention scores must first be converted to the digital domain. This work presents a tunable-temperature analog softmax circuit in GlobalFoundries 22-nm fully depleted silicon-on-insulator (FDSOI) technology that operates directly on CIM-generated score voltages without intermediate analog-to-digital conversion. Each input score is converted into a time-domain event using a shared falling ramp. The corresponding comparator transition samples an RC-decaying reference to generate an exponential weight, which is then processed by an in-circuit normalization stage. In contrast to analog softmax circuits that rely on transistor weak-inversion behavior for exponentiation, the proposed architecture controls the softmax response through the ramp slope and RC time constant, enabling programmable effective temperature. The 128-element architecture is evaluated using transistor-level and post-layout extracted simulations, including multi-level input vectors, capacitance variation and mismatch, process and temperature variation, monte carlo analysis, and shared-interconnect parasitics. The complete 128-element implementation occupies 9453.42~μm2 including the shared global ramp circuitry, while each replicated softmax element occupies 70.2~μm2. The circuit achieves a 242.97-ns evaluation latency at 13.44~mW total power, corresponding to 25.5~pJ per output element. The simultaneous 128-element evaluation achieves an RMSE of 24.46~mV relative to the ideal softmax response. The extracted circuit characteristics are further incorporated into a MemTorch-based hardware-aware Transformer model, where the proposed softmax achieves a validation loss within 2.5% of the ideal-softmax baseline.