# VEXP: A Low-Cost RISC-V ISA Extension for Accelerated Softmax Computation in Transformers

arXiv: https://arxiv.org/abs/2504.11227 · HTML: https://mlsanity.com/paper/2504.11227/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 15 Apr 2025.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers](https://mlsanity.com/paper/2103.09301/) (2021)
- [NN-LUT: Neural Approximation of Non-Linear Operations for Efficient Transformer Inference](https://mlsanity.com/paper/2112.02191/) (2021)
- [ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameters](https://mlsanity.com/paper/2402.10930/) (2024)
- [ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers](https://mlsanity.com/paper/2307.03493/) (2023)
- [Snitch: A tiny Pseudo Dual-Issue Processor for Area and Energy Efficient Execution of Floating-Point Intensive Workloads](https://mlsanity.com/paper/2002.10143/) (2020)
- [Stream Semantic Registers: A Lightweight RISC-V ISA Extension Achieving Full Compute Utilization in Single-Issue Cores](https://mlsanity.com/paper/1911.08356/) (2019)
- [Hyft: A Reconfigurable Softmax Accelerator with Hybrid Numeric Format for both Training and Inference](https://mlsanity.com/paper/2311.13290/) (2023)
- [GPTQT: Quantize Large Language Models Twice to Push the Efficiency](https://mlsanity.com/paper/2407.02891/) (2024)
- [MiniFloat-NN and ExSdotp: An ISA Extension and a Modular Open Hardware Unit for Low-Precision Training on RISC-V cores](https://mlsanity.com/paper/2207.03192/) (2022)
- [A Flexible Template for Edge Generative AI with High-Accuracy Accelerated Softmax & GELU](https://mlsanity.com/paper/2412.06321/) (2024)
- [Optimizing Foundation Model Inference on a Many-tiny-core Open-source RISC-V Platform](https://mlsanity.com/paper/2405.19284/) (2024)
- [Occamy: A 432-Core 28.1 DP-GFLOP/s/W 83% FPU Utilization Dual-Chiplet, Dual-HBM2E RISC-V-based Accelerator for Stencil and Sparse Linear Algebra Computations with 8-to-64-bit Floating-Point Support in 12nm FinFET](https://mlsanity.com/paper/2406.15068/) (2024)

## Built on by (most cited first, 4 of 4)

- [Vectorized FlashAttention with Low-cost Exponential Computation in RISC-V Vector Processors](https://mlsanity.com/paper/2510.06834/) (2025)
- [Integrated electro-optic attention nonlinearities for transformers](https://mlsanity.com/paper/2604.09512/) (2026)
- [FlashAttention for Scalable Vector Architectures](https://mlsanity.com/paper/2608.18656/) (2026)
- [RISC-V and machine learning: a survey](https://mlsanity.com/paper/2609.20677/) (2026)

## Abstract (arXiv)

While Transformers are dominated by Floating-Point (FP) Matrix-Multiplications, their aggressive acceleration through dedicated hardware or many-core programmable systems has shifted the performance bottleneck to non-linear functions like Softmax. Accelerating Softmax is challenging due to its non-pointwise, non-linear nature, with exponentiation as the most demanding step. To address this, we design a custom arithmetic block for Bfloat16 exponentiation leveraging a novel approximation algorithm based on Schraudolph's method, and we integrate it into the Floating-Point Unit (FPU) of the RISC-V cores of a compute cluster, through custom Instruction Set Architecture (ISA) extensions, with a negligible area overhead of 1%. By optimizing the software kernels to leverage the extension, we execute Softmax with 162.7× less latency and 74.3× less energy compared to the baseline cluster, achieving an 8.2× performance improvement and 4.1× higher energy efficiency for the FlashAttention-2 kernel in GPT-2 configuration. Moreover, the proposed approach enables a multi-cluster system to efficiently execute end-to-end inference of pre-trained Transformer models, such as GPT-2, GPT-3 and ViT, achieving up to 5.8× and 3.6× reduction in latency and energy consumption, respectively, without requiring re-training and with negligible accuracy loss.