mlsanityData to 7 Oct 2026

SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processors

arXiv 2411.17847 ↗First posted 26 Nov 2024cs.AR cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Recent research efforts focus on reducing the computational and memory overheads of Large Language Models (LLMs) to make them feasible on resource-constrained devices. Despite advancements in compression techniques, non-linear operators like Softmax and Layernorm remain bottlenecks due to their sensitivity to quantization. We propose SoftmAP, a software-hardware co-design methodology that implements an integer-only low-precision Softmax using In-Memory Compute (IMC) hardware. Our method achieves up to three orders of magnitude improvement in the energy-delay product compared to A100 and RTX3090 GPUs, making LLMs more deployable without compromising performance.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 26 of its 40 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

26 of 40 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2023A Survey of Large Language Modelscited by 2,537
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
14 more
  1. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  2. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  3. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  4. 2018Online normalizer calculation for softmaxcited by 154
  5. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  6. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  7. 2021Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformerscited by 52
  8. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  9. 2023Softmax Bias Correction for Quantized Generative Modelscited by 15
  10. 2024ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameterscited by 14
  11. 2023Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantizationcited by 13
  12. 2022In-memory Associative Processors: Tutorial, Potential, and Challengescited by 6
  13. 2024Efficiently Distilling LLMs for Edge Applicationscited by 5
  14. 2024BF-IMNA: A Bit Fluid In-Memory Neural Architecture for Neural Network Accelerationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Exploring the Dynamic Scheduling Space of Real-Time Generative AI Applications on Emerging Heterogeneous Systemscited by 1
  2. 2026Mugi: Value Level Parallelism For Efficient LLMscited by 0
  3. 2026Hardware-Efficient Softmax and Layer Normalization with Guaranteed Normalization for Edge Devicescited by 0