# TMA-Adaptive FP8 Grouped GEMM: Eliminating Padding Requirements in Low-Precision Training and Inference on Hopper

arXiv: https://arxiv.org/abs/2508.16584 · HTML: https://mlsanity.com/paper/2508.16584/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2025.

## Built on by (most cited first, 4 of 4)

- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Intern-S1: A Scientific Multimodal Foundation Model](https://mlsanity.com/paper/2508.15763/) (2025)
- [SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations](https://mlsanity.com/paper/2512.14080/) (2025)
- [Token-Operations-Oriented Inference Optimization Techniques for Large Models](https://mlsanity.com/paper/2606.20295/) (2026)

## Abstract (arXiv)

Current FP8 grouped GEMM implementations require padding each group to a fixed alignment (e.g., 128), incurring memory and computational overhead. We propose textitTMA-Adaptive FP8 Grouped GEMM, which eliminates padding by dynamically adapting to variable group dimensions via (1) a TMA descriptor pool with log2(blockM) preconfigured descriptors to handle all residual row cases through dynamic runtime selection and dual-phase load-store operations, achieving comprehensive coverage with minimal overhead, and (2) TMA-alignment-aware management to satisfy 16-byte global memory alignment and 128-byte shared memory alignment. Experiments demonstrate 1.7% to 20.4% speed up with up to 23.8% memory reduction compared to padding operation plus state-of-the-art FP8 grouped GEMM, while maintaining full numerical equivalence for valid data. The source code is publicly available at an anonymous repository: https://github.com/sukoncon/TMA-Adaptive-FP8-Grouped-GEMM.