# TriPLU: Bypassing the Gate with Direct Trilinear Product FFNs in Tiny Language Models

arXiv: https://arxiv.org/abs/2608.20360 · HTML: https://mlsanity.com/paper/2608.20360/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [Neural Arithmetic Logic Units](https://mlsanity.com/paper/1808.00508/) (2018)
- [Neural Arithmetic Units](https://mlsanity.com/paper/2001.05016/) (2020)
- [Neural Power Units](https://mlsanity.com/paper/2006.01681/) (2020)
- [A Primer for Neural Arithmetic Logic Modules](https://mlsanity.com/paper/2101.09530/) (2021)
- [Dissecting Multiplication in Transformers: Insights into LLMs](https://mlsanity.com/paper/2407.15360/) (2024)
- [PowLU: An Activation Function for Stable Pre-Training of LLMs](https://mlsanity.com/paper/2605.25704/) (2026)
- [More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations](https://mlsanity.com/paper/2605.26647/) (2026)
- [Sample-Efficient Language Modeling with Linear Attention and Lightweight Enhancements](https://mlsanity.com/paper/2511.05560/) (2025)
- [PolyGLU: State-Conditional Activation Routing in Transformer Feed-Forward Networks](https://mlsanity.com/paper/2603.13347/) (2026)
- [Modeling Nonlinear Feature Interactions with Product-Unit Residual Networks](https://mlsanity.com/paper/2606.06861/) (2026)

## Abstract (arXiv)

We study whether tiny decoder-only language models benefit from feed-forward layers that directly multiply learned feature projections. TriPLU, a Trilinear Product Linear Unit, replaces the usual gated FFN branch with a product-only degree-3 branch that multiplies three projected streams coordinatewise. In a character-level TinyStories 1M-byte prefix study, TriPLU reaches a mean best validation loss of 1.0637, compared with 1.1017 for closely matched SwiGLU, 1.0780 for a degree-4 product control, and 1.1026 for a degree-2 control. In train-only Byte-BPE experiments, TriPLU also lowers validation and heldout bits per byte on TinyStories and WikiText-2 raw under low-learning-rate settings, with PMI-slice evidence suggesting gains on seen middle- and high-PMI adjacent-token pairs. Constant-learning-rate diagnostics show that product-branch normalization can reduce the high-learning-rate best-checkpoint gap, although final BPB still degrades under hot schedules. The resulting claim is deliberately narrow: direct product FFNs can improve fixed-budget small-model loss in specific low-compute regimes, but the branch is optimization-sensitive and does not establish FLOP-normalized efficiency, scaling behavior, or broad LLM performance.