# Is SwiGLU's Open Positive Tail Necessary? Evidence from Closed-Tail Gating with MemGLU

arXiv: https://arxiv.org/abs/2608.07323 · HTML: https://mlsanity.com/paper/2608.07323/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 7 Aug 2026.

## Builds on (most cited first, 12 of 12)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Scaling FP8 training to trillion-token LLMs](https://mlsanity.com/paper/2409.12517/) (2024)
- [CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models](https://mlsanity.com/paper/2404.08763/) (2024)
- [Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models](https://mlsanity.com/paper/2411.03884/) (2024)
- [Deriving Activation Functions Using Integration](https://mlsanity.com/paper/2411.13010/) (2024)
- [Expanded Gating Ranges Improve Activation Functions](https://mlsanity.com/paper/2405.20768/) (2024)
- [PowLU: An Activation Function for Stable Pre-Training of LLMs](https://mlsanity.com/paper/2605.25704/) (2026)
- [More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations](https://mlsanity.com/paper/2605.26647/) (2026)

## Abstract (arXiv)

We test whether decoder-only language-model FFNs require SwiGLU's open positive tail. We introduce MemGLU as a closed-tail comparator derived from a memristive branch geometry. Across paired 9M and 30M pretraining runs with three seeds, MemGLU remains within about 0.1% of SwiGLU in validation NLL. Trained SwiGLU checkpoints are sensitive to positive-tail suppression, while mechanism diagnostics show that the two models use their gates differently despite similar losses. These results suggest that models adapt to the gate geometry available during pretraining. At the tested scales, SwiGLU's open positive tail is not necessary for decoder-only language-model FFNs.