# Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks

arXiv: https://arxiv.org/abs/1906.04393 · HTML: https://mlsanity.com/paper/1906.04393/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2019.

## Built on by (most cited first, 26 of 26)

- [Compacter: Efficient Low-Rank Hypercomplex Adapter Layers](https://mlsanity.com/paper/2106.04647/) (2021)
- [Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers](https://mlsanity.com/paper/2109.10686/) (2021)
- [Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameters](https://mlsanity.com/paper/2102.08597/) (2021)
- [PHNNs: Lightweight Neural Networks via Parameterized Hypercomplex Convolutions](https://mlsanity.com/paper/2110.04176/) (2021)
- [Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings](https://mlsanity.com/paper/2103.02886/) (2021)
- [Quaternion-valued Correlation Learning for Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2305.07283/) (2023)
- [Parameterized Hypercomplex Graph Neural Networks for Graph Classification](https://mlsanity.com/paper/2103.16584/) (2021)
- [Learning Speech Emotion Representations in the Quaternion Domain](https://mlsanity.com/paper/2204.02385/) (2022)
- [Paramixer: Parameterizing Mixing Links in Sparse Factors Works Better than Dot-Product Self-Attention](https://mlsanity.com/paper/2204.10670/) (2022)
- [Quaternion Factorization Machines: A Lightweight Solution to Intricate Feature Interaction Modelling](https://mlsanity.com/paper/2104.01716/) (2021)
- [DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration](https://mlsanity.com/paper/2506.11104/) (2025)
- [Two-view Graph Neural Networks for Knowledge Graph Completion](https://mlsanity.com/paper/2112.09231/) (2021)
- [AI Progress Should Be Measured by Capability-Per-Resource, Not Scale Alone: A Framework for Gradient-Guided Resource Allocation in LLMs](https://mlsanity.com/paper/2511.01077/) (2025)
- [Adapting Pre-trained Language Models for Quantum Natural Language Processing](https://mlsanity.com/paper/2302.13812/) (2023)
- [Deconstructing Attention: Investigating Design Principles for Effective Language Modeling](https://mlsanity.com/paper/2510.11602/) (2025)
- [Demystifying the Hypercomplex: Inductive Biases in Hypercomplex Deep Learning](https://mlsanity.com/paper/2405.07024/) (2024)
- [Improving Quaternion Neural Networks with Quaternionic Activation Functions](https://mlsanity.com/paper/2406.16481/) (2024)
- [Quaternion Self-Attention with Shared Scores](https://mlsanity.com/paper/2605.24920/) (2026)
- [Efficient Attribute Injection for Pretrained Language Models](https://mlsanity.com/paper/2109.07953/) (2021)
- [Quaternion-Based Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2111.10536/) (2021)
- [Experimental Results regarding multiple Machine Learning via Quaternions](https://mlsanity.com/paper/2308.01946/) (2023)
- [An Effective, Robust and Fairness-aware Hate Speech Detection Framework](https://mlsanity.com/paper/2409.17191/) (2024)
- [Downsized and Compromised?: Assessing the Faithfulness of Model Compression](https://mlsanity.com/paper/2510.06125/) (2025)
- [QC-GAN: A Parameter-Efficient Quaternion Conformer GAN for High-Fidelity Speech Enhancement](https://mlsanity.com/paper/2606.18611/) (2026)
- [Bridge the Gaps: Heterogeneous Attributed Graph Clustering via Quaternion Representation Learning](https://mlsanity.com/paper/2606.23199/) (2026)
- [QSV: Quat-Sphere-Vision for Coupled Quaternion Attention on Spherical Lattices](https://mlsanity.com/paper/2609.30592/) (2026)

## Abstract (arXiv)

Many state-of-the-art neural models for NLP are heavily parameterized and thus memory inefficient. This paper proposes a series of lightweight and memory efficient neural architectures for a potpourri of natural language processing (NLP) tasks. To this end, our models exploit computation using Quaternion algebra and hypercomplex spaces, enabling not only expressive inter-component interactions but also significantly (75%) reduced parameter size due to lesser degrees of freedom in the Hamilton product. We propose Quaternion variants of models, giving rise to new architectures such as the Quaternion attention Model and Quaternion Transformer. Extensive experiments on a battery of NLP tasks demonstrates the utility of proposed Quaternion-inspired models, enabling up to 75% reduction in parameter size without significant loss in performance.