# DataMUX: Data Multiplexing for Neural Networks

arXiv: https://arxiv.org/abs/2202.09318 · HTML: https://mlsanity.com/paper/2202.09318/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 18 Feb 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers](https://mlsanity.com/paper/1811.04918/) (2018)
- [Supermasks in Superposition](https://mlsanity.com/paper/2006.14769/) (2020)
- [Proving the Lottery Ticket Hypothesis: Pruning is All You Need](https://mlsanity.com/paper/2002.00585/) (2020)
- [UniT: Multimodal Multitask Learning with a Unified Transformer](https://mlsanity.com/paper/2102.10772/) (2021)
- [Training independent subnetworks for robust prediction](https://mlsanity.com/paper/2010.06610/) (2020)
- [Superposition of many models into one](https://mlsanity.com/paper/1902.05522/) (2019)
- [Overparameterized Neural Networks Implement Associative Memory](https://mlsanity.com/paper/1909.12362/) (2019)
- [MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks](https://mlsanity.com/paper/2103.06132/) (2021)
- [MUXConv: Information Multiplexing in Convolutional Neural Networks](https://mlsanity.com/paper/2003.13880/) (2020)
- [Aggregated Learning: A Vector-Quantization Approach to Learning Neural Network Classifiers](https://mlsanity.com/paper/2001.03955/) (2020)

## Built on by (most cited first, 18 of 18)

- [RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs](https://mlsanity.com/paper/2404.08555/) (2024)
- [OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts](https://mlsanity.com/paper/2503.22952/) (2025)
- [SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding](https://mlsanity.com/paper/2412.09604/) (2024)
- [Block Transformer: Global-to-Local Language Modeling for Fast Inference](https://mlsanity.com/paper/2406.02657/) (2024)
- [MIMONets: Multiple-Input-Multiple-Output Neural Networks Exploiting Computation in Superposition](https://mlsanity.com/paper/2312.02829/) (2023)
- [Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition](https://mlsanity.com/paper/2410.05603/) (2024)
- [MOSEL: Inference Serving Using Dynamic Modality Selection](https://mlsanity.com/paper/2310.18481/) (2023)
- [MUX-PLMs: Data Multiplexing for High-throughput Language Models](https://mlsanity.com/paper/2302.12441/) (2023)
- [PruMUX: Augmenting Data Multiplexing with Model Compression](https://mlsanity.com/paper/2305.14706/) (2023)
- [Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules](https://mlsanity.com/paper/2310.15724/) (2023)
- [RevMUX: Data Multiplexing with Reversible Adapters for Efficient LLM Batch Inference](https://mlsanity.com/paper/2410.04519/) (2024)
- [ConcatPlexer: Additional Dim1 Batching for Faster ViTs](https://mlsanity.com/paper/2308.11199/) (2023)
- [SAE: Single Architecture Ensemble Neural Networks](https://mlsanity.com/paper/2402.06580/) (2024)
- [Compressive Meta-Learning](https://mlsanity.com/paper/2508.11090/) (2025)
- [Semantic Multiplexing](https://mlsanity.com/paper/2511.13779/) (2025)
- [Superposed Latent Autoencoder](https://mlsanity.com/paper/2609.01158/) (2026)
- [Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs](https://mlsanity.com/paper/2609.29845/) (2026)
- [Superposed Inference for Hyperdimensional Computing](https://mlsanity.com/paper/2609.32320/) (2026)

## Abstract (arXiv)

In this paper, we introduce data multiplexing (DataMUX), a technique that enables deep neural networks to process multiple inputs simultaneously using a single compact representation. DataMUX demonstrates that neural networks are capable of generating accurate predictions over mixtures of inputs, resulting in increased throughput with minimal extra memory requirements. Our approach uses two key components -- 1) a multiplexing layer that performs a fixed linear transformation to each input before combining them to create a mixed representation of the same size as a single input, which is then processed by the base network, and 2) a demultiplexing layer that converts the base network's output back into independent representations before producing predictions for each input. We show the viability of DataMUX for different architectures (Transformers, and to a lesser extent MLPs and CNNs) across six different tasks spanning sentence classification, named entity recognition and image classification. For instance, DataMUX for Transformers can multiplex up to 20x/40x inputs, achieving 11x/18x increase in throughput with minimal absolute performance drops of <2% and <4% respectively on MNLI, a natural language inference task. We also provide a theoretical construction for multiplexing in self-attention networks and analyze the effect of various design elements in DataMUX.