# Controlling Computation versus Quality for Neural Sequence Models

arXiv: https://arxiv.org/abs/2002.07106 · HTML: https://mlsanity.com/paper/2002.07106/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Feb 2020.

## Built on by (most cited first, 27 of 27)

- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Confident Adaptive Language Modeling](https://mlsanity.com/paper/2207.07061/) (2022)
- [Mixture-of-Depths: Dynamically allocating compute in transformer-based language models](https://mlsanity.com/paper/2404.02258/) (2024)
- [Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference](https://mlsanity.com/paper/2304.04947/) (2023)
- [Efficiently Controlling Multiple Risks with Pareto Testing](https://mlsanity.com/paper/2210.07913/) (2022)
- [Learning to Skip for Language Modeling](https://mlsanity.com/paper/2311.15436/) (2023)
- [Inspecting the concept knowledge graph encoded by modern language models](https://mlsanity.com/paper/2105.13471/) (2021)
- [CEEBERT: Cross-Domain Inference in Early Exit BERT](https://mlsanity.com/paper/2405.15039/) (2024)
- [Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules](https://mlsanity.com/paper/2407.06677/) (2024)
- [Unsupervised Early Exit in DNNs with Multiple Exits](https://mlsanity.com/paper/2209.09480/) (2022)
- [Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference](https://mlsanity.com/paper/2405.14105/) (2024)
- [When in Doubt, Summon the Titans: Efficient Inference with Large Models](https://mlsanity.com/paper/2110.10305/) (2021)
- [MultiMed: Multilingual Medical Speech Recognition via Attention Encoder Decoder](https://mlsanity.com/paper/2409.14074/) (2024)
- [IOT: Instance-wise Layer Reordering for Transformer Structures](https://mlsanity.com/paper/2103.03457/) (2021)
- [DAdEE: Unsupervised Domain Adaptation in Early Exit PLMs](https://mlsanity.com/paper/2410.04424/) (2024)
- [Learning to Group Auxiliary Datasets for Molecule](https://mlsanity.com/paper/2307.04052/) (2023)
- [Beyond Greedy Exits: Improved Early Exit Decisions for Risk Control and Reliability](https://mlsanity.com/paper/2509.23666/) (2025)
- [LexGen: Domain-aware Multilingual Lexicon Generation](https://mlsanity.com/paper/2405.11200/) (2024)
- [Decoding at the Speed of Thought: Harnessing Parallel Decoding of Lexical Units for LLMs](https://mlsanity.com/paper/2405.15208/) (2024)
- [Dynamic Multi-Branch Layers for On-Device Neural Machine Translation](https://mlsanity.com/paper/2105.06679/) (2021)
- [FNetAR: Mixing Tokens with Autoregressive Fourier Transforms](https://mlsanity.com/paper/2107.10932/) (2021)
- [Learning When Not to Attend Globally](https://mlsanity.com/paper/2512.22562/) (2025)
- [Greedy-Gnorm: A Gradient Matrix Norm-Based Alternative to Attention Entropy for Head Pruning](https://mlsanity.com/paper/2602.04491/) (2026)
- [ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shaping](https://mlsanity.com/paper/2608.24411/) (2026)
- [SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM](https://mlsanity.com/paper/2406.06571/) (2024)

## Abstract (arXiv)

Most neural networks utilize the same amount of compute for every example independent of the inherent complexity of the input. Further, methods that adapt the amount of computation to the example focus on finding a fixed inference-time computational graph per example, ignoring any external computational budgets or varying inference time limitations. In this work, we utilize conditional computation to make neural sequence models (Transformer) more efficient and computation-aware during inference. We first modify the Transformer architecture, making each set of operations conditionally executable depending on the output of a learned control network. We then train this model in a multi-task setting, where each task corresponds to a particular computation budget. This allows us to train a single model that can be controlled to operate on different points of the computation-quality trade-off curve, depending on the available computation budget at inference time. We evaluate our approach on two tasks: (i) WMT English-French Translation and (ii) Unsupervised representation learning (BERT). Our experiments demonstrate that the proposed Conditional Computation Transformer (CCT) is competitive with vanilla Transformers when allowed to utilize its full computational budget, while improving significantly over computationally equivalent baselines when operating on smaller computational budgets.