# DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering

arXiv: https://arxiv.org/abs/2005.00697 · HTML: https://mlsanity.com/paper/2005.00697/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2020.

## Built on by (most cited first, 29 of 29)

- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference](https://mlsanity.com/paper/2105.11618/) (2021)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [Transkimmer: Transformer Learns to Layer-wise Skim](https://mlsanity.com/paper/2205.07324/) (2022)
- [Retrieval-Free Knowledge-Grounded Dialogue Response Generation with Adapters](https://mlsanity.com/paper/2105.06232/) (2021)
- [Distilled Dual-Encoder Model for Vision-Language Understanding](https://mlsanity.com/paper/2112.08723/) (2021)
- [Block-Skim: Efficient Question Answering for Transformer](https://mlsanity.com/paper/2112.08560/) (2021)
- [Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognition](https://mlsanity.com/paper/2307.10757/) (2023)
- [Exploring Extreme Parameter Compression for Pre-trained Language Models](https://mlsanity.com/paper/2205.10036/) (2022)
- [Differentially Private Model Compression](https://mlsanity.com/paper/2206.01838/) (2022)
- [A Survey on Deep Learning Event Extraction: Approaches and Applications](https://mlsanity.com/paper/2107.02126/) (2021)
- [Adapting by Pruning: A Case Study on BERT](https://mlsanity.com/paper/2105.03343/) (2021)
- [Modeling Context in Answer Sentence Selection Systems on a Latency Budget](https://mlsanity.com/paper/2101.12093/) (2021)
- [RAAT: Relation-Augmented Attention Transformer for Relation Modeling in Document-Level Event Extraction](https://mlsanity.com/paper/2206.03377/) (2022)
- [Optimizing Inference Performance of Transformers on CPUs](https://mlsanity.com/paper/2102.06621/) (2021)
- [SDR: Efficient Neural Re-ranking using Succinct Document Representation](https://mlsanity.com/paper/2110.02065/) (2021)
- [BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models](https://mlsanity.com/paper/2310.01329/) (2023)
- [Investigating the Role of Feed-Forward Networks in Transformers Using Parallel Attention and Feed-Forward Net Design](https://mlsanity.com/paper/2305.13297/) (2023)
- [Improving Text Semantic Similarity Modeling through a 3D Siamese Network](https://mlsanity.com/paper/2307.09274/) (2023)
- [EAVE: Efficient Product Attribute Value Extraction via Lightweight Sparse-layer Interaction](https://mlsanity.com/paper/2406.06839/) (2024)
- [VIRT: Improving Representation-based Models for Text Matching through Virtual Interaction](https://mlsanity.com/paper/2112.04195/) (2021)
- [Question Generation for Evaluating Cross-Dataset Shifts in Multi-modal Grounding](https://mlsanity.com/paper/2201.09639/) (2022)
- [AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems](https://mlsanity.com/paper/2301.09262/) (2023)
- [Modularity in Deep Learning: A Survey](https://mlsanity.com/paper/2310.01154/) (2023)
- [Once is Enough: A Light-Weight Cross-Attention for Fast Sentence Pair Modeling](https://mlsanity.com/paper/2210.05261/) (2022)
- [A Monte Carlo Language Model Pipeline for Zero-Shot Sociopolitical Event Extraction](https://mlsanity.com/paper/2305.15051/) (2023)
- [Comparing Neighbors Together Makes it Easy: Jointly Comparing Multiple Candidates for Efficient and Effective Retrieval](https://mlsanity.com/paper/2405.12801/) (2024)
- [Reasoning Paths as Signals: Augmenting Multi-hop Fact Verification through Structural Reasoning Progression](https://mlsanity.com/paper/2506.07075/) (2025)
- [Token-Event-Role Structure-based Multi-Channel Document-Level Event Extraction](https://mlsanity.com/paper/2306.17733/) (2023)

## Abstract (arXiv)

Transformer-based QA models use input-wide self-attention -- i.e. across both the question and the input passage -- at all layers, causing them to be slow and memory-intensive. It turns out that we can get by without input-wide self-attention at all layers, especially in the lower layers. We introduce DeFormer, a decomposed transformer, which substitutes the full self-attention with question-wide and passage-wide self-attentions in the lower layers. This allows for question-independent processing of the input text representations, which in turn enables pre-computing passage representations reducing runtime compute drastically. Furthermore, because DeFormer is largely similar to the original model, we can initialize DeFormer with the pre-training weights of a standard transformer, and directly fine-tune on the target QA dataset. We show DeFormer versions of BERT and XLNet can be used to speed up QA by over 4.3x and with simple distillation-based losses they incur only a 1% drop in accuracy. We open source the code at https://github.com/StonyBrookNLP/deformer.