mlsanityData to 7 Oct 2026

FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference

arXiv 2212.08153 ↗First posted 15 Dec 2022cs.CL cs.AI cs.LGcited by 31

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Fusion-in-Decoder (FiD) is a powerful retrieval-augmented language model that sets the state-of-the-art on many knowledge-intensive NLP tasks. However, the architecture used for FiD was chosen by making minimal modifications to a standard T5 model, which our analysis shows to be highly suboptimal for a retrieval-augmented model. In particular, FiD allocates the bulk of FLOPs to the encoder, while the majority of inference time results from memory bandwidth constraints in the decoder. We propose two simple changes to the FiD architecture to alleviate memory bandwidth constraints, and speed up inference by 7x. This allows us to use a much larger decoder at modest cost. We denote FiD with the above modifications as FiDO, and show that it strongly improves performance over existing FiD models for a wide range of inference budgets. For example, FiDO-Large-XXL performs faster inference than FiD-Base and achieves better performance than FiD-Large.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 33 of its 49 references that are in the map. First posted on arXiv 15 Dec 2022.

Builds on

33 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2020Knowledge Distillation: A Surveycited by 1,578
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
18 more
  1. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  2. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  3. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  4. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  5. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  6. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  7. 2022Efficiently Scaling Transformer Inferencecited by 529
  8. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  9. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  10. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  11. 2022Memorizing Transformerscited by 219
  12. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  13. 2021KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answeringcited by 66
  14. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  15. 2021Mention Memory: incorporating textual knowledge into Transformers through entity mention attentioncited by 42
  16. 2022Decoupled Context Processing for Context Augmented Language Modelingcited by 24
  17. 2021ReadTwice: Reading Very Large Documents with Memoriescited by 14
  18. 2022Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?cited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 12; 2025: 3; 2026: 3.

31 papers.

Sort:

By area:

  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  2. 2024Yi: Open Foundation Models by 01.AIcited by 706
  3. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 235
  4. 2023Speculative Decoding with Big Little Decodercited by 163
  5. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  6. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  7. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 71
  8. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
  9. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategycited by 50
  10. 2023RepoFusion: Training Code Models to Understand Your Repositorycited by 40
  11. 2023Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your computecited by 17
  12. 2023Bridging the KB-Text Gap: Leveraging Structured Knowledge-aware Pre-training for KBQAcited by 17
19 more
  1. 2023CarExpert: Leveraging Large Language Models for In-Car Conversational Question Answeringcited by 10
  2. 2023Optimizing Retrieval-augmented Reader Models via Token Eliminationcited by 10
  3. 2023GLIMMER: generalized late-interaction memory rerankercited by 7
  4. 2025GRAM: Generative Recommendation via Semantic-aware Multi-granular Late Fusioncited by 7
  5. 2023BTR: Binary Token Representations for Efficient Retrieval Augmented Language Modelscited by 5
  6. 2023Beyond Traditional Teaching: The Potential of Large Language Models and Chatbots in Graduate Engineering Educationcited by 4
  7. 2024Context-Aware Assistant Selection for Improved Inference Acceleration with Large Language Modelscited by 4
  8. 2024Arctic-TILT. Business Document Understanding at Sub-Billion Scalecited by 3
  9. 2024Efficient Encoder-Decoder Transformer Decoding for Decomposable Taskscited by 2
  10. 2024Multi-Granularity Guided Fusion-in-Decodercited by 2
  11. 2024MATTER: Memory-Augmented Transformer Using Heterogeneous Knowledge Sourcescited by 2
  12. 2024FastFiD: Improve Inference Efficiency of Open Domain Question Answering via Sentence Selectioncited by 2
  13. 2025Gumbel Reranking: Differentiable End-to-End Reranker Optimizationcited by 2
  14. 2024It's About Time: Incorporating Temporality in Retrieval Augmented Language Modelscited by 1
  15. 2026SAGE: SLO-Aware Adaptive Retrieval for Production RAG Systemscited by 1
  16. 2023Detrimental Contexts in Open-Domain Question Answeringcited by 0
  17. 2025Struc-EMB: The Potential of Structure-Aware Encoding in Language Embeddingscited by 0
  18. 2026AstroRAG -- A Pagerank-Based Retrieval-Augmented Generation Pipeline for Question Answering in Astronomycited by 0
  19. 2026Privacy-Preserving RAG via Multi-Agent Semantic Rewriting: Achieving Confidentiality Without Compromising Contextual Fidelitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MLP Memory: A Retriever-Pretrained Memory for Large Language Models
  2. 2026ReFilter: Improving Robustness of Retrieval-Augmented Generation via Gated Filter
  3. 2026Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory
  4. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding
  5. 2024PLD+: Accelerating LLM inference by leveraging Language Model Artifacts
  6. 2025REFRAG: Rethinking RAG based Decoding
  7. 2025Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models
  8. 2026Test-Time Compute for Frozen Embedding Models through Agentic Program Search