mlsanityData to 7 Oct 2026

Generative Representational Instruction Tuning

arXiv 2402.09906 ↗First posted 15 Feb 2024cs.CL cs.AI cs.LGcited by 227

Not on the map. Why?

Cited by 38 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

All text-based language problems can be reduced to either generation or embedding. Current models only perform well at one or the other. We introduce generative representational instruction tuning (GRIT) whereby a large language model is trained to handle both generative and embedding tasks by distinguishing between them through instructions. Compared to other open models, our resulting GritLM 7B sets a new state of the art on the Massive Text Embedding Benchmark (MTEB) and outperforms all models up to its size on a range of generative tasks. By scaling up further, GritLM 8x7B outperforms all open generative language models that we tried while still being among the best embedding models. Notably, we find that GRIT matches training on only generative or embedding data, thus we can unify both at no performance loss. Among other benefits, the unification via GRIT speeds up Retrieval-Augmented Generation (RAG) by > 60% for long documents, by no longer requiring separate retrieval and generation models. Models, code, etc. are freely available at https://github.com/ContextualAI/gritlm.

Key facts. Cited by 227 papers in the map (data through 7 Oct 2026). Builds on 163 of its 189 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

163 of 189 references.

References · 163

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2023GPT-4 Technical Reportcited by 18,132
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2017Proximal Policy Optimization Algorithmscited by 14,441
  10. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  11. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  12. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
38 more
  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2016Layer Normalizationcited by 5,579
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  11. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  13. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  14. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  15. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  16. 2020Longformer: The Long-Document Transformercited by 3,362
  17. 2023Mistral 7Bcited by 3,065
  18. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  19. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  20. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  21. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  22. 2017Billion-scale similarity search with GPUscited by 2,477
  23. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  24. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  25. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  26. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  27. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  28. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  29. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  30. 2024Mixtral of Expertscited by 1,785
  31. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  32. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  33. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  34. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  35. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  36. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  37. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  38. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308

50 most cited of 163 All 163 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 66; 2025: 97; 2026: 64.

227 papers.

Sort:

By area:

  1. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  2. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  3. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  5. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  6. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 223
  7. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  8. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  9. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  10. 2025On the Theoretical Limitations of Embedding-Based Retrievalcited by 98
  11. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 81
  12. 2024Making Text Embedders Few-Shot Learnerscited by 78
38 more
  1. 2025ReasonIR: Training Retrievers for Reasoning Taskscited by 76
  2. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  3. 2024Repetition Improves Language Model Embeddingscited by 70
  4. 2025Rank1: Test-Time Compute for Reranking in Information Retrievalcited by 68
  5. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 67
  6. 2024Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Modelscited by 60
  7. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  8. 2024CoIR: A Comprehensive Benchmark for Code Information Retrieval Modelscited by 59
  9. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 47
  10. 2024LitSearch: A Retrieval Benchmark for Scientific Literature Searchcited by 47
  11. 2025Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Modelscited by 46
  12. 2025ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubricscited by 44
  13. 2024Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Modelscited by 39
  14. 2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
  15. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
  16. 2025KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Modelcited by 35
  17. 2024RAR-b: Reasoning as Retrieval Benchmarkcited by 33
  18. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  19. 2025Training Sparse Mixture Of Experts Text Embedding Modelscited by 32
  20. 2025RAG-RL: Advancing Retrieval-Augmented Generation via RL and Curriculum Learningcited by 30
  21. 2025Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Miningcited by 27
  22. 2025RaDeR: Reasoning-aware Dense Retrieval Modelscited by 27
  23. 2024The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embeddingcited by 26
  24. 2024Context Embeddings for Efficient Answer Generation in RAGcited by 26
  25. 2024FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularitycited by 26
  26. 2025Large Language Models are Powerful Electronic Health Record Encoderscited by 25
  27. 2025Diffusion vs. Autoregressive Language Models: A Text Embedding Perspectivecited by 23
  28. 2025PropRAG: Guiding Retrieval with Beam Search over Proposition Pathscited by 22
  29. 2024RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generationcited by 20
  30. 2024ATM: Adversarial Tuning Multi-agent System Makes a Robust Retrieval-Augmented Generatorcited by 19
  31. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmarkcited by 19
  32. 2024Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessmentcited by 19
  33. 2025DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrieverscited by 19
  34. 2025Improving Code Localization with Repository Memorycited by 19
  35. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 19
  36. 2024ChatRetriever: Adapting Large Language Models for Generalized and Robust Conversational Dense Retrievalcited by 18
  37. 2024Linq-Embed-Mistral Technical Reportcited by 18
  38. 2024Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMscited by 18

50 most cited of 227 All 227 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model
  2. 2025HIRAG: Hierarchical-Thought Instruction-Tuning Retrieval-Augmented Generation
  3. 2025Exploring Training and Inference Scaling Laws in Generative Retrieval
  4. 2024CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Tasks
  5. 2024XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation
  6. 2024FunnelRAG: A Coarse-to-Fine Progressive Retrieval Paradigm for RAG
  7. 2024GASE: Generatively Augmented Sentence Encoding
  8. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models