mlsanityData to 7 Oct 2026

LaMDA: Language Models for Dialog Applications

arXiv 2201.08239 ↗First posted 20 Jan 2022cs.CL cs.AIcited by 1,218

Not on the map. Why?

Cited by 148 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present LaMDA: Language Models for Dialog Applications. LaMDA is a family of Transformer-based neural language models specialized for dialog, which have up to 137B parameters and are pre-trained on 1.56T words of public dialog data and web text. While model scaling alone can improve quality, it shows less improvements on safety and factual grounding. We demonstrate that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding. The first challenge, safety, involves ensuring that the model's responses are consistent with a set of human values, such as preventing harmful suggestions and unfair bias. We quantify safety using a metric based on an illustrative set of human values, and we find that filtering candidate responses using a LaMDA classifier fine-tuned with a small amount of crowdworker-annotated data offers a promising approach to improving model safety. The second challenge, factual grounding, involves enabling the model to consult external knowledge sources, such as an information retrieval system, a language translator, and a calculator. We quantify factuality using a groundedness metric, and we find that our approach enables the model to generate responses grounded in known sources, rather than responses that merely sound plausible. Finally, we explore the use of LaMDA in the domains of education and content recommendations, and analyze their helpfulness and role consistency.

Key facts. Cited by 1,218 papers in the map (data through 7 Oct 2026). Builds on 87 of its 120 references that are in the map. First posted on arXiv 20 Jan 2022.

Builds on

87 of 120 references.

References · 87

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2018Deep contextualized word representationscited by 2,285
38 more
  1. 2020Extracting Training Data from Large Language Modelscited by 2,144
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  3. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2020GLU Variants Improve Transformercited by 1,724
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  8. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  9. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  10. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  12. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  13. 2021Ethical and social risks of harm from Language Modelscited by 957
  14. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  15. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  16. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  17. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  18. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  19. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  20. 2020Recipes for building an open-domain chatbotcited by 593
  21. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  22. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  23. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  24. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  25. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  26. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  27. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  28. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  29. 2015Skip-Thought Vectorscited by 321
  30. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  31. 2019Measuring Bias in Contextualized Word Representationscited by 275
  32. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  33. 2021Internet-Augmented Dialogue Generationcited by 217
  34. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  35. 2019Gender Bias in Contextualized Word Embeddingscited by 203
  36. 2015Semi-supervised Sequence Learningcited by 198
  37. 2015A Neural Conversational Modelcited by 195
  38. 2019TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agentscited by 188

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 188; 2023: 545; 2024: 296; 2025: 141; 2026: 48.

1,218 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
38 more
  1. 2023Qwen Technical Reportcited by 3,273
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  9. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  11. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  12. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  13. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  14. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  15. 2023PaLM 2 Technical Reportcited by 1,154
  16. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  17. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  18. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  19. 2023BloombergGPT: A Large Language Model for Financecited by 923
  20. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  21. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  22. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  23. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  24. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  25. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  26. 2022A Generalist Agentcited by 802
  27. 2023StarCoder: may the source be with you!cited by 790
  28. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  29. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  30. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  31. 2022Galactica: A Large Language Model for Sciencecited by 736
  32. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  33. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  34. 2022A Survey on In-context Learningcited by 717
  35. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  36. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  37. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  38. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663

50 most cited of 1,218 All 1,218 (JSON)