mlsanityData to 14 Oct 2026

Nonparametric Masked Language Modeling

arXiv 2212.01349 ↗First posted 2 Dec 2022cs.CL cs.AI cs.LGcited by 40

Not on the map. Why?

Abstract (arXiv)

Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases. We introduce NPM, the first nonparametric masked language model that replaces this softmax with a nonparametric distribution over every phrase in a reference corpus. NPM fills in the [MASK] solely from retrieving a token from a text corpus. We show that NPM can be efficiently trained with a contrastive objective and an in-batch approximation to full corpus retrieval. Zero-shot evaluation on 16 tasks including classification, fact probing and question answering demonstrates that NPM outperforms significantly larger parametric models, either with or without a retrieve-and-generate approach. It is particularly better at dealing with rare patterns (word senses or facts) and predicting rare or nearly unseen words (e.g., non-Latin script). We release the model and code at github.com/facebookresearch/NPM.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 63 of its 95 references that are in the map. First posted on arXiv 2 Dec 2022.

Builds on

63 of 95 references.

References · 63

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  12. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
38 more
  1. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2017Billion-scale similarity search with GPUscited by 2,497
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  8. 2019Language Models as Knowledge Bases?cited by 1,936
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  13. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  14. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  15. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  16. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  17. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  18. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  19. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  20. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  21. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 296
  22. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  23. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
  24. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 152
  25. 2017Cross-lingual Entity Alignment via Joint Attribute-Preserving Embeddingcited by 127
  26. 2020XOR QA: Cross-lingual Open-Retrieval Question Answeringcited by 121
  27. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 119
  28. 2022Training Language Models with Memory Augmentationcited by 113
  29. 2021Efficient Nearest Neighbor Language Modelscited by 93
  30. 2020Learning Dense Representations of Phrases at Scalecited by 91
  31. 2022PEER: A Collaborative Language Modelcited by 86
  32. 2019Scalable Zero-shot Entity Linking with Dense Entity Retrievalcited by 85
  33. 2021Adaptive Semiparametric Language Modelscited by 83
  34. 2016Efficient softmax approximation for GPUscited by 67
  35. 2019Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Indexcited by 64
  36. 2021Few-Shot Question Answering by Pretraining Span Selectioncited by 62
  37. 2022Neuro-Symbolic Language Modeling with Automaton-augmented Retrievalcited by 60
  38. 2022Describing Differences between Text Distributions with Natural Languagecited by 54

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 14; 2024: 20; 2025: 2; 2026: 3.

40 papers.

Sort:

By area:

  1. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  2. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  3. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 532
  4. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  5. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  6. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  7. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 195
  8. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 153
  9. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 79
  10. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 79
  11. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 75
  12. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
28 more
  1. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  2. 2024SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMscited by 67
  3. 2024Scaling Retrieval-Based Language Models with a Trillion-Token Datastorecited by 56
  4. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  5. 2023Privacy Implications of Retrieval-Based Language Modelscited by 32
  6. 2023Copy Is All You Needcited by 27
  7. 2024Privacy Preserving Prompt Engineering: A Surveycited by 25
  8. 2023Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Modelscited by 23
  9. 2024Multi-FAct: Assessing Factuality of Multilingual LLMs using FActScorecited by 18
  10. 2025Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Modelscited by 16
  11. 2023kNN-Adapter: Efficient Domain Adaptation for Black-Box Language Modelscited by 14
  12. 2024Verifiable by Design: Aligning Language Models to Quote from Pre-Training Datacited by 13
  13. 2024Multi-group Uncertainty Quantification for Long-form Text Generationcited by 13
  14. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  15. 2024Retrieval is Accurate Generationcited by 10
  16. 2024Preference-Guided Refactored Tuning for Retrieval Augmented Code Generationcited by 10
  17. 2024Efficient Prompt Caching via Embedding Similaritycited by 9
  18. 2023HistAlign: Improving Context Dependency in Language Generation by Aligning with Historycited by 6
  19. 2023BTR: Binary Token Representations for Efficient Retrieval Augmented Language Modelscited by 5
  20. 2026Cram Less to Fit More: Training Data Pruning Improves Memorization of Factscited by 3
  21. 2022Empowering Sentence Encoders with Prompting and Label Retrieval for Zero-shot Text Classificationcited by 2
  22. 2024Scaling the Vocabulary of Non-autoregressive Models for Efficient Generative Retrievalcited by 2
  23. 2025Deep Space Weather Model: Long-Range Solar Flare Prediction from Multi-Wavelength Imagescited by 2
  24. 2024R4: Reinforced Retriever-Reorder-Responder for Retrieval-Augmented Large Language Modelscited by 1
  25. 2024Semi-Parametric Retrieval via Binary Bag-of-Tokens Indexcited by 0
  26. 2024DNA-SE: Towards Deep Neural-Nets Assisted Semiparametric Estimationcited by 0
  27. 2026Compression Beyond the Uncompressed: A Two-Stage Training Recipe for Soft Context Compression in RAGcited by 0
  28. 2026Retrieval-Centric Deep Learning in Growing Nonparametric Neural Networkscited by 0