mlsanityData to 7 Oct 2026

An Empirical Study on Few-shot Knowledge Probing for Pretrained Language Models

arXiv 2109.02772 ↗First posted 6 Sep 2021cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Prompt-based knowledge probing for 1-hop relations has been used to measure how much world knowledge is stored in pretrained language models. Existing work uses considerable amounts of data to tune the prompts for better performance. In this work, we compare a variety of approaches under a few-shot knowledge probing setting, where only a small number (e.g., 10 or 20) of example triples are available. In addition, we create a new dataset named TREx-2p, which contains 2-hop relations. We report that few-shot examples can strongly boost the probing performance for both 1-hop and 2-hop relations. In particular, we find that a simple-yet-effective approach of finetuning the bias vectors in the model outperforms existing prompt-engineering methods. Our dataset and code are available at urlhttps://github.com/cloudygoose/fewshotlama.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 6 Sep 2021.

Builds on

26 of 32 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
14 more
  1. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  2. 2019How Can We Know What Language Models Know?cited by 953
  3. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  4. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  5. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  6. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  7. 2017DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoningcited by 265
  8. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  9. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  10. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  11. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  12. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  13. 2019E-BERT: Efficient-Yet-Effective Entity Embeddings for BERTcited by 96
  14. 2021BERTese: Learning to Speak to BERTcited by 82

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 3; 2024: 0; 2025: 1.

8 papers.

  1. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 249
  2. 2022Prompting as Probing: Using Language Models for Knowledge Base Constructioncited by 32
  3. 2022Controlling the Focus of Pretrained Language Generation Modelscited by 4
  4. 2022ELECTRA is a Zero-Shot Learner, Toocited by 4
  5. 2023LLM2KB: Constructing Knowledge Bases using instruction tuned context aware Large Language Modelscited by 4
  6. 2023KGQuiz: Evaluating the Generalization of Encoded Knowledge in Large Language Modelscited by 4
  7. 2022PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completioncited by 3
  8. 2025Noise or Nuance: An Investigation Into Useful Information and Filtering For LLM Driven AKBCcited by 0