mlsanityData to 14 Oct 2026

BERTese: Learning to Speak to BERT

arXiv 2103.05327 ↗First posted 9 Mar 2021cs.CL cs.LGcited by 82

Not on the map. Why?

Abstract (arXiv)

Large pre-trained language models have been shown to encode large amounts of world and commonsense knowledge in their parameters, leading to substantial interest in methods for extracting that knowledge. In past work, knowledge was extracted by taking manually-authored queries and gathering paraphrases for them using a separate pipeline. In this work, we propose a method for automatically rewriting queries into "BERTese", a paraphrase query that is directly optimized towards better knowledge extraction. To encourage meaningful rewrites, we add auxiliary loss functions that encourage the query to correspond to actual language tokens. We empirically show our approach outperforms competing baselines, obviating the need for complex pipelines. Moreover, BERTese provides some insight into the type of language that helps language models perform knowledge extraction.

Key facts. Cited by 82 papers in the map (data through 14 Oct 2026). Builds on 16 of its 21 references that are in the map. First posted on arXiv 9 Mar 2021.

Builds on

16 of 21 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  8. 2019Language Models as Knowledge Bases?cited by 1,936
  9. 2019How Can We Know What Language Models Know?cited by 959
  10. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  11. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 249
  12. 2020How Context Affects Language Models' Factual Predictionscited by 181
4 more
  1. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  2. 2019Inducing Relational Knowledge from BERTcited by 75
  3. 2017Ask the Right Questions: Active Question Reformulation with Reinforcement Learningcited by 44
  4. 2017Task-Oriented Query Reformulation with Reinforcement Learningcited by 39

Built on by

Cited by year

Citing papers per year: 2021: 12; 2022: 16; 2023: 35; 2024: 16; 2025: 3.

82 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  2. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  3. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  4. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  5. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  6. 2022Optimizing Prompts for Text-to-Image Generationcited by 207
  7. 2023Data-centric Artificial Intelligence: A Surveycited by 203
  8. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  9. 2023Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Datacited by 119
  10. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  11. 2022Black-box Prompt Learning for Pre-trained Language Modelscited by 70
  12. 2023Graph Prompt Learning: A Comprehensive Survey and Beyondcited by 61
38 more
  1. 2023A Practical Survey on Zero-shot Prompt Design for In-context Learningcited by 49
  2. 2024PRewrite: Prompt Rewriting with Reinforcement Learningcited by 45
  3. 2023Prompting Large Language Models With the Socratic Methodcited by 43
  4. 2022No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligencecited by 41
  5. 2023Learning to Rewrite Prompts for Personalized Text Generationcited by 40
  6. 2023ArGue: Attribute-Guided Prompt Tuning for Vision-Language Modelscited by 39
  7. 2021Good Examples Make A Faster Learner: Simple Demonstration-based Learning for Low-resource NERcited by 38
  8. 2023Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Promptingcited by 36
  9. 2023Language Models as Black-Box Optimizers for Vision-Language Modelscited by 36
  10. 2022Prompting as Probing: Using Language Models for Knowledge Base Constructioncited by 32
  11. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  12. 2023Revisiting Few-Shot Object Detection with Vision-Language Modelscited by 32
  13. 2021Relational World Knowledge Representation in Contextual Language Models: A Reviewcited by 31
  14. 2022Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal Viewcited by 31
  15. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  16. 2023Towards Ontology Construction with Language Modelscited by 23
  17. 2021P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Promptscited by 22
  18. 2023Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Viewscited by 21
  19. 2022MetaPrompting: Learning to Learn Better Promptscited by 19
  20. 2023BatchPrompt: Accomplish more with lesscited by 16
  21. 2021Distilling Relation Embeddings from Pre-trained Language Modelscited by 15
  22. 2023PTP: Boosting Stability and Performance of Prompt Tuning with Perturbation-Based Regularizercited by 15
  23. 2024SocraSynth: Multi-LLM Reasoning with Conditional Statisticscited by 13
  24. 2024All-in-one Weather-degraded Image Restoration via Adaptive Degradation-aware Self-prompting Modelcited by 13
  25. 2024Prompt Refinement with Image Pivot for Text-to-Image Generationcited by 11
  26. 2023On the Evolution of Knowledge Graphs: A Survey and Perspectivecited by 10
  27. 2022Learning a Better Initialization for Soft Prompts via Meta-Learningcited by 9
  28. 2023A Survey on Prompting Techniques in LLMscited by 9
  29. 2023LLM4Causal: Democratized Causal Tools for Everyone via Large Language Modelcited by 9
  30. 2024How to Make LLMs Forget: On Reversing In-Context Knowledge Editscited by 9
  31. 2021An Empirical Study on Few-shot Knowledge Probing for Pretrained Language Modelscited by 8
  32. 2023Can discrete information extraction prompts generalize across language models?cited by 8
  33. 2023Plum: Prompt Learning using Metaheuristiccited by 8
  34. 2021Unified Multimodal Pre-training and Prompt-based Tuning for Vision-Language Understanding and Generationcited by 7
  35. 2022Pre-trained Token-replaced Detection Model as Few-shot Learnercited by 7
  36. 2022Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understandingcited by 7
  37. 2022Probing via Promptingcited by 7
  38. 2023Dialogue for Prompting: a Policy-Gradient-Based Discrete Prompt Generation for Few-shot Learningcited by 7

50 most cited of 82 All 82 (JSON)