# Can Language Models Be Specific? How?

arXiv: https://arxiv.org/abs/2210.05159 · HTML: https://mlsanity.com/paper/2210.05159/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://mlsanity.com/paper/2010.15980/) (2020)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models](https://mlsanity.com/paper/2010.06189/) (2020)
- [Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models](https://mlsanity.com/paper/2005.00683/) (2020)
- [Language Models are Open Knowledge Graphs](https://mlsanity.com/paper/2010.11967/) (2020)
- [Inducing Relational Knowledge from BERT](https://mlsanity.com/paper/1911.12753/) (2019)
- [Non-Monotonic Sequential Text Generation](https://mlsanity.com/paper/1902.02192/) (2019)
- [Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration](https://mlsanity.com/paper/2109.06304/) (2021)
- [Assessing Phrasal Representation and Composition in Transformers](https://mlsanity.com/paper/2010.03763/) (2020)
- [How to Query Language Models?](https://mlsanity.com/paper/2108.01928/) (2021)
- [Domain Agnostic Real-Valued Specificity Prediction](https://mlsanity.com/paper/1811.05085/) (2018)

## Built on by (most cited first, 8 of 8)

- [Why Does ChatGPT Fall Short in Providing Truthful Answers?](https://mlsanity.com/paper/2304.10513/) (2023)
- [Quantifying Association Capabilities of Large Language Models and Its Implications on Privacy Leakage](https://mlsanity.com/paper/2305.12707/) (2023)
- [Narrowing the Knowledge Evaluation Gap: Open-Domain Question Answering with Multi-Granularity Answers](https://mlsanity.com/paper/2401.04695/) (2024)
- [The Queen of England is not England's Queen: On the Lack of Factual Coherency in PLMs](https://mlsanity.com/paper/2402.01453/) (2024)
- [Abstraction Alignment: Comparing Model-Learned and Human-Encoded Conceptual Relationships](https://mlsanity.com/paper/2407.12543/) (2024)
- [InFact: Informativeness Alignment for Improved LLM Factuality](https://mlsanity.com/paper/2505.20487/) (2025)
- [Granuscore: A Reference-Free Measure of Granularity for Text Analysis and Question Answering](https://mlsanity.com/paper/2605.26620/) (2026)
- [Text Fact Transfer](https://mlsanity.com/paper/2310.14486/) (2023)

## Abstract (arXiv)

"He is a person", "Paris is located on the earth". Both statements are correct but meaningless - due to lack of specificity. In this paper, we propose to measure how specific the language of pre-trained language models (PLMs) is. To achieve this, we introduce a novel approach to build a benchmark for specificity testing by forming masked token prediction tasks with prompts. For instance, given "Toronto is located in [MASK].", we want to test whether a more specific answer will be better filled in by PLMs, e.g., Ontario instead of Canada. From our evaluations, we show that existing PLMs have only a slight preference for more specific answers. We identify underlying factors affecting the specificity and design two prompt-based methods to improve the specificity. Results show that the specificity of the models can be improved by the proposed methods without additional training. We hope this work can bring to awareness the notion of specificity of language models and encourage the research community to further explore this important but understudied problem.