# Improving Language Models Meaning Understanding and Consistency by Learning Conceptual Roles from Dictionary

arXiv: https://arxiv.org/abs/2310.15541 · HTML: https://mlsanity.com/paper/2310.15541/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 61 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Compacter: Efficient Low-Rank Hypercomplex Adapter Layers](https://mlsanity.com/paper/2106.04647/) (2021)
- [Dataless Knowledge Fusion by Merging Weights of Language Models](https://mlsanity.com/paper/2212.09849/) (2022)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)
- [Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks](https://mlsanity.com/paper/1811.01088/) (2018)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations](https://mlsanity.com/paper/2205.11822/) (2022)
- [Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2004.03829/) (2020)
- [Exploring and Predicting Transferability across NLP Tasks](https://mlsanity.com/paper/2005.00770/) (2020)
- [Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?](https://mlsanity.com/paper/2005.00628/) (2020)
- [KLUE: Korean Language Understanding Evaluation](https://mlsanity.com/paper/2105.09680/) (2021)
- [Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures](https://mlsanity.com/paper/2007.08970/) (2020)
- [Towards Improving Adversarial Training of NLP Models](https://mlsanity.com/paper/2109.00544/) (2021)
- [Logic-Guided Data Augmentation and Regularization for Consistent Question Answering](https://mlsanity.com/paper/2004.10157/) (2020)
- [Understanding by Understanding Not: Modeling Negation in Language Models](https://mlsanity.com/paper/2105.03519/) (2021)
- [UnNatural Language Inference](https://mlsanity.com/paper/2101.00010/) (2020)

## Built on by (most cited first, 4 of 4)

- [Knowledge Conflicts for LLMs: A Survey](https://mlsanity.com/paper/2403.08319/) (2024)
- [Logical Consistency of Large Language Models in Fact-checking](https://mlsanity.com/paper/2412.16100/) (2024)
- [Firm or Fickle? Evaluating Large Language Models Consistency in Sequential Interactions](https://mlsanity.com/paper/2503.22353/) (2025)
- [Does Instruction Tuning Make LLMs More Consistent?](https://mlsanity.com/paper/2404.15206/) (2024)

## Abstract (arXiv)

The non-humanlike behaviour of contemporary pre-trained language models (PLMs) is a leading cause undermining their trustworthiness. A striking phenomenon of such faulty behaviours is the generation of inconsistent predictions, which produces logically contradictory results, such as generating different predictions for texts delivering the same meaning or violating logical properties. Previous studies exploited data augmentation or implemented specialised loss functions to alleviate the issue. However, their usage is limited, because they consume expensive training resources for large-sized PLMs and can only handle a certain consistency type. To this end, we propose a practical approach that alleviates the inconsistent behaviour issue by fundamentally improving PLMs' meaning awareness. Based on the conceptual role theory, our method allows PLMs to capture accurate meaning by learning precise interrelationships between concepts from word-definition pairs in a dictionary. Next, we propose an efficient parameter integration technique that updates only a few additional parameters to combine the learned interrelationship with PLMs' pre-trained knowledge. Our experimental results reveal that the approach can concurrently improve multiple types of consistency, enables efficient knowledge integration, and easily applies to other languages.