# Scientific and Creative Analogies in Pretrained Language Models

arXiv: https://arxiv.org/abs/2211.15268 · HTML: https://mlsanity.com/paper/2211.15268/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 23 references that are in the map.
- First posted on arXiv 28 Nov 2022.

## Builds on (most cited first, 6 of 6)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?](https://mlsanity.com/paper/2105.04949/) (2021)
- [Multilingual Culture-Independent Word Analogy Datasets](https://mlsanity.com/paper/1911.10038/) (2019)
- [Learning Universal Representations from Word to Sentence](https://mlsanity.com/paper/2009.04656/) (2020)

## Built on by (most cited first, 7 of 7)

- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [ANALOGYKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base](https://mlsanity.com/paper/2305.05994/) (2023)
- [ANALOGICAL -- A Novel Benchmark for Long Text Analogy Evaluation in Large Language Models](https://mlsanity.com/paper/2305.05050/) (2023)
- [Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction](https://mlsanity.com/paper/2305.12660/) (2023)
- [Can language models learn analogical reasoning? Investigating training objectives and comparisons to human performance](https://mlsanity.com/paper/2310.05597/) (2023)
- [ARN: Analogical Reasoning on Narratives](https://mlsanity.com/paper/2310.00996/) (2023)
- [RelBERT: Embedding Relations with Language Models](https://mlsanity.com/paper/2310.00299/) (2023)

## Abstract (arXiv)

This paper examines the encoding of analogy in large-scale pretrained language models, such as BERT and GPT-2. Existing analogy datasets typically focus on a limited set of analogical relations, with a high similarity of the two domains between which the analogy holds. As a more realistic setup, we introduce the Scientific and Creative Analogy dataset (SCAN), a novel analogy dataset containing systematic mappings of multiple attributes and relational structures across dissimilar domains. Using this dataset, we test the analogical reasoning capabilities of several widely-used pretrained language models (LMs). We find that state-of-the-art LMs achieve low performance on these complex analogy tasks, highlighting the challenges still posed by analogy understanding.