mlsanityData to 7 Oct 2026

BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?

arXiv 2105.04949 ↗First posted 11 May 2021cs.CL cs.LGcited by 56

Not on the map. Why?

Abstract (arXiv)

Analogies play a central role in human commonsense reasoning. The ability to recognize analogies such as "eye is to seeing what ear is to hearing", sometimes referred to as analogical proportions, shape how we structure knowledge and understand language. Surprisingly, however, the task of identifying such analogies has not yet received much attention in the language model era. In this paper, we analyze the capabilities of transformer-based language models on this unsupervised task, using benchmarks obtained from educational settings, as well as more commonly used datasets. We find that off-the-shelf language models can identify analogies to a certain extent, but struggle with abstract and complex relations, and results are highly sensitive to model architecture and hyperparameters. Overall the best results were obtained with GPT-2 and RoBERTa, while configurations using BERT were not able to outperform word embedding models. Our results raise important questions for future work about how, and to what extent, pre-trained language models capture knowledge about abstract semantic relations.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 39 of its 75 references that are in the map. First posted on arXiv 11 May 2021.

Builds on

39 of 75 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  7. 2019Language Models as Knowledge Bases?cited by 1,922
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  9. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  10. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  12. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
27 more
  1. 2019How Can We Know What Language Models Know?cited by 953
  2. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  3. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  4. 2015Listen, Attend and Spellcited by 542
  5. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  6. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  7. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  8. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  9. 2019Assessing BERT's Syntactic Abilitiescited by 191
  10. 2014Recurrent Neural Network Regularizationcited by 189
  11. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  12. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  13. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 149
  14. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  15. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  16. 2019Inducing Relational Knowledge from BERTcited by 75
  17. 2019Analogies Explained: Towards Understanding Word Embeddingscited by 63
  18. 2018Towards Understanding Linear Word Analogiescited by 57
  19. 2017Accelerating Innovation Through Analogy Miningcited by 32
  20. 2015Take and Took, Gaggle and Goose, Book and Read: Evaluating the Utility of Vector Differences for Lexical Relation Learningcited by 29
  21. 2019Fair is Better than Sensational:Man is to Doctor as Woman is to Doctorcited by 27
  22. 2016Issues in evaluating semantic spaces using word analogiescited by 26
  23. 2019Quantity doesn't buy quality syntax with neural language modelscited by 25
  24. 2020Analogies minus analogy test: measuring regularities in word embeddingscited by 17
  25. 2014Analogical Dissimilarity: Definition, Algorithms and Two Experiments in Machine Learningcited by 15
  26. 2020Towards Analogy-Based Explanations in Machine Learningcited by 7
  27. 2017Compositional Approaches for Representing Relations Between Words: A Comparative Studycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 12; 2023: 21; 2024: 14; 2025: 4; 2026: 2.

56 papers.

Sort:

By area:

  1. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  2. 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
  3. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  4. 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Modelscited by 48
  5. 2023PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learningcited by 31
  6. 2022E-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoningcited by 28
  7. 2022Analogy Generation by Prompting Large Language Models: A Case Study of InstructGPTcited by 27
  8. 2023StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understandingcited by 26
  9. 2022Life is a Circus and We are the Clowns: Automatically Finding Analogies between Situations and Processescited by 22
  10. 2022Multimodal Analogical Reasoning over Knowledge Graphscited by 21
  11. 2021A comparative evaluation and analysis of three generations of Distributional Semantic Modelscited by 19
  12. 2021Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids' Representationscited by 16
38 more
  1. 2021Distilling Relation Embeddings from Pre-trained Language Modelscited by 15
  2. 2023Do large language models solve verbal analogies like children do?cited by 13
  3. 2024AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogiescited by 13
  4. 2023ANALOGICAL -- A Novel Benchmark for Long Text Analogy Evaluation in Large Language Modelscited by 12
  5. 2023Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abductioncited by 11
  6. 2023Can language models learn analogical reasoning? Investigating training objectives and comparisons to human performancecited by 9
  7. 2022Distilling Hypernymy Relations from Language Models: On the Effectiveness of Zero-Shot Taxonomy Inductioncited by 8
  8. 2023ARN: Analogical Reasoning on Narrativescited by 8
  9. 2024ParallelPARC: A Scalable Pipeline for Generating Natural-Language Analogiescited by 8
  10. 2022Understanding Narratives through Dimensions of Analogycited by 7
  11. 2022Scientific and Creative Analogies in Pretrained Language Modelscited by 7
  12. 2024Creativity in AI: Progresses and Challengescited by 7
  13. 2022Quasi-symbolic Semantic Geometry over Transformer-based Variational AutoEncodercited by 5
  14. 2022Can Pretrained Language Models (Yet) Reason Deductively?cited by 5
  15. 2024Automatic Extraction of Metaphoric Analogies from Literary Texts: Task Formulation, Dataset Construction, and Evaluationcited by 5
  16. 2025SenWiCh: Sense-Annotation of Low-Resource Languages for WiC using Hybrid Methodscited by 4
  17. 2022Generalizable Neuro-symbolic Systems for Commonsense Question Answeringcited by 3
  18. 2023FAME: Flexible, Scalable Analogy Mappings Enginecited by 3
  19. 2024Boosting Scientific Concepts Understanding: Can Analogy from Teacher Models Empower Student Models?cited by 3
  20. 2023Enabling High-Level Machine Reasoning with Cognitive Neuro-Symbolic Systemscited by 2
  21. 2024Any four real numbers are on all fours with analogycited by 2
  22. 2024The creative psychometric item generator: a framework for item generation and validation using large language modelscited by 2
  23. 2024Can Multimodal Large Language Model Think Analogically?cited by 2
  24. 2022Does Wikidata Support Analogical Reasoning?cited by 1
  25. 2023A RelEntLess Benchmark for Modelling Graded Relations between Named Entitiescited by 1
  26. 2023Bidirectional Attention as a Mixture of Continuous Word Expertscited by 1
  27. 2023Why Do We Need Neuro-symbolic AI to Model Pragmatic Analogies?cited by 1
  28. 2023Large language models converge toward human-like concept organizationcited by 1
  29. 2023RelBERT: Embedding Relations with Language Modelscited by 1
  30. 2023On the Relationship between Sentence Analogy Identification and Sentence Structure Encoding in Large Language Modelscited by 1
  31. 2024Exploring Prompt-Based Methods for Zero-Shot Hypernym Prediction with Large Language Modelscited by 1
  32. 2024Past Meets Present: Creating Historical Analogy with Large Language Modelscited by 1
  33. 2024KnowledgePrompts: Exploring the Abilities of Large Language Models to Solve Proportional Analogies via Knowledge-Enhanced Promptingcited by 1
  34. 2025Multilingual LLMs Are Not Multilingual Thinkers: Evidence from Hindi Analogy Evaluationcited by 1
  35. 2025Analogical Structure, Minimal Contextual Cues and Contrastive Distractors: Input Design for Sample-Efficient Linguistic Rule Inductioncited by 1
  36. 2023An Investigation on Word Embedding Offset Clustering as Relationship Classificationcited by 0
  37. 2023Solving Hard Analogy Questions with Relation Embedding Chainscited by 0
  38. 2023Text Fact Transfercited by 0

50 most cited of 56 All 56 (JSON)