# ANALOGICAL -- A Novel Benchmark for Long Text Analogy Evaluation in Large Language Models

arXiv: https://arxiv.org/abs/2305.05050 · HTML: https://mlsanity.com/paper/2305.05050/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 60 references that are in the map.
- First posted on arXiv 8 May 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)
- [LinkBERT: Pretraining Language Models with Document Links](https://mlsanity.com/paper/2203.15827/) (2022)
- [Evaluating Commonsense in Pre-trained Language Models](https://mlsanity.com/paper/1911.11931/) (2019)
- [Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures](https://mlsanity.com/paper/2007.08970/) (2020)
- [The paradox of the compositionality of natural language: a neural machine translation case study](https://mlsanity.com/paper/2108.05885/) (2021)
- [BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?](https://mlsanity.com/paper/2105.04949/) (2021)
- [A Unified approach for Conventional Zero-shot, Generalized Zero-shot and Few-shot Learning](https://mlsanity.com/paper/1706.08653/) (2017)
- [Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoning](https://mlsanity.com/paper/2106.03921/) (2021)
- [Draw Me a Flower: Processing and Grounding Abstraction in Natural Language](https://mlsanity.com/paper/2106.14321/) (2021)
- [ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understanding](https://mlsanity.com/paper/2109.06838/) (2021)
- [Scientific and Creative Analogies in Pretrained Language Models](https://mlsanity.com/paper/2211.15268/) (2022)
- [WordRep: A Benchmark for Research on Learning Word Representations](https://mlsanity.com/paper/1407.1640/) (2014)

## Built on by (most cited first, 12 of 12)

- [LogiDynamics: Unraveling the Dynamics of Inductive, Abductive and Deductive Logical Inferences in LLM Reasoning](https://mlsanity.com/paper/2502.11176/) (2025)
- [ARN: Analogical Reasoning on Narratives](https://mlsanity.com/paper/2310.00996/) (2023)
- [Creativity in AI: Progresses and Challenges](https://mlsanity.com/paper/2410.17218/) (2024)
- [Evaluating the Logical Reasoning Abilities of Large Reasoning Models](https://mlsanity.com/paper/2505.11854/) (2025)
- [Measuring Combinatorial Creativity in Large Language Models and Humans](https://mlsanity.com/paper/2509.21043/) (2025)
- [Deciphering Textual Authenticity: A Generalized Strategy through the Lens of Large Language Semantics for Detecting Human vs. Machine-Generated Text](https://mlsanity.com/paper/2401.09407/) (2024)
- [The Curious Case of Analogies: Investigating Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2511.20344/) (2025)
- [Why Do We Need Neuro-symbolic AI to Model Pragmatic Analogies?](https://mlsanity.com/paper/2308.01936/) (2023)
- [Enhancing multimodal analogical reasoning with Logic Augmented Generation](https://mlsanity.com/paper/2504.11190/) (2025)
- [Meanings are like Onions: a Layered Approach to Metaphor Processing](https://mlsanity.com/paper/2507.10354/) (2025)
- [Analogical Proportions and Creativity: A Preliminary Study](https://mlsanity.com/paper/2310.13500/) (2023)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)

## Abstract (arXiv)

Over the past decade, analogies, in the form of word-level analogies, have played a significant role as an intrinsic measure of evaluating the quality of word embedding methods such as word2vec. Modern large language models (LLMs), however, are primarily evaluated on extrinsic measures based on benchmarks such as GLUE and SuperGLUE, and there are only a few investigations on whether LLMs can draw analogies between long texts. In this paper, we present ANALOGICAL, a new benchmark to intrinsically evaluate LLMs across a taxonomy of analogies of long text with six levels of complexity -- (i) word, (ii) word vs. sentence, (iii) syntactic, (iv) negation, (v) entailment, and (vi) metaphor. Using thirteen datasets and three different distance measures, we evaluate the abilities of eight LLMs in identifying analogical pairs in the semantic vector space. Our evaluation finds that it is increasingly challenging for LLMs to identify analogies when going up the analogy taxonomy.