mlsanityData to 7 Oct 2026

Traffic Light or Light Traffic? Investigating Phrasal Semantics in Large Language Models

arXiv 2410.02308 ↗First posted 3 Oct 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Phrases are fundamental linguistic units through which humans convey semantics. This study critically examines the capacity of API-based large language models (LLMs) to comprehend phrase semantics, utilizing three human-annotated datasets. We assess the performance of LLMs in executing phrase semantic reasoning tasks guided by natural language instructions and explore the impact of common prompting techniques, including few-shot demonstrations and Chain-of-Thought reasoning. Our findings reveal that LLMs greatly outperform traditional embedding methods across the datasets; however, they do not show a significant advantage over fine-tuned methods. The effectiveness of advanced prompting strategies shows variability. We conduct detailed error analyses to interpret the limitations faced by LLMs in comprehending phrase semantics. Code and data can be found at https://github.com/memray/llm_phrase_semantics.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 13 of its 20 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

13 of 20 references.

References · 13

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  8. 2022A Survey on In-context Learningcited by 717
  9. 2023Deductive Verification of Chain-of-Thought Reasoningcited by 122
  10. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
  11. 2021Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Explorationcited by 44
  12. 2023CoF-CoT: Enhancing Large Language Models with Coarse-to-Fine Chain-of-Thought Prompting for Multi-domain NLU Taskscited by 18
1 more
  1. 2013Domain and Function: A Dual-Space Model of Semantic Relations and Compositionscited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models
  2. 2024Rethinking Semantic Parsing for Large Language Models: Enhancing LLM Performance with Semantic Hints
  3. 2023Large Linguistic Models: Investigating LLMs' metalinguistic abilities
  4. 2025Memorization or Reasoning? Exploring the Idiom Understanding of LLMs
  5. 2023Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds
  6. 2023Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners
  7. 2025Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoning
  8. 2025Method-Based Reasoning for Large Language Models: Extraction, Reuse, and Continuous Improvement