# Evaluating Large Language Models on Multiword Expressions in Multilingual and Code-Switched Contexts

arXiv: https://arxiv.org/abs/2504.20051 · HTML: https://mlsanity.com/paper/2504.20051/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 21 references that are in the map.
- First posted on arXiv 10 Apr 2025.

## Builds on (most cited first, 10 of 10)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Testing the Ability of Language Models to Interpret Figurative Language](https://mlsanity.com/paper/2204.12632/) (2022)
- [A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologies](https://mlsanity.com/paper/2301.01967/) (2023)
- [SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding](https://mlsanity.com/paper/2204.10050/) (2022)
- [Investigating Robustness of Dialog Models to Popular Figurative Language Constructs](https://mlsanity.com/paper/2110.00687/) (2021)
- [Impact of Co-occurrence on Factual Knowledge of Large Language Models](https://mlsanity.com/paper/2310.08256/) (2023)
- [WinoDict: Probing language models for in-context word acquisition](https://mlsanity.com/paper/2209.12153/) (2022)
- [Sign of the Times: Evaluating the use of Large Language Models for Idiomaticity Detection](https://mlsanity.com/paper/2405.09279/) (2024)
- [A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models](https://mlsanity.com/paper/2405.10579/) (2024)
- [Semantics of Multiword Expressions in Transformer-Based Models: A Survey](https://mlsanity.com/paper/2401.15393/) (2024)

## Abstract (arXiv)

Multiword expressions, characterised by non-compositional meanings and syntactic irregularities, are an example of nuanced language. These expressions can be used literally or idiomatically, leading to significant changes in meaning. While large language models have demonstrated strong performance across many tasks, their ability to handle such linguistic subtleties remains uncertain. Therefore, this study evaluates how state-of-the-art language models process the ambiguity of potentially idiomatic multiword expressions, particularly in contexts that are less frequent, where models are less likely to rely on memorisation. By evaluating models across in Portuguese and Galician, in addition to English, and using a novel code-switched dataset and a novel task, we find that large language models, despite their strengths, struggle with nuanced language. In particular, we find that the latest models, including GPT-4, fail to outperform the xlm-roBERTa-base baselines in both detection and semantic tasks, with especially poor performance on the novel tasks we introduce, despite its similarity to existing tasks. Overall, our results demonstrate that multiword expressions, especially those which are ambiguous, continue to be a challenge to models.