# Evaluating Adjective-Noun Compositionality in LLMs: Functional vs Representational Perspectives

arXiv: https://arxiv.org/abs/2603.09994 · HTML: https://mlsanity.com/paper/2603.09994/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 66 references that are in the map.
- First posted on arXiv 14 Feb 2026.

## Builds on (most cited first, 23 of 23)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023)
- [Compositionality decomposed: how do neural networks generalise?](https://mlsanity.com/paper/1908.08351/) (2019)
- [COGS: A Compositional Generalization Challenge Based on Semantic Interpretation](https://mlsanity.com/paper/2010.05465/) (2020)
- [LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations](https://mlsanity.com/paper/2410.02707/) (2024)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Understanding and Patching Compositional Reasoning in LLMs](https://mlsanity.com/paper/2402.14328/) (2024)
- [Characterizing Intrinsic Compositionality in Transformers with Tree Projections](https://mlsanity.com/paper/2211.01288/) (2022)
- [Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability](https://mlsanity.com/paper/2407.15720/) (2024)
- [How are Prompts Different in Terms of Sensitivity?](https://mlsanity.com/paper/2311.07230/) (2023)
- [Learned feature representations are biased by complexity, learning order, position, and more](https://mlsanity.com/paper/2405.05847/) (2024)
- [Posing Fair Generalization Tasks for Natural Language Inference](https://mlsanity.com/paper/1911.00811/) (2019)
- [Can LLM find the green circle? Investigation and Human-guided tool manipulation for compositional generalization](https://mlsanity.com/paper/2312.07763/) (2023)
- [Analyzing the Inner Workings of Transformers in Compositional Generalization](https://mlsanity.com/paper/2502.15277/) (2025)
- [Representing Syntax and Composition with Geometric Transformations](https://mlsanity.com/paper/2106.01904/) (2021)
- [Understanding Subword Compositionality of Large Language Models](https://mlsanity.com/paper/2508.17953/) (2025)
- [Quantifying Compositionality of Classic and State-of-the-Art Embeddings](https://mlsanity.com/paper/2509.19332/) (2025)
- [From Words to Worlds: Compositionality for Cognitive Architectures](https://mlsanity.com/paper/2407.13419/) (2024)

## Abstract (arXiv)

Compositionality is considered central to language abilities. As performant language systems, how do large language models (LLMs) do on compositional tasks? We evaluate adjective-noun compositionality in LLMs using two complementary setups: prompt-based functional assessment and a representational analysis of internal model states. Our results reveal a striking divergence between task performance and internal states. While LLMs reliably develop compositional representations, they fail to translate consistently into functional task success across model variants. Consequently, we highlight the importance of contrastive evaluation for obtaining a more complete understanding of model capabilities.