# "You Are An Expert Linguistic Annotator": Limits of LLMs as Analyzers of Abstract Meaning Representation

arXiv: https://arxiv.org/abs/2310.17793 · HTML: https://mlsanity.com/paper/2310.17793/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 26 references that are in the map.
- First posted on arXiv 26 Oct 2023.

## Builds on (most cited first, 12 of 12)

- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [Constrained Language Models Yield Few-Shot Semantic Parsers](https://mlsanity.com/paper/2104.08768/) (2021)
- [Graph Pre-training for AMR Parsing and Generation](https://mlsanity.com/paper/2203.07836/) (2022)
- [Few-Shot Semantic Parsing with Language Models Trained On Code](https://mlsanity.com/paper/2112.08696/) (2021)
- [Maximum Bayes Smatch Ensemble Distillation for AMR Parsing](https://mlsanity.com/paper/2112.07790/) (2021)
- [SMATCH++: Standardized and Extended Evaluation of Semantic Graphs](https://mlsanity.com/paper/2305.06993/) (2023)
- [Comprehensive Supersense Disambiguation of English Prepositions and Possessives](https://mlsanity.com/paper/1805.04905/) (2018)
- [Weisfeiler-Leman in the BAMBOO: Novel AMR Graph Metrics and a Benchmark for AMR Graph Similarity](https://mlsanity.com/paper/2108.11949/) (2021)
- [SemBleu: A Robust Metric for AMR Parsing Evaluation](https://mlsanity.com/paper/1905.10726/) (2019)
- [SEMA: an Extended Semantic Evaluation Metric for AMR](https://mlsanity.com/paper/1905.12069/) (2019)

## Built on by (most cited first, 15 of 15)

- ["Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?](https://mlsanity.com/paper/2406.17600/) (2024)
- [Superlatives in Context: Modeling the Implicit Semantics of Superlatives](https://mlsanity.com/paper/2405.20967/) (2024)
- [MASSIVE Multilingual Abstract Meaning Representation: A Dataset and Baselines for Hallucination Detection](https://mlsanity.com/paper/2405.19285/) (2024)
- [Characterizing Language Use in a Collaborative Situated Game](https://mlsanity.com/paper/2512.03381/) (2025)
- [Annotating FrameNet via Structure-Conditioned Language Generation](https://mlsanity.com/paper/2406.04834/) (2024)
- [Retrieval-Augmented Semantic Parsing: Improving Generalization with Lexical Knowledge](https://mlsanity.com/paper/2412.10207/) (2024)
- [Mapping the Course for Prompt-based Structured Prediction](https://mlsanity.com/paper/2508.15090/) (2025)
- [Grounded Misunderstandings in Asymmetric Dialogue: A Perspectivist Annotation Scheme for MapTask](https://mlsanity.com/paper/2511.03718/) (2025)
- [Can Uniform Meaning Representation Help GPT-4 Translate from Indigenous Languages?](https://mlsanity.com/paper/2502.08900/) (2025)
- [Generating Text from Uniform Meaning Representation](https://mlsanity.com/paper/2502.11973/) (2025)
- [Statistical Parsing for Logical Information Retrieval](https://mlsanity.com/paper/2602.12170/) (2026)
- [Speaking of Language: Reflections on Metalanguage Research in NLP](https://mlsanity.com/paper/2604.02645/) (2026)
- [Multi-Modal Semantic Parsing for the Interpretation of Tombstone Inscriptions](https://mlsanity.com/paper/2507.04377/) (2025)
- [ding-01 :ARG0: An AMR Corpus for Spontaneous French Dialogue](https://mlsanity.com/paper/2508.12819/) (2025)
- [Scoring Both Directions: LLMs realize the MRS they cannot reliably parse](https://mlsanity.com/paper/2609.30071/) (2026)

## Abstract (arXiv)

Large language models (LLMs) show amazing proficiency and fluency in the use of language. Does this mean that they have also acquired insightful linguistic knowledge about the language, to an extent that they can serve as an "expert linguistic annotator"? In this paper, we examine the successes and limitations of the GPT-3, ChatGPT, and GPT-4 models in analysis of sentence meaning structure, focusing on the Abstract Meaning Representation (AMR; Banarescu et al. 2013) parsing formalism, which provides rich graphical representations of sentence meaning structure while abstracting away from surface forms. We compare models' analysis of this semantic structure across two settings: 1) direct production of AMR parses based on zero- and few-shot prompts, and 2) indirect partial reconstruction of AMR via metalinguistic natural language queries (e.g., "Identify the primary event of this sentence, and the predicate corresponding to that event."). Across these settings, we find that models can reliably reproduce the basic format of AMR, and can often capture core event, argument, and modifier structure -- however, model outputs are prone to frequent and major errors, and holistic analysis of parse acceptability shows that even with few-shot demonstrations, models have virtually 0% success in producing fully accurate parses. Eliciting natural language responses produces similar patterns of errors. Overall, our findings indicate that these models out-of-the-box can capture aspects of semantic structure, but there remain key limitations in their ability to support fully accurate semantic analyses or parses.