# Can LLMs Extract Frame-Semantic Arguments?

arXiv: https://arxiv.org/abs/2502.12516 · HTML: https://mlsanity.com/paper/2502.12516/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation](https://mlsanity.com/paper/2308.15363/) (2023)
- [MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning](https://mlsanity.com/paper/2310.16049/) (2023)
- [Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting](https://mlsanity.com/paper/2305.07004/) (2023)
- [Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models](https://mlsanity.com/paper/2408.02442/) (2024)
- [Frame-Semantic Parsing with Softmax-Margin Segmental RNNs and a Syntactic Scaffold](https://mlsanity.com/paper/1706.09528/) (2017)
- [Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings](https://mlsanity.com/paper/2410.13153/) (2024)
- [A Double-Graph Based Framework for Frame Semantic Parsing](https://mlsanity.com/paper/2206.09158/) (2022)
- [Potential and Limitations of LLMs in Capturing Structured Semantics: A Case Study on SRL](https://mlsanity.com/paper/2405.06410/) (2024)
- [Coarse-to-Fine Dual Encoders are Better Frame Identification Learners](https://mlsanity.com/paper/2310.13316/) (2023)
- [Output Format Biases in the Evaluation of Large Language Models for Code Translation](https://mlsanity.com/paper/2403.17214/) (2024)
- [Encoding Syntactic Constituency Paths for Frame-Semantic Parsing with Graph Convolutional Networks](https://mlsanity.com/paper/2011.13210/) (2020)
- [A Graph-Based Neural Model for End-to-End Frame Semantic Parsing](https://mlsanity.com/paper/2109.12319/) (2021)
- [Annotating FrameNet via Structure-Conditioned Language Generation](https://mlsanity.com/paper/2406.04834/) (2024)
- [Knowledge-augmented Frame Semantic Parsing with Hybrid Prompt-tuning](https://mlsanity.com/paper/2303.14375/) (2023)

## Built on by (most cited first, 3 of 3)

- [FRASE: Structured Representations for Generalizable SPARQL Query Generation](https://mlsanity.com/paper/2503.22144/) (2025)
- [Evaluating the Impact of LLM-Assisted Annotation in a Perspectivized Setting: the Case of FrameNet Annotation](https://mlsanity.com/paper/2510.25904/) (2025)
- [The Multilingual FrameNet Corpus](https://mlsanity.com/paper/2608.23037/) (2026)

## Abstract (arXiv)

Frame-semantic parsing is a critical task in natural language understanding, yet the ability of large language models (LLMs) to extract frame-semantic arguments remains underexplored. This paper presents a comprehensive evaluation of LLMs on frame-semantic argument identification, analyzing the impact of input representation formats, model architectures, and generalization to unseen and out-of-domain samples. Our experiments, spanning models from 0.5B to 78B parameters, reveal that JSON-based representations significantly enhance performance, and while larger models generally perform better, smaller models can achieve competitive results through fine-tuning. We also introduce a novel approach to frame identification leveraging predicted frame elements, achieving state-of-the-art performance on ambiguous targets. Despite strong generalization capabilities, our analysis finds that LLMs still struggle with out-of-domain data.