# Can Large Language Models Understand Context?

arXiv: https://arxiv.org/abs/2402.00858 · HTML: https://mlsanity.com/paper/2402.00858/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 65 references that are in the map.
- First posted on arXiv 1 Feb 2024.

## Builds on (most cited first, 30 of 36)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [End-to-end Neural Coreference Resolution](https://mlsanity.com/paper/1707.07045/) (2017)
- [MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines](https://mlsanity.com/paper/2007.12720/) (2020)
- [Open-Domain Question Answering Goes Conversational via Question Rewriting](https://mlsanity.com/paper/2010.04898/) (2020)
- [Neural Belief Tracker: Data-Driven Dialogue State Tracking](https://mlsanity.com/paper/1606.03777/) (2016)
- [Focused Transformer: Contrastive Training for Context Scaling](https://mlsanity.com/paper/2307.03170/) (2023)
- [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023)
- [Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization](https://mlsanity.com/paper/2305.14152/) (2023)
- [CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset](https://mlsanity.com/paper/2002.11893/) (2020)
- [Improving Multi-turn Dialogue Modelling with Utterance ReWriter](https://mlsanity.com/paper/1906.07004/) (2019)
- [A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset](https://mlsanity.com/paper/1806.00358/) (2018)
- [Pruning vs Quantization: Which is Better?](https://mlsanity.com/paper/2307.02973/) (2023)
- [ChatGPT for Zero-shot Dialogue State Tracking: A Solution or an Opportunity?](https://mlsanity.com/paper/2306.01386/) (2023)
- [Coreference Resolution through a seq2seq Transition-Based System](https://mlsanity.com/paper/2211.12142/) (2022)

## Built on by (most cited first, 26 of 26)

- [Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?](https://mlsanity.com/paper/2410.23856/) (2024)
- [PropertyGPT: LLM-driven Formal Verification of Smart Contracts through Retrieval-Augmented Property Generation](https://mlsanity.com/paper/2405.02580/) (2024)
- [TableMaster: A Recipe to Advance Table Understanding with Language Models](https://mlsanity.com/paper/2501.19378/) (2025)
- [Bridging Context Gaps: Leveraging Coreference Resolution for Long Contextual Understanding](https://mlsanity.com/paper/2410.01671/) (2024)
- [Graph2text or Graph2token: A Perspective of Large Language Models for Graph Learning](https://mlsanity.com/paper/2501.01124/) (2025)
- [Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICs](https://mlsanity.com/paper/2412.14304/) (2024)
- [Evaluating the Sensitivity of LLMs to Prior Context](https://mlsanity.com/paper/2506.00069/) (2025)
- [Why Not Act on What You Know? Unleashing Safety Potential of LLMs via Self-Aware Guard Enhancement](https://mlsanity.com/paper/2505.12060/) (2025)
- [Large Language Models for Automatic Detection of Sensitive Topics](https://mlsanity.com/paper/2409.00940/) (2024)
- [Distinct social-linguistic processing between humans and large audio-language models: Evidence from model-brain alignment](https://mlsanity.com/paper/2503.19586/) (2025)
- [ExpressivityBench: Can LLMs Communicate Implicitly?](https://mlsanity.com/paper/2411.08010/) (2024)
- [Envisioning an AI-Enhanced Mental Health Ecosystem](https://mlsanity.com/paper/2503.14883/) (2025)
- [BanglaIPA: Towards Robust Text-to-IPA Transcription with Contextual Rewriting in Bengali](https://mlsanity.com/paper/2601.01778/) (2026)
- [A Novel Multi-Agent Architecture to Reduce Hallucinations of Large Language Models in Multi-Step Structural Modeling](https://mlsanity.com/paper/2603.07728/) (2026)
- [Solving the Challenge Set without Solving the Task: On Winograd Schemas as a Test of Pronominal Coreference Resolution](https://mlsanity.com/paper/2410.09448/) (2024)
- [A Review on Large Language Models for Visual Analytics](https://mlsanity.com/paper/2503.15176/) (2025)
- [Multimodal Transformer Models for Turn-taking Prediction: Effects on Conversational Dynamics of Human-Agent Interaction during Cooperative Gameplay](https://mlsanity.com/paper/2503.16432/) (2025)
- [Human vs. LLM-Based Thematic Analysis for Digital Mental Health Research: Proof-of-Concept Comparative Study](https://mlsanity.com/paper/2507.08002/) (2025)
- [Performance Gains of LLMs With Humans in a World of LLMs Versus Humans](https://mlsanity.com/paper/2505.08902/) (2025)
- [GAP: Graph-Assisted Prompts for Dialogue-based Medication Recommendation](https://mlsanity.com/paper/2505.12888/) (2025)
- [The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes](https://mlsanity.com/paper/2606.11470/) (2026)
- [The Boundaries of Automation: A Theory of Persistent Human Participation](https://mlsanity.com/paper/2607.21547/) (2026)
- [Can Language Models Understand mmWave Data? Benchmarking Large Language Models for mmWave Radar-Based Human Understanding](https://mlsanity.com/paper/2608.14179/) (2026)
- [Scaling Multi-Agent Systems with Prospect-State Propagation](https://mlsanity.com/paper/2609.08033/) (2026)
- [Evaluation of Contextual Understanding in Large Language Models](https://mlsanity.com/paper/2609.09004/) (2026)
- [Do LLMs Understand Context? A Knowledge Graph-Based Evaluation Framework](https://mlsanity.com/paper/2609.30484/) (2026)

## Abstract (arXiv)

Understanding context is key to understanding human language, an ability which Large Language Models (LLMs) have been increasingly seen to demonstrate to an impressive extent. However, though the evaluation of LLMs encompasses various domains within the realm of Natural Language Processing, limited attention has been paid to probing their linguistic capability of understanding contextual features. This paper introduces a context understanding benchmark by adapting existing datasets to suit the evaluation of generative models. This benchmark comprises of four distinct tasks and nine datasets, all featuring prompts designed to assess the models' ability to understand context. First, we evaluate the performance of LLMs under the in-context learning pretraining scenario. Experimental results indicate that pre-trained dense models struggle with understanding more nuanced contextual features when compared to state-of-the-art fine-tuned models. Second, as LLM compression holds growing significance in both research and real-world applications, we assess the context understanding of quantized models under in-context-learning settings. We find that 3-bit post-training quantization leads to varying degrees of performance reduction on our benchmark. We conduct an extensive analysis of these scenarios to substantiate our experimental results.