mlsanityData to 7 Oct 2026

Reasoning Factual Knowledge in Structured Data with Large Language Models

arXiv 2408.12188 ↗First posted 22 Aug 2024cs.CL cs.AIcited by 3

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) have made remarkable progress in various natural language processing tasks as a benefit of their capability to comprehend and reason with factual knowledge. However, a significant amount of factual knowledge is stored in structured data, which possesses unique characteristics that differ from the unstructured texts used for pretraining. This difference can introduce imperceptible inference parameter deviations, posing challenges for LLMs in effectively utilizing and reasoning with structured data to accurately infer factual knowledge. To this end, we propose a benchmark named StructFact, to evaluate the structural reasoning capabilities of LLMs in inferring factual knowledge. StructFact comprises 8,340 factual questions encompassing various tasks, domains, timelines, and regions. This benchmark allows us to investigate the capability of LLMs across five factual tasks derived from the unique characteristics of structural facts. Extensive experiments on a set of LLMs with different training strategies reveal the limitations of current LLMs in inferring factual knowledge from structured data. We present this benchmark as a compass to navigate the strengths and weaknesses of LLMs in reasoning with structured data for knowledge-sensitive tasks, and to encourage advancements in related real-world applications. Please find our code at https://github.com/EganGu/StructFact.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 23 of its 66 references that are in the map. First posted on arXiv 22 Aug 2024.

Builds on

23 of 66 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools18 Jun
  2. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 21

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2024Qwen2 Technical Reportcited by 2,076
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  5. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  6. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  7. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  8. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  9. 2023Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sourcescited by 121
  10. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 98
  11. 2021Open Domain Question Answering over Tables via Dense Retrievalcited by 74
  12. 2023Tabular Representation, Noisy Operators, and Impacts on Table Structure Understanding Tasks in LLMscited by 49
9 more
  1. 2024Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMscited by 40
  2. 2023Rethinking Tabular Data Understanding with Large Language Modelscited by 34
  3. 2024StructLM: Towards Building Generalist Models for Structured Knowledge Groundingcited by 29
  4. 2023TABLET: Learning From Instructions For Tabular Datacited by 28
  5. 2021Combining pre-trained language models and structured knowledgecited by 25
  6. 2024Tabular Data Augmentation for Machine Learning: Progress and Prospects of Embracing Generative AIcited by 15
  7. 2023Investigating Table-to-Text Generation Capabilities of LLMs in Real-World Information Seeking Scenarioscited by 12
  8. 2024TrustUQA: A Trustful Framework for Unified Structured Data Question Answeringcited by 12
  9. 2024MFORT-QA: Multi-hop Few-shot Open Rich Table Question Answeringcited by 7

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2025Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoningcited by 23
  3. 2024Self-Explained Keywords Empower Large Language Models for Code Generationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024StrucText-Eval: Evaluating Large Language Model's Reasoning Ability in Structure-Rich Text
  2. 2024Struct-X: Enhancing Large Language Models Reasoning with Structured Data
  3. 2024StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs
  4. 2025LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data
  5. 2026π2: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models
  6. 2025Improving Factuality in LLMs via Inference-Time Knowledge Graph Construction
  7. 2025RealFactBench: A Benchmark for Evaluating Large Language Models in Real-World Fact-Checking
  8. 2024CLR-Fact: Evaluating the Complex Logical Reasoning Capability of Large Language Models over Factual Knowledge