# StructLM: Towards Building Generalist Models for Structured Knowledge Grounding

arXiv: https://arxiv.org/abs/2402.16671 · HTML: https://mlsanity.com/paper/2402.16671/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 65 references that are in the map.
- First posted on arXiv 26 Feb 2024.

## Builds on (most cited first, 30 of 56)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://mlsanity.com/paper/2308.09583/) (2023)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://mlsanity.com/paper/1909.02164/) (2019)
- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [The Web as a Knowledge-base for Answering Complex Questions](https://mlsanity.com/paper/1803.06643/) (2018)
- [Llemma: An Open Language Model For Mathematics](https://mlsanity.com/paper/2310.10631/) (2023)
- [Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning](https://mlsanity.com/paper/2209.14610/) (2022)
- [StructGPT: A General Framework for Large Language Model to Reason over Structured Data](https://mlsanity.com/paper/2305.09645/) (2023)
- [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data](https://mlsanity.com/paper/2004.07347/) (2020)
- [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://mlsanity.com/paper/2107.07653/) (2021)
- [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://mlsanity.com/paper/2201.05966/) (2022)
- [Binding Language Models in Symbolic Languages](https://mlsanity.com/paper/2210.02875/) (2022)
- [RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL](https://mlsanity.com/paper/2302.05965/) (2023)
- [Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning](https://mlsanity.com/paper/2301.13808/) (2023)

## Built on by (most cited first, 29 of 29)

- [TableBench: A Comprehensive and Complex Benchmark for Table Question Answering](https://mlsanity.com/paper/2408.09174/) (2024)
- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL](https://mlsanity.com/paper/2406.08426/) (2024)
- [Efficient Continual Pre-training by Mitigating the Stability Gap](https://mlsanity.com/paper/2406.14833/) (2024)
- [ProTrix: Building Models for Planning and Reasoning over Tables with Sentence Context](https://mlsanity.com/paper/2403.02177/) (2024)
- [Kanana: Compute-efficient Bilingual Language Models](https://mlsanity.com/paper/2502.18934/) (2025)
- [TARGET: Benchmarking Table Retrieval for Generative Tasks](https://mlsanity.com/paper/2505.11545/) (2025)
- [ExeSQL: Self-Taught Text-to-SQL Models with Execution-Driven Bootstrapping for SQL Dialects](https://mlsanity.com/paper/2505.17231/) (2025)
- [Table-LLM-Specialist: Language Model Specialists for Tables using Iterative Generator-Validator Fine-tuning](https://mlsanity.com/paper/2410.12164/) (2024)
- [Toward Structured Knowledge Reasoning: Contrastive Retrieval-Augmented Generation on Experience](https://mlsanity.com/paper/2506.00842/) (2025)
- [Grounding LLM Reasoning with Knowledge Graphs](https://mlsanity.com/paper/2502.13247/) (2025)
- [Table-r1: Self-supervised and Reinforcement Learning for Program-based Table Reasoning in Small Language Models](https://mlsanity.com/paper/2506.06137/) (2025)
- [Table as a Modality for Large Language Models](https://mlsanity.com/paper/2512.00947/) (2025)
- [Chain-of-Query: Unleashing the Power of LLMs in SQL-Aided Table Understanding via Multi-Agent Collaboration](https://mlsanity.com/paper/2508.15809/) (2025)
- [TabularMath: Understanding Math Reasoning over Tables with Large Language Models](https://mlsanity.com/paper/2505.19563/) (2025)
- [TableLoRA: Low-rank Adaptation on Table Structure Understanding for Large Language Models](https://mlsanity.com/paper/2503.04396/) (2025)
- [Pandora: A Code-Driven Large Language Model Agent for Unified Reasoning Across Diverse Structured Knowledge](https://mlsanity.com/paper/2504.12734/) (2025)
- [Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities](https://mlsanity.com/paper/2505.23838/) (2025)
- [Reasoning Factual Knowledge in Structured Data with Large Language Models](https://mlsanity.com/paper/2408.12188/) (2024)
- [TabVer: Tabular Fact Verification with Natural Logic](https://mlsanity.com/paper/2411.01093/) (2024)
- [TableZoomer: A Collaborative Agent Framework for Large-scale Table Question Answering](https://mlsanity.com/paper/2509.01312/) (2025)
- [Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning](https://mlsanity.com/paper/2603.24004/) (2026)
- [eC-Tab2Text: Aspect-Based Text Generation from e-Commerce Product Tables](https://mlsanity.com/paper/2502.14820/) (2025)
- [TableReasoner: Advancing Table Reasoning Framework with Large Language Models](https://mlsanity.com/paper/2507.08046/) (2025)
- [TopBench: A Benchmark for Implicit Predictive Reasoning in Tabular Question Answering](https://mlsanity.com/paper/2604.28076/) (2026)
- [Pandora: Leveraging Code-driven Knowledge Transfer for Unified Structured Knowledge Reasoning](https://mlsanity.com/paper/2508.17905/) (2025)
- [LLaSA: Large Language and Structured Data Assistant](https://mlsanity.com/paper/2411.14460/) (2024)
- [Structure-BiEval: A Self-Supervised, Dual-Track Framework for Decoupling Structure and Content in LLM Evaluation for Web Information Systems](https://mlsanity.com/paper/2601.19923/) (2026)
- [DeepTable: Structural Attention Biases and Tree Path Encoding for Hierarchical Table Understanding](https://mlsanity.com/paper/2609.07707/) (2026)

## Abstract (arXiv)

Structured data sources, such as tables, graphs, and databases, are ubiquitous knowledge sources. Despite the demonstrated capabilities of large language models (LLMs) on plain text, their proficiency in interpreting and utilizing structured data remains limited. Our investigation reveals a notable deficiency in LLMs' ability to process structured data, e.g., ChatGPT lags behind state-of-the-art (SoTA) model by an average of 35%. To augment the Structured Knowledge Grounding (SKG) capabilities in LLMs, we have developed a comprehensive instruction tuning dataset comprising 1.1 million examples. Utilizing this dataset, we train a series of models, referred to as StructLM, based on the Mistral and the CodeLlama model family, ranging from 7B to 34B parameters. Our StructLM series surpasses task-specific models on 16 out of 18 evaluated datasets and establishes new SoTA performance on 8 SKG tasks. Furthermore, StructLM demonstrates strong generalization across 6 novel held-out SKG tasks, outperforming TableLlama by an average of 35% and Flan-UL2 20B by an average of 10%. Contrary to expectations, we observe that scaling model size offers marginal benefits, with StructLM-34B showing only slight improvements over StructLM-7B. This suggests that structured knowledge grounding is still a challenging task and requires more innovative design to push to a new level.