mlsanityData to 14 Oct 2026

Effective Distillation of Table-based Reasoning Ability from LLMs

arXiv 2309.13182 ↗First posted 22 Sep 2023cs.CLcited by 23

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of natural language processing tasks. However, their enormous parameter size and extremely high requirements for compute power pose challenges for their practical deployment. Recent research has revealed that specific capabilities of LLMs, such as numerical reasoning, can be transferred to smaller models through distillation. Some studies explore the potential of leveraging LLMs to perform table-based reasoning. However, there has been no prior work focusing on table reasoning skills in smaller models specifically tailored for scientific table-to-text generation tasks. In this paper, we propose a novel table-based reasoning distillation approach, with the aim of distilling LLMs into tailored smaller models. Our experimental results have shown that a 220 million parameter model (Flan-T5-base) fine-tuned using distilled data, not only achieves a significant improvement compared to traditionally fine-tuned baselines, but also surpasses specific LLMs on a scientific table-to-text generation dataset. Our code is available at https://github.com/Bernard-Yang/DistillTableCoT.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 50 of its 58 references that are in the map. First posted on arXiv 22 Sep 2023.

Builds on

50 of 58 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables22 May
  2. 2023Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning31 Jan
  3. 2022ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples22 Oct
  4. 2022Large Language Models are few(1)-shot Table Reasoners13 Oct

Other references · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  12. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
34 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  3. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  4. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  5. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  6. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  7. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  8. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
  9. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 434
  10. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 428
  11. 2022Large Language Models Are Reasoning Teacherscited by 329
  12. 2022Teaching Small Language Models to Reasoncited by 284
  13. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 268
  14. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 251
  15. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 235
  16. 2022Binding Language Models in Symbolic Languagescited by 228
  17. 2017Challenges in Data-to-Document Generationcited by 226
  18. 2021FeTaQA: Free-form Table Question Answeringcited by 175
  19. 2021FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured informationcited by 167
  20. 2020Logical Natural Language Generation from Open-Domain Tablescited by 90
  21. 2020INFOTABS: Inference on Tables as Semi-structured Datacited by 86
  22. 2020Logic2Text: High-Fidelity Natural Language Generation from Logical Formscited by 45
  23. 2023Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategiescited by 45
  24. 2022OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answeringcited by 29
  25. 2021Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric Viewcited by 24
  26. 2022STAR: SQL Guided Pre-Training for Context-dependent Text-to-SQL Parsingcited by 19
  27. 2022EtriCA: Event-Triggered Context-Aware Story Generation Augmented by Cross Attentioncited by 17
  28. 2022PLOG: Table-to-Logic Pretraining for Logical Table-to-Text Generationcited by 16
  29. 2023TwistList: Resources and Baselines for Tongue Twister Generationcited by 16
  30. 2022Inducing and Using Alignments for Transition-based AMR Parsingcited by 12
  31. 2023Generate, Transform, Answer: Question Specific Tool Synthesis for Tabular Datacited by 11
  32. 2022Improving Chinese Story Generation via Awareness of Syntactic Dependencies and Semanticscited by 10
  33. 2023Improving Medical Dialogue Generation with Abstract Meaning Representationscited by 10
  34. 2019Surf at MEDIQA 2019: Improving Performance of Natural Language Inference in the Clinical Domain by Adopting Pre-trained Language Modelcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 10; 2025: 6; 2026: 2.

23 papers.

Sort:

By area:

  1. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  2. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 159
  3. 2024A Survey of Table Reasoning with Large Language Modelscited by 29
  4. 2024TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Datacited by 19
  5. 2025RealHiTBench: A Comprehensive Realistic Hierarchical Table Benchmark for Evaluating LLM-Based Table Analysiscited by 12
  6. 2023From Classification to Clinical Insights: Towards Analyzing and Reasoning About Mobile and Behavioral Health Data With Large Language Modelscited by 11
  7. 2024SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluationcited by 11
  8. 2024Crafting Customisable Characters with LLMs: A Persona-Driven Role-Playing Agent Frameworkcited by 10
  9. 2023Improving Biomedical Abstractive Summarisation with Knowledge Aggregation from Citation Paperscited by 9
  10. 2024Uncovering Limitations of Large Language Models in Information Seeking from Tablescited by 9
  11. 2025Mitigating Spurious Correlations Between Question and Answer via Chain-of-Thought Correctness Perception Distillationcited by 6
  12. 2023A Cross-Attention Augmented Model for Event-Triggered Context-Aware Story Generationcited by 5
11 more
  1. 2024Train & Constrain: Phonologically Informed Tongue-Twister Generation from Topics and Paraphrasescited by 3
  2. 2025Improving Table Understanding with LLMs and Entity-Oriented Searchcited by 3
  3. 2024Emphasising Structured Information: Integrating Abstract Meaning Representation into LLMs for Enhanced Open-Domain Dialogue Evaluationcited by 2
  4. 2025Planning for Success: Exploring LLM Long-term Planning Capabilities in Table Understandingcited by 2
  5. 2025Self-Correction Distillation for Structured Data Question Answeringcited by 2
  6. 2024TREB: a BERT attempt for imputing tabular data imputationcited by 1
  7. 2025ITUNLP at SemEval-2025 Task 8: Question-Answering over Tabular Data: A Zero-Shot Approach using LLM-Driven Code Generationcited by 1
  8. 2023ACL Anthology Helper: A Tool to Retrieve and Manage Literature from ACL Anthologycited by 0
  9. 2024BioMNER: A Dataset for Biomedical Method Entity Recognitioncited by 0
  10. 2026LLM-Enabled Multi-Agent Systems: Empirical Evaluation and Insights into Emerging Design Patterns & Paradigmscited by 0
  11. 2026Cross-Domain, Multi-Task Data-to-Text Generation without In-Domain Training Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Improving Mathematical Reasoning Capabilities of Small Language Models via Feedback-Driven Distillation
  2. 2023Mixed Distillation Helps Smaller Language Model Better Reasoning
  3. 2023TAP4LLM: Table Provider on Sampling, Augmenting, and Packing Semi-structured Data for Large Language Model Reasoning
  4. 2024Tree-of-Table: Unleashing the Power of LLMs for Enhanced Large-Scale Table Understanding
  5. 2024Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data
  6. 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillation
  7. 2025RUST-BENCH: Benchmarking LLM Reasoning on Unstructured Text within Structured Tables
  8. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application