# TACT: Advancing Complex Aggregative Reasoning with Information Extraction Tools

arXiv: https://arxiv.org/abs/2406.03618 · HTML: https://mlsanity.com/paper/2406.03618/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 15 of its 67 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 15 of 15)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [LooGLE: Can Long-Context Language Models Understand Long Contexts?](https://mlsanity.com/paper/2311.04939/) (2023)
- [TALM: Tool Augmented Language Models](https://mlsanity.com/paper/2205.12255/) (2022)
- [MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning](https://mlsanity.com/paper/2402.17231/) (2024)
- [FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models](https://mlsanity.com/paper/2402.14116/) (2024)
- [Leveraging Code to Improve In-context Learning for Semantic Parsing](https://mlsanity.com/paper/2311.09519/) (2023)
- [FinLLMs: A Framework for Financial Reasoning Dataset Generation with Large Language Models](https://mlsanity.com/paper/2401.10744/) (2024)

## Built on by (most cited first, 5 of 5)

- [Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP](https://mlsanity.com/paper/2407.00402/) (2024)
- [CoverBench: A Challenging Benchmark for Complex Claim Verification](https://mlsanity.com/paper/2408.03325/) (2024)
- [AI Research Agents Narrow Scientific Exploration](https://mlsanity.com/paper/2605.27905/) (2026)
- [Structuring the Unstructured: A Systematic Review of Text-to-Structure Generation for Agentic AI with a Universal Evaluation Framework](https://mlsanity.com/paper/2508.12257/) (2025)
- [Visual Editing with LLM-based Tool Chaining: An Efficient Distillation Approach for Real-Time Applications](https://mlsanity.com/paper/2410.02952/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) often do not perform well on queries that require the aggregation of information across texts. To better evaluate this setting and facilitate modeling efforts, we introduce TACT - Text And Calculations through Tables, a dataset crafted to evaluate LLMs' reasoning and computational abilities using complex instructions. TACT contains challenging instructions that demand stitching information scattered across one or more texts, and performing complex integration on this information to generate the answer. We construct this dataset by leveraging an existing dataset of texts and their associated tables. For each such tables, we formulate new queries, and gather their respective answers. We demonstrate that all contemporary LLMs perform poorly on this dataset, achieving an accuracy below 38%. To pinpoint the difficulties and thoroughly dissect the problem, we analyze model performance across three components: table-generation, Pandas command-generation, and execution. Unexpectedly, we discover that each component presents substantial challenges for current LLMs. These insights lead us to propose a focused modeling framework, which we refer to as IE as a tool. Specifically, we propose to add "tools" for each of the above steps, and implement each such tool with few-shot prompting. This approach shows an improvement over existing prompting techniques, offering a promising direction for enhancing model capabilities in these tasks.