# PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion

arXiv: https://arxiv.org/abs/2311.01767 · HTML: https://mlsanity.com/paper/2311.01767/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 3 Nov 2023.

## Builds on (most cited first, 30 of 30)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs](https://mlsanity.com/paper/2210.12283/) (2022)
- [TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs](https://mlsanity.com/paper/2303.16434/) (2023)
- [Is ChatGPT a Good Sentiment Analyzer? A Preliminary Study](https://mlsanity.com/paper/2304.04339/) (2023)

## Built on by (most cited first, 30 of 30)

- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [OS-Copilot: Towards Generalist Computer Agents with Self-Improvement](https://mlsanity.com/paper/2402.07456/) (2024)
- [PPTAgent: Generating and Evaluating Presentations Beyond Text-to-Slides](https://mlsanity.com/paper/2501.03936/) (2025)
- [AutoPresent: Designing Structured Visuals from Scratch](https://mlsanity.com/paper/2501.00912/) (2025)
- [Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents](https://mlsanity.com/paper/2407.00993/) (2024)
- [ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents](https://mlsanity.com/paper/2508.14040/) (2025)
- [AgentStudio: A Toolkit for Building General Virtual Agents](https://mlsanity.com/paper/2403.17918/) (2024)
- [OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use](https://mlsanity.com/paper/2508.04482/) (2025)
- [Foundations and Recent Trends in Multimodal Mobile Agents: A Survey](https://mlsanity.com/paper/2411.02006/) (2024)
- [Efficient Continual Pre-training by Mitigating the Stability Gap](https://mlsanity.com/paper/2406.14833/) (2024)
- [PPTBench: Towards Holistic Evaluation of Large Language Models for PowerPoint Layout and Design Understanding](https://mlsanity.com/paper/2512.02624/) (2025)
- [Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systems](https://mlsanity.com/paper/2411.09972/) (2024)
- [Talk to Your Slides: High-Efficiency Slide Editing via Language-Driven Structured Data Manipulation](https://mlsanity.com/paper/2505.11604/) (2025)
- [PPTArena: A Benchmark for PowerPoint Editing](https://mlsanity.com/paper/2512.03042/) (2025)
- [DECKBench: Benchmarking Multi-Agent Frameworks for Academic Slide Generation and Editing](https://mlsanity.com/paper/2602.13318/) (2026)
- [PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion](https://mlsanity.com/paper/2403.03788/) (2024)
- [PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks](https://mlsanity.com/paper/2606.31154/) (2026)
- [VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Models](https://mlsanity.com/paper/2312.04087/) (2023)
- [APEX: Academic Poster Editing Agentic Expert](https://mlsanity.com/paper/2601.04794/) (2026)
- [Textual-to-Visual Iterative Self-Verification for Slide Generation](https://mlsanity.com/paper/2502.15412/) (2025)
- [VLM Q-Learning: Aligning Vision-Language Models for Interactive Decision-Making](https://mlsanity.com/paper/2505.03181/) (2025)
- [Design First, Code Later: Aesthetically Pleasing Template-Free Slides Generation](https://mlsanity.com/paper/2605.26451/) (2026)
- [OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding](https://mlsanity.com/paper/2607.27155/) (2026)
- [Multi-Agent Synergy-Driven Iterative Visual Narrative Synthesis](https://mlsanity.com/paper/2507.13285/) (2025)
- [How Should Agents Read Demonstrations? Hierarchical Structure Beats Flat Action Logs](https://mlsanity.com/paper/2606.20978/) (2026)
- [PPTBench: Can Coding Agents Reconstruct the Visual World through Structured, Editable Slides](https://mlsanity.com/paper/2609.29718/) (2026)
- [Don't Shoot The Breeze: Topic Continuity Model Using Nonlinear Naive Bayes With Attention](https://mlsanity.com/paper/2602.09312/) (2026)
- [DeepSlide: From Artifacts to Presentation Delivery](https://mlsanity.com/paper/2605.15202/) (2026)
- [ReDeck: Step-Level Render-Grounded Refinement for Document-to-Slide Generation](https://mlsanity.com/paper/2609.00194/) (2026)
- [ReFigBench: Benchmarking Scientific Figure Reconstruction as Editable PowerPoint Artifacts](https://mlsanity.com/paper/2609.18844/) (2026)

## Abstract (arXiv)

Recent evaluations of Large Language Models (LLMs) have centered around testing their zero-shot/few-shot capabilities for basic natural language tasks and their ability to translate instructions into tool APIs. However, the evaluation of LLMs utilizing complex tools to finish multi-turn, multi-modal instructions in a complex multi-modal environment has not been investigated. To address this gap, we introduce the PowerPoint Task Completion (PPTC) benchmark to assess LLMs' ability to create and edit PPT files based on user instructions. It contains 279 multi-turn sessions covering diverse topics and hundreds of instructions involving multi-modal operations. We also propose the PPTX-Match Evaluation System that evaluates if LLMs finish the instruction based on the prediction file rather than the label API sequence, thus it supports various LLM-generated API sequences. We measure 3 closed LLMs and 6 open-source LLMs. The results show that GPT-4 outperforms other LLMs with 75.1% accuracy in single-turn dialogue testing but faces challenges in completing entire sessions, achieving just 6% session accuracy. We find three main error causes in our benchmark: error accumulation in the multi-turn session, long PPT template processing, and multi-modality perception. These pose great challenges for future LLM and agent systems. We release the data, code, and evaluation system of PPTC at urlhttps://github.com/gydpku/PPTC.