# MASSW: A New Dataset and Benchmark Tasks for AI-Assisted Scientific Workflows

arXiv: https://arxiv.org/abs/2406.06357 · HTML: https://mlsanity.com/paper/2406.06357/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 54 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 25 of 25)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Multilingual E5 Text Embeddings: A Technical Report](https://mlsanity.com/paper/2402.05672/) (2024)
- [A Survey of Evaluation Metrics Used for NLG Systems](https://mlsanity.com/paper/2008.12009/) (2020)
- [TLDR: Extreme Summarization of Scientific Documents](https://mlsanity.com/paper/2004.15011/) (2020)
- [PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstracts](https://mlsanity.com/paper/1710.06071/) (2017)
- [Visualizing Large-scale and High-dimensional Data](https://mlsanity.com/paper/1602.00370/) (2016)
- [POTATO: The Portable Text Annotation Tool](https://mlsanity.com/paper/2212.08620/) (2022)
- [LitLLM: A Toolkit for Scientific Literature Review](https://mlsanity.com/paper/2402.01788/) (2024)
- [On the Creativity of Large Language Models](https://mlsanity.com/paper/2304.00008/) (2023)
- [WikiAsp: A Dataset for Multi-domain Aspect-based Summarization](https://mlsanity.com/paper/2011.07832/) (2020)
- [Pretrained Language Models for Sequential Sentence Classification](https://mlsanity.com/paper/1909.04054/) (2019)
- [SQuALITY: Building a Long-Document Summarization Dataset the Hard Way](https://mlsanity.com/paper/2205.11465/) (2022)
- [Accelerating Innovation Through Analogy Mining](https://mlsanity.com/paper/1706.05585/) (2017)
- [Scim: Intelligent Skimming Support for Scientific Papers](https://mlsanity.com/paper/2205.04561/) (2022)
- [OASum: Large-Scale Open Domain Aspect-based Summarization](https://mlsanity.com/paper/2212.09233/) (2022)
- [ACLSum: A New Dataset for Aspect-based Summarization of Scientific Publications](https://mlsanity.com/paper/2403.05303/) (2024)
- [OpenAsp: A Benchmark for Multi-document Open Aspect-based Summarization](https://mlsanity.com/paper/2312.04440/) (2023)
- [Synergizing Human Expertise and AI Efficiency with Language Model for Microscopy Operation and Automated Experiment Design](https://mlsanity.com/paper/2401.13803/) (2024)
- [LexAbSumm: Aspect-based Summarization of Legal Decisions](https://mlsanity.com/paper/2404.00594/) (2024)

## Built on by (most cited first, 19 of 19)

- [Agent Laboratory: Using LLM Agents as Research Assistants](https://mlsanity.com/paper/2501.04227/) (2025)
- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://mlsanity.com/paper/2409.04109/) (2024)
- [AgentRxiv: Towards Collaborative Autonomous Research](https://mlsanity.com/paper/2503.18102/) (2025)
- [Automated Hypothesis Validation with Agentic Sequential Falsifications](https://mlsanity.com/paper/2502.09858/) (2025)
- [Cocoa: Co-Planning and Co-Execution with AI Agents](https://mlsanity.com/paper/2412.10999/) (2024)
- [HypoBench: Towards Systematic and Principled Benchmarking for Hypothesis Generation](https://mlsanity.com/paper/2504.11524/) (2025)
- [SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding](https://mlsanity.com/paper/2408.15545/) (2024)
- [Literature Meets Data: A Synergistic Approach to Hypothesis Generation](https://mlsanity.com/paper/2410.17309/) (2024)
- [Be.FM: Open Foundation Models for Human Behavior](https://mlsanity.com/paper/2505.23058/) (2025)
- [Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluator](https://mlsanity.com/paper/2507.11810/) (2025)
- [From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systems](https://mlsanity.com/paper/2503.01424/) (2025)
- [CLIMATEAGENT: Multi-Agent Orchestration for Complex Climate Data Science Workflows](https://mlsanity.com/paper/2511.20109/) (2025)
- [CHIMERA: A Knowledge Base of Scientific Idea Recombinations for Research Analysis and Ideation](https://mlsanity.com/paper/2505.20779/) (2025)
- [EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistants](https://mlsanity.com/paper/2502.20309/) (2025)
- [Idea2Plan: Exploring AI-Powered Research Planning](https://mlsanity.com/paper/2510.24891/) (2025)
- [Bridging AI and Science: Implications from a Large-Scale Literature Analysis of AI4Science](https://mlsanity.com/paper/2412.09628/) (2024)
- [MapExplorer: New Content Generation from Low-Dimensional Visualizations](https://mlsanity.com/paper/2412.18673/) (2024)
- [Language Models Should be Used to Surface the Unwritten Code of Science and Society](https://mlsanity.com/paper/2505.18942/) (2025)
- [BehaviorBench: Benchmarking Foundation Models for Behavioral Science Tasks](https://mlsanity.com/paper/2606.24162/) (2026)

## Abstract (arXiv)

Scientific innovation relies on detailed workflows, which include critical steps such as analyzing literature, generating ideas, validating these ideas, interpreting results, and inspiring follow-up research. However, scientific publications that document these workflows are extensive and unstructured. This makes it difficult for both human researchers and AI systems to effectively navigate and explore the space of scientific innovation. To address this issue, we introduce MASSW, a comprehensive text dataset on Multi-Aspect Summarization of Scientific Workflows. MASSW includes more than 152,000 peer-reviewed publications from 17 leading computer science conferences spanning the past 50 years. Using Large Language Models (LLMs), we automatically extract five core aspects from these publications -- context, key idea, method, outcome, and projected impact -- which correspond to five key steps in the research workflow. These structured summaries facilitate a variety of downstream tasks and analyses. The quality of the LLM-extracted summaries is validated by comparing them with human annotations. We demonstrate the utility of MASSW through multiple novel machine-learning tasks that can be benchmarked using this new dataset, which make various types of predictions and recommendations along the scientific workflow. MASSW holds significant potential for researchers to create and benchmark new AI methods for optimizing scientific workflows and fostering scientific innovation in the field. Our dataset is openly available at urlhttps://github.com/xingjian-zhang/massw.