mlsanityData to 7 Oct 2026

PROST: Physical Reasoning of Objects through Space and Time

arXiv 2106.03634 ↗First posted 7 Jun 2021cs.CLcited by 48

Not on the map. Why?

Abstract (arXiv)

We present a new probing dataset named PROST: Physical Reasoning about Objects Through Space and Time. This dataset contains 18,736 multiple-choice questions made from 14 manually curated templates, covering 10 physical reasoning concepts. All questions are designed to probe both causal and masked language models in a zero-shot setting. We conduct an extensive analysis which demonstrates that state-of-the-art pretrained models are inadequate at physical reasoning: they are influenced by the order in which answer options are presented to them, they struggle when the superlative in a question is inverted (e.g., most <-> least), and increasing the amount of pretraining data and parameters only yields minimal improvements. These results provide support for the hypothesis that current pretrained models' ability to reason about physical interactions is inherently limited by a lack of real world experience. By highlighting these limitations, we hope to motivate the development of models with a human-like understanding of the physical world.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 36 of its 54 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

36 of 54 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
24 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  3. 2019Language Models as Knowledge Bases?cited by 1,922
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  5. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  6. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  7. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  8. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  9. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  10. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  11. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  12. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  13. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  14. 2020Experience Grounds Languagecited by 241
  15. 2019Assessing BERT's Syntactic Abilitiescited by 191
  16. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  17. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  18. 2019PHYRE: A New Benchmark for Physical Reasoningcited by 137
  19. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  20. 2020Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Modelscited by 110
  21. 2020Grounded Language Learning Fast and Slowcited by 54
  22. 2017Visual Semantic Planning using Deep Successor Representationscited by 45
  23. 2017Verb Physics: Relative Physical Knowledge of Actions and Objectscited by 26
  24. 2019How Large Are Lions? Inducing Distributions over Quantitative Attributescited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 10; 2023: 16; 2024: 8; 2025: 8; 2026: 4.

48 papers.

Sort:

By area:

  1. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  2. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  3. 2023A Comprehensive Overview of Large Language Modelscited by 639
  4. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  5. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  6. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  7. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  8. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
  9. 2023NEWTON: Are Large Language Models Capable of Physical Reasoning?cited by 45
  10. 2023Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statementscited by 44
  11. 2022Things not Written in Text: Exploring Spatial Commonsense from Visual Signalscited by 41
  12. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
36 more
  1. 2024Self-Boosting Large Language Models with Synthetic Preference Datacited by 40
  2. 2023MacGyver: Are Large Language Models Creative Problem Solvers?cited by 33
  3. 2021The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Colorcited by 32
  4. 2024LLMPhy: Parameter-Identifiable Physical Reasoning Combining Large Language Models and Physics Enginescited by 29
  5. 2022RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Questioncited by 17
  6. 2023VNHSGE: VietNamese High School Graduation Examination Dataset for Large Language Modelscited by 15
  7. 2023Crystal: Introspective Reasoners Reinforced with Self-Feedbackcited by 15
  8. 2022Predicting Fine-Tuning Performance with Probingcited by 13
  9. 2023Long-Tailed Question Answering in an Open Worldcited by 7
  10. 2024Desiderata for the Context Use of Question Answering Systemscited by 7
  11. 2025Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Designcited by 6
  12. 2022Pretraining on Interactions for Learning Grounded Affordance Representationscited by 5
  13. 2024Scene Exploration by Vision-Language Modelscited by 5
  14. 2024Precise Model Benchmarking with Only a Few Observationscited by 5
  15. 2021Natural Language Processing with Commonsense Knowledge: A Surveycited by 4
  16. 2023Probing Physical Reasoning with Counter-Commonsense Contextcited by 4
  17. 2025Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inferencecited by 4
  18. 2024TEXT2AFFORD: Probing Object Affordance Prediction abilities of Language Models solely from Textcited by 3
  19. 2024PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Datasetcited by 3
  20. 2025AgentCoMa: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarioscited by 3
  21. 2022Do Trajectories Encode Verb Meaning?cited by 2
  22. 2022On Reality and the Limits of Language Data: Aligning LLMs with Human Normscited by 2
  23. 2023POSQA: Probe the World Models of LLMs with Size Comparisonscited by 2
  24. 2023Catwalk: A Unified Language Model Evaluation Framework for Many Datasetscited by 2
  25. 2025WinoWhat: A Parallel Corpus of Paraphrased WinoGrande Sentences with Common Sense Categorizationcited by 2
  26. 2025InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoningcited by 2
  27. 2026It Takes a MAESTRO To Prune Bad Expertscited by 2
  28. 2026Modular Cognitive Architecture Emerges in Large Language Modelscited by 2
  29. 2023Implicit Affordance Acquisition via Causal Action-Effect Modeling in the Video Domaincited by 1
  30. 2024Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models?cited by 1
  31. 2025A Study of Commonsense Reasoning over Visual Object Propertiescited by 1
  32. 2025GuessingGame: Measuring the Informativeness of Open-Ended Questions in Large Language Modelscited by 1
  33. 2023Context Matters: Data-Efficient Augmentation of Large Language Models for Scientific Applicationscited by 0
  34. 2025From Threat to Tool: Leveraging Refusal-Aware Injection Attacks for Safety Alignmentcited by 0
  35. 2026Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectoriescited by 0
  36. 2026Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimizationcited by 0