# PhysNLU: A Language Resource for Evaluating Natural Language Understanding and Explanation Coherence in Physics

arXiv: https://arxiv.org/abs/2201.04275 · HTML: https://mlsanity.com/paper/2201.04275/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 38 references that are in the map.
- First posted on arXiv 12 Jan 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers](https://mlsanity.com/paper/2106.15772/) (2021)
- [Proof Artifact Co-training for Theorem Proving with Language Models](https://mlsanity.com/paper/2102.06203/) (2021)
- [MathBERT: A Pre-Trained Model for Mathematical Formula Understanding](https://mlsanity.com/paper/2105.00377/) (2021)
- [Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models](https://mlsanity.com/paper/2005.10389/) (2020)
- [MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education](https://mlsanity.com/paper/2106.07340/) (2021)
- [HolStep: A Machine Learning Dataset for Higher-order Logic Theorem Proving](https://mlsanity.com/paper/1703.00426/) (2017)
- [Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations](https://mlsanity.com/paper/1909.00142/) (2019)
- [Natural Language Premise Selection: Finding Supporting Statements for Mathematical Text](https://mlsanity.com/paper/2004.14959/) (2020)
- [Learning to Prove from Synthetic Theorems](https://mlsanity.com/paper/2006.11259/) (2020)
- [Learning Equational Theorem Proving](https://mlsanity.com/paper/2102.05547/) (2021)
- [Similarity-Based Equational Inference in Physics](https://mlsanity.com/paper/2103.13496/) (2021)
- [Automatic Derivation Of Formulas Using Reforcement Learning](https://mlsanity.com/paper/1808.04946/) (2018)

## Built on by (most cited first, 3 of 3)

- [RKLD: Reverse KL-Divergence-based Knowledge Distillation for Unlearning Personal Information in Large Language Models](https://mlsanity.com/paper/2406.01983/) (2024)
- [A Symbolic Framework for Evaluating Mathematical Reasoning and Generalisation with Transformers](https://mlsanity.com/paper/2305.12563/) (2023)
- [A Survey in Mathematical Language Processing](https://mlsanity.com/paper/2205.15231/) (2022)

## Abstract (arXiv)

In order for language models to aid physics research, they must first encode representations of mathematical and natural language discourse which lead to coherent explanations, with correct ordering and relevance of statements. We present a collection of datasets developed to evaluate the performance of language models in this regard, which measure capabilities with respect to sentence ordering, position, section prediction, and discourse coherence. Analysis of the data reveals equations and sub-disciplines which are most common in physics discourse, as well as the sentence-level frequency of equations and expressions. We present baselines that demonstrate how contemporary language models are challenged by coherence related tasks in physics, even when trained on mathematical natural language objectives.