# Exploring Graph Representations of Logical Forms for Language Modeling

arXiv: https://arxiv.org/abs/2505.14523 · HTML: https://mlsanity.com/paper/2505.14523/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 63 references that are in the map.
- First posted on arXiv 20 May 2025.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [Representing Long-Range Context for Graph Neural Networks with Global Attention](https://mlsanity.com/paper/2201.08821/) (2022)
- [Should You Mask 15% in Masked Language Modeling?](https://mlsanity.com/paper/2202.08005/) (2022)
- [Semantics-aware BERT for Language Understanding](https://mlsanity.com/paper/1909.02209/) (2019)
- [On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model](https://mlsanity.com/paper/2204.13509/) (2022)
- [Training Trajectories of Language Models Across Scales](https://mlsanity.com/paper/2212.09803/) (2022)
- [Directed Graph Convolutional Network](https://mlsanity.com/paper/2004.13970/) (2020)
- [GraphGen: A Scalable Approach to Domain-agnostic Labeled Graph Generation](https://mlsanity.com/paper/2001.08184/) (2020)
- [Graph-to-Sequence Learning using Gated Graph Neural Networks](https://mlsanity.com/paper/1806.09835/) (2018)
- [Do Syntax Trees Help Pre-trained Transformers Extract Information?](https://mlsanity.com/paper/2008.09084/) (2020)
- [Graph2Seq: Graph to Sequence Learning with Attention-based Neural Networks](https://mlsanity.com/paper/1804.00823/) (2018)
- [Overfitting Mechanism and Avoidance in Deep Neural Networks](https://mlsanity.com/paper/1901.06566/) (2019)
- [Syntax-Enhanced Pre-trained Model](https://mlsanity.com/paper/2012.14116/) (2020)
- [Infusing Finetuning with Semantic Dependencies](https://mlsanity.com/paper/2012.05395/) (2020)
- [Edge-based sequential graph generation with recurrent neural networks](https://mlsanity.com/paper/2002.00102/) (2020)
- [Extracting Temporal Event Relation with Syntax-guided Graph Transformer](https://mlsanity.com/paper/2104.09570/) (2021)
- [Lexicosyntactic Inference in Neural Models](https://mlsanity.com/paper/1808.06232/) (2018)
- [Robust Incremental Neural Semantic Graph Parsing](https://mlsanity.com/paper/1704.07092/) (2017)

## Abstract (arXiv)

We make the case for language models over logical forms (LFLMs), arguing that such models are more data-efficient than their textual counterparts. To that end, we introduce the Graph-based Formal-Logical Distributional Semantics (GFoLDS) prototype, a pretrained LM over graph representations of logical forms, as a proof-of-concept of LFLMs. Using GFoLDS, we present strong experimental evidence that LFLMs can leverage the built-in, basic linguistic knowledge inherent in such models to immediately begin learning more complex patterns. On downstream tasks, we show that GFoLDS vastly outperforms textual, transformer LMs (BERT) pretrained on the same data, indicating that LFLMs can learn with substantially less data than models over plain text. Furthermore, we show that the performance of this model is likely to scale with additional parameters and pretraining data, suggesting the viability of LFLMs in real-world applications.