# Pre-training Text-to-Text Transformers for Concept-centric Common Sense

arXiv: https://arxiv.org/abs/2011.07956 · HTML: https://mlsanity.com/paper/2011.07956/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2020.

## Built on by (most cited first, 30 of 42)

- [ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://mlsanity.com/paper/2107.02137/) (2021)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](https://mlsanity.com/paper/2210.13312/) (2022)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [Controlled Text Generation with Natural Language Instructions](https://mlsanity.com/paper/2304.14293/) (2023)
- [ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification](https://mlsanity.com/paper/2203.02225/) (2022)
- [EventBERT: A Pre-Trained Model for Event Correlation Reasoning](https://mlsanity.com/paper/2110.06533/) (2021)
- [Dict-BERT: Enhancing Language Model Pre-training with Dictionary](https://mlsanity.com/paper/2110.06490/) (2021)
- [Visualize Before You Write: Imagination-Guided Open-Ended Text Generation](https://mlsanity.com/paper/2210.03765/) (2022)
- [A review of ensemble learning and data augmentation models for class imbalanced problems: combination, implementation and evaluation](https://mlsanity.com/paper/2304.02858/) (2023)
- [Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training](https://mlsanity.com/paper/2206.00621/) (2022)
- [KILM: Knowledge Injection into Encoder-Decoder Language Models](https://mlsanity.com/paper/2302.09170/) (2023)
- [Retrieval Enhanced Model for Commonsense Generation](https://mlsanity.com/paper/2105.11174/) (2021)
- [Write and Paint: Generative Vision-Language Models are Unified Modal Learners](https://mlsanity.com/paper/2206.07699/) (2022)
- [Go Back in Time: Generating Flashbacks in Stories with Event Temporal Prompts](https://mlsanity.com/paper/2205.01898/) (2022)
- [Retrieval Augmentation for Commonsense Reasoning: A Unified Approach](https://mlsanity.com/paper/2210.12887/) (2022)
- [Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting](https://mlsanity.com/paper/2101.00416/) (2021)
- [Knowledge Infused Decoding](https://mlsanity.com/paper/2204.03084/) (2022)
- [VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models](https://mlsanity.com/paper/2205.15237/) (2022)
- [Multiview Contextual Commonsense Inference: A New Dataset and Task](https://mlsanity.com/paper/2210.02890/) (2022)
- [Structure Inducing Pre-Training](https://mlsanity.com/paper/2103.10334/) (2021)
- [Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-modal Knowledge Transfer](https://mlsanity.com/paper/2203.07519/) (2022)
- [Clues Before Answers: Generation-Enhanced Multiple-Choice QA](https://mlsanity.com/paper/2205.00274/) (2022)
- [Modelling Commonsense Properties using Pre-Trained Bi-Encoders](https://mlsanity.com/paper/2210.02771/) (2022)
- [R2D2: Robust Data-to-Text with Replacement Detection](https://mlsanity.com/paper/2205.12467/) (2022)
- [Controlled Language Generation for Language Learning Items](https://mlsanity.com/paper/2211.15731/) (2022)
- [KGR^4: Retrieval, Retrospect, Refine and Rethink for Commonsense Generation](https://mlsanity.com/paper/2112.08266/) (2021)
- [ArK: Augmented Reality with Knowledge Interactive Emergent Ability](https://mlsanity.com/paper/2305.00970/) (2023)
- [Commonsense Knowledge Transfer for Pre-trained Language Models](https://mlsanity.com/paper/2306.02388/) (2023)
- [Improving Text Auto-Completion with Next Phrase Prediction](https://mlsanity.com/paper/2109.07067/) (2021)

## Abstract (arXiv)

Pre-trained language models (PTLM) have achieved impressive results in a range of natural language understanding (NLU) and generation (NLG) tasks. However, current pre-training objectives such as masked token prediction (for BERT-style PTLMs) and masked span infilling (for T5-style PTLMs) do not explicitly model the relational commonsense knowledge about everyday concepts, which is crucial to many downstream tasks that need common sense to understand or generate. To augment PTLMs with concept-centric commonsense knowledge, in this paper, we propose both generative and contrastive objectives for learning common sense from the text, and use them as intermediate self-supervised learning tasks for incrementally pre-training PTLMs (before task-specific fine-tuning on downstream datasets). Furthermore, we develop a joint pre-training framework to unify generative and contrastive objectives so that they can mutually reinforce each other. Extensive experimental results show that our method, concept-aware language model (CALM), can pack more commonsense knowledge into the parameters of a pre-trained text-to-text transformer without relying on external knowledge graphs, yielding better performance on both NLU and NLG tasks. We show that while only incrementally pre-trained on a relatively small corpus for a few steps, CALM outperforms baseline methods by a consistent margin and even comparable with some larger PTLMs, which suggests that CALM can serve as a general, plug-and-play method for improving the commonsense reasoning ability of a PTLM.