# Enhancing Language Representation with Constructional Information for Natural Language Understanding

arXiv: https://arxiv.org/abs/2306.02819 · HTML: https://mlsanity.com/paper/2306.02819/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 64 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Hypergraph Neural Networks](https://mlsanity.com/paper/1809.09401/) (2018)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [Relational Graph Attention Network for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2004.12362/) (2020)
- [Be More with Less: Hypergraph Attention Networks for Inductive Text Classification](https://mlsanity.com/paper/2011.00387/) (2020)
- [The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlative](https://mlsanity.com/paper/2210.13181/) (2022)
- [CxGBERT: BERT meets Construction Grammar](https://mlsanity.com/paper/2011.04134/) (2020)
- [Construction Grammar Provides Unique Insight into Neural Language Models](https://mlsanity.com/paper/2302.02178/) (2023)
- [SemEval-2020 Task 5: Counterfactual Recognition](https://mlsanity.com/paper/2008.00563/) (2020)
- [Frequency vs. Association for Constraint Selection in Usage-Based Construction Grammar](https://mlsanity.com/paper/1904.05529/) (2019)
- [Exposure and Emergence in Usage-Based Grammar: Computational Experiments in 35 Languages](https://mlsanity.com/paper/2211.14160/) (2022)
- [Stability of Syntactic Dialect Classification Over Space and Time](https://mlsanity.com/paper/2209.04958/) (2022)
- [Exploring the Constructicon: Linguistic Analysis of a Computational CxG](https://mlsanity.com/paper/2301.12642/) (2023)
- [Processing Natural Language About Ongoing Actions](https://mlsanity.com/paper/1607.06875/) (2016)
- [Learn from Structural Scope: Improving Aspect-Level Sentiment Analysis with Hybrid Graph Convolutional Networks](https://mlsanity.com/paper/2204.12784/) (2022)

## Built on by (most cited first, 2 of 2)

- [The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies](https://mlsanity.com/paper/2407.19354/) (2024)
- [Analysis of Argument Structure Constructions in the Large Language Model BERT](https://mlsanity.com/paper/2408.04270/) (2024)

## Abstract (arXiv)

Natural language understanding (NLU) is an essential branch of natural language processing, which relies on representations generated by pre-trained language models (PLMs). However, PLMs primarily focus on acquiring lexico-semantic information, while they may be unable to adequately handle the meaning of constructions. To address this issue, we introduce construction grammar (CxG), which highlights the pairings of form and meaning, to enrich language representation. We adopt usage-based construction grammar as the basis of our work, which is highly compatible with statistical models such as PLMs. Then a HyCxG framework is proposed to enhance language representation through a three-stage solution. First, all constructions are extracted from sentences via a slot-constraints approach. As constructions can overlap with each other, bringing redundancy and imbalance, we formulate the conditional max coverage problem for selecting the discriminative constructions. Finally, we propose a relational hypergraph attention network to acquire representation from constructional information by capturing high-order word interactions among constructions. Extensive experiments demonstrate the superiority of the proposed model on a variety of NLU tasks.