# HUBERT Untangles BERT to Improve Transfer across NLP Tasks

arXiv: https://arxiv.org/abs/1910.12647 · HTML: https://mlsanity.com/paper/1910.12647/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Oct 2019.

## Built on by (most cited first, 4 of 4)

- [Disentangling Representations of Text by Masking Transformers](https://mlsanity.com/paper/2104.07155/) (2021)
- [Probabilistic Analogical Mapping with Semantic Relation Networks](https://mlsanity.com/paper/2103.16704/) (2021)
- [From `Snippet-lects' to Doculects and Dialects: Leveraging Neural Representations of Speech for Placing Audio Signals in a Language Landscape](https://mlsanity.com/paper/2305.18602/) (2023)
- [Know Yourself and Know Your Neighbour: A Syntactically Informed Self-Supervised Compositional Sentence Representation Learning Framework using a Recursive Hypernetwork](https://mlsanity.com/paper/2610.16547/) (2026)

## Abstract (arXiv)

We introduce HUBERT which combines the structured-representational power of Tensor-Product Representations (TPRs) and BERT, a pre-trained bidirectional Transformer language model. We show that there is shared structure between different NLP datasets that HUBERT, but not BERT, is able to learn and leverage. We validate the effectiveness of our model on the GLUE benchmark and HANS dataset. Our experiment results show that untangling data-specific semantics from general language structure is key for better transfer among NLP tasks.