# When BERT Meets Quantum Temporal Convolution Learning for Text Classification in Heterogeneous Computing

arXiv: https://arxiv.org/abs/2203.03550 · HTML: https://mlsanity.com/paper/2203.03550/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 38 references that are in the map.
- First posted on arXiv 17 Feb 2022.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Supervised learning with quantum enhanced feature spaces](https://mlsanity.com/paper/1804.11326/) (2018)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Power of data in quantum machine learning](https://mlsanity.com/paper/2011.01938/) (2020)
- [A rigorous and robust quantum speed-up in supervised machine learning](https://mlsanity.com/paper/2010.02174/) (2020)
- [Variational Quantum Circuits for Deep Reinforcement Learning](https://mlsanity.com/paper/1907.00397/) (2019)
- [Federated Learning for Keyword Spotting](https://mlsanity.com/paper/1810.05512/) (2018)
- [Federated Quantum Machine Learning](https://mlsanity.com/paper/2103.12010/) (2021)
- [QNLP in Practice: Running Compositional Models of Meaning on a Quantum Computer](https://mlsanity.com/paper/2102.12846/) (2021)
- [Decentralizing Feature Extraction with Quantum Convolutional Neural Network for Automatic Speech Recognition](https://mlsanity.com/paper/2010.13309/) (2020)
- [Foundations for Near-Term Quantum Natural Language Processing](https://mlsanity.com/paper/2012.03755/) (2020)
- [Quantum Natural Language Processing on Near-Term Quantum Computers](https://mlsanity.com/paper/2005.04147/) (2020)
- [Federated Learning of N-gram Language Models](https://mlsanity.com/paper/1910.03432/) (2019)
- [QTN-VQC: An End-to-End Learning framework for Quantum Neural Networks](https://mlsanity.com/paper/2110.03861/) (2021)
- [Multi-task Language Modeling for Improving Speech Recognition of Rare Words](https://mlsanity.com/paper/2011.11715/) (2020)
- [PATE-AAE: Incorporating Adversarial Autoencoder into Private Aggregation of Teacher Ensembles for Spoken Command Classification](https://mlsanity.com/paper/2104.01271/) (2021)

## Built on by (most cited first, 30 of 40)

- [Quantum Self-Attention Neural Networks for Text Classification](https://mlsanity.com/paper/2205.05625/) (2022)
- [Theoretical Error Performance Analysis for Variational Quantum Circuit Based Functional Regression](https://mlsanity.com/paper/2206.04804/) (2022)
- [PQLM -- Multilingual Decentralized Portable Quantum Language Model for Privacy Protection](https://mlsanity.com/paper/2210.03221/) (2022)
- [Quantum deep recurrent reinforcement learning](https://mlsanity.com/paper/2210.14876/) (2022)
- [Learning to Program Variational Quantum Circuits with Fast Weights](https://mlsanity.com/paper/2402.17760/) (2024)
- [Asynchronous training of quantum reinforcement learning](https://mlsanity.com/paper/2301.05096/) (2023)
- [Towards Quantum Federated Learning](https://mlsanity.com/paper/2306.09912/) (2023)
- [Certified Robustness of Quantum Classifiers against Adversarial Examples through Quantum Noise](https://mlsanity.com/paper/2211.00887/) (2022)
- [Federated Quantum Machine Learning with Differential Privacy](https://mlsanity.com/paper/2310.06973/) (2023)
- [Learning to Measure Quantum Neural Networks](https://mlsanity.com/paper/2501.05663/) (2025)
- [Federated Quantum Natural Gradient Descent for Quantum Federated Learning](https://mlsanity.com/paper/2209.00564/) (2022)
- [Reservoir Computing via Quantum Recurrent Neural Networks](https://mlsanity.com/paper/2211.02612/) (2022)
- [Parallel Hybrid Networks: an interplay between quantum and classical neural networks](https://mlsanity.com/paper/2303.03227/) (2023)
- [Optimizing Quantum Federated Learning Based on Federated Quantum Natural Gradient Descent](https://mlsanity.com/paper/2303.08116/) (2023)
- [Efficient quantum recurrent reinforcement learning via quantum reservoir computing](https://mlsanity.com/paper/2309.07339/) (2023)
- [Differentiable Quantum Architecture Search in Asynchronous Quantum Reinforcement Learning](https://mlsanity.com/paper/2407.18202/) (2024)
- [Quantum-Train-Based Distributed Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2412.08845/) (2024)
- [Toward Large-Scale Distributed Quantum Long Short-Term Memory with Modular Quantum Computers](https://mlsanity.com/paper/2503.14088/) (2025)
- [Quantum Machine Learning Architecture Search via Deep Reinforcement Learning](https://mlsanity.com/paper/2407.20147/) (2024)
- [Evolutionary Optimization for Designing Variational Quantum Circuits with High Model Capacity](https://mlsanity.com/paper/2412.12484/) (2024)
- [Quantum Federated Learning With Quantum Networks](https://mlsanity.com/paper/2310.15084/) (2023)
- [An Introduction to Quantum Reinforcement Learning (QRL)](https://mlsanity.com/paper/2409.05846/) (2024)
- [Adaptive Non-local Observable on Quantum Neural Networks](https://mlsanity.com/paper/2504.13414/) (2025)
- [Differentiable Quantum Architecture Search in Quantum-Enhanced Neural Network Parameter Generation](https://mlsanity.com/paper/2505.09653/) (2025)
- [Learning to Program Quantum Measurements for Machine Learning](https://mlsanity.com/paper/2505.13525/) (2025)
- [A hybrid quantum-classical fusion neural network to improve protein-ligand binding affinity predictions for drug discovery](https://mlsanity.com/paper/2309.03919/) (2023)
- [Quantum Natural Language Processing: A Comprehensive Review of Models, Methods, and Applications](https://mlsanity.com/paper/2504.09909/) (2025)
- [Transfer Learning Analysis of Variational Quantum Circuits](https://mlsanity.com/paper/2501.01507/) (2025)
- [Introduction to Quantum Machine Learning and Quantum Architecture Search](https://mlsanity.com/paper/2504.16131/) (2025)
- [Quantum deep Q learning with distributed prioritized experience replay](https://mlsanity.com/paper/2304.09648/) (2023)

## Abstract (arXiv)

The rapid development of quantum computing has demonstrated many unique characteristics of quantum advantages, such as richer feature representation and more secured protection on model parameters. This work proposes a vertical federated learning architecture based on variational quantum circuits to demonstrate the competitive performance of a quantum-enhanced pre-trained BERT model for text classification. In particular, our proposed hybrid classical-quantum model consists of a novel random quantum temporal convolution (QTC) learning framework replacing some layers in the BERT-based decoder. Our experiments on intent classification show that our proposed BERT-QTC model attains competitive experimental results in the Snips and ATIS spoken language datasets. Particularly, the BERT-QTC boosts the performance of the existing quantum circuit-based language model in two text classification datasets by 1.57% and 1.52% relative improvements. Furthermore, BERT-QTC can be feasibly deployed on both existing commercial-accessible quantum computation hardware and CPU-based interface for ensuring data isolation.