mlsanityData to 14 Oct 2026

The Emergence of Chunking Structures with Hierarchical RNN

arXiv 2309.04919 ↗First posted 10 Sep 2023cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

In Natural Language Processing (NLP), predicting linguistic structures, such as parsing and chunking, has mostly relied on manual annotations of syntactic structures. This paper introduces an unsupervised approach to chunking, a syntactic task that involves grouping words in a non-hierarchical manner. We present a Hierarchical Recurrent Neural Network (HRNN) designed to model word-to-chunk and chunk-to-sentence compositions. Our approach involves a two-stage training process: pretraining with an unsupervised parser and finetuning on downstream NLP tasks. Experiments on multiple datasets reveal a notable improvement of unsupervised chunking performance in both pretraining and finetuning stages. Interestingly, we observe that the emergence of the chunking structure is transient during the neural model's downstream-task training. This study contributes to the advancement of unsupervised syntactic structure discovery and opens avenues for further research in linguistic theory.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 34 of its 84 references that are in the map. First posted on arXiv 10 Sep 2023.

Builds on

34 of 84 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  7. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  9. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  10. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 396
  11. 2016Rationalizing Neural Predictionscited by 281
  12. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215
22 more
  1. 2019Fine-tune BERT for Extractive Summarizationcited by 122
  2. 2016Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervisioncited by 103
  3. 2018Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networkscited by 101
  4. 2019Compound Probabilistic Context-Free Grammars for Grammar Inductioncited by 76
  5. 2019Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoderscited by 60
  6. 2017Neural Language Modeling by Jointly Learning Syntax and Lexiconcited by 57
  7. 2018A Survey on Open Information Extractioncited by 51
  8. 2017Learning to Compose Task-Specific Tree Structurescited by 48
  9. 2020Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Inductioncited by 47
  10. 2015Transition-Based Dependency Parsing with Stack Long Short-Term Memorycited by 45
  11. 2020English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Toocited by 39
  12. 2019Unsupervised Paraphrasing by Simulated Annealingcited by 37
  13. 2020Unsupervised Text Generation by Learning from Searchcited by 36
  14. 2020StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modelingcited by 33
  15. 2016Learning to Compose Words into Sentences with Reinforcement Learningcited by 30
  16. 2019Decomposable Neural Paraphrase Generationcited by 30
  17. 2020On the Role of Supervision in Unsupervised Constituency Parsingcited by 18
  18. 2022Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarizationcited by 12
  19. 2021Weakly Supervised Explainable Phrasal Reasoning with Neural Fuzzy Logiccited by 10
  20. 2016Coupling Distributed and Symbolic Execution for Natural Language Queriescited by 9
  21. 2019An Imitation Learning Approach to Unsupervised Parsingcited by 7
  22. 2018JUMPER: Learning When to Make Classification Decisions in Readingcited by 6

Built on by

Cited by year

Citing papers per year: 2023: 2.

2 papers.

  1. 2023Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Modelscited by 7
  2. 2023Ensemble Distillation for Unsupervised Constituency Parsingcited by 6