mlsanityData to 7 Oct 2026

Aligning Brain Activity with Advanced Transformer Models: Exploring the Role of Punctuation in Semantic Processing

arXiv 2501.06278 ↗First posted 10 Jan 2025cs.CL cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

This research examines the congruence between neural activity and advanced transformer models, emphasizing the semantic significance of punctuation in text understanding. Utilizing an innovative approach originally proposed by Toneva and Wehbe, we evaluate four advanced transformer models RoBERTa, DistiliBERT, ALBERT, and ELECTRA against neural activity data. Our findings indicate that RoBERTa exhibits the closest alignment with neural activity, surpassing BERT in accuracy. Furthermore, we investigate the impact of punctuation removal on model performance and neural alignment, revealing that BERT's accuracy enhances in the absence of punctuation. This study contributes to the comprehension of how neural networks represent language and the influence of punctuation on semantic processing within the human brain.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 13 of its 45 references that are in the map. First posted on arXiv 10 Jan 2025.

Builds on

13 of 45 references.

References · 13

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  4. 2018Deep contextualized word representationscited by 2,285
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  7. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  8. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  9. 2019Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)cited by 128
  10. 2018On the Practical Computational Power of Finite Precision RNNs for Language Recognitioncited by 117
  11. 2018Finding Syntax in Human Encephalography with Beam Searchcited by 41
  12. 2017Recurrent Neural Networks as Weighted Language Recognizerscited by 30
1 more
  1. 2018Rational Recurrencescited by 18

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025ReasoningGuard: Safeguarding Large Reasoning Models with Inference-time Safety Aha Momentscited by 2