Learning to Look at the Other Side: A Semantic Probing Study of Word Embeddings in LLMs with Enabled Bidirectional Attention
Not on the map. Why?
Abstract (arXiv)
Autoregressive Large Language Models (LLMs) demonstrate exceptional performance in language understanding and generation. However, their application in text embedding tasks has been relatively slow, along with the analysis of their semantic representation in probing tasks, due to the constraints of the unidirectional attention mechanism. This paper aims to explore whether such constraints can be overcome by enabling bidirectional attention in LLMs. We tested different variants of the Llama architecture through additional training steps, progressively enabling bidirectional attention and unsupervised/supervised contrastive learning.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 29 of its 59 references that are in the map. First posted on arXiv 2 Oct 2025.
Builds on
29 of 59 references.
References · 29
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
- 2018Deep contextualized word representationscited by 2,290
- 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
- 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
- 2019Designing and Interpreting Probes with Control Taskscited by 640
- 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
- 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
- 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
- 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
17 more
- 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
- 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
- 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
- 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
- 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
- 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
- 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 156
- 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
- 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 45
- 2024Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semanticscited by 44
- 2024Probing the Emergence of Cross-lingual Alignment during LLM Trainingcited by 39
- 2020Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding wordscited by 28
- 2016Problems With Evaluation of Word Embeddings Using Word Similarity Taskscited by 25
- 2025MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilitiescited by 5
- 2018A Neurobiologically Motivated Analysis of Distributional Semantic Modelscited by 3
- 2022Contextualized Sensorimotor Norms: multi-dimensional measures of sensorimotor strength for ambiguous English words, in contextcited by 3
- 2024Semantics or spelling? Probing contextual word embeddings with orthographic noisecited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.