mlsanityData to 7 Oct 2026

Can Language Representation Models Think in Bets?

arXiv 2210.07519 ↗First posted 14 Oct 2022cs.CL cs.AIcited by 3

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In recent years, transformer-based language representation models (LRMs) have achieved state-of-the-art results on difficult natural language understanding problems, such as question answering and text summarization. As these models are integrated into real-world applications, evaluating their ability to make rational decisions is an important research agenda, with practical ramifications. This article investigates LRMs' rational decision-making ability through a carefully designed set of decision-making benchmarks and experiments. Inspired by classic work in cognitive science, we model the decision-making problem as a bet. We then investigate an LRM's ability to choose outcomes that have optimal, or at minimum, positive expected gain. Through a robust body of experiments on four established LRMs, we show that a model is only able to `think in bets' if it is first fine-tuned on bet questions with an identical structure. Modifying the bet question's structure, while still retaining its fundamental characteristics, decreases an LRM's performance by more than 25%, on average, although absolute performance remains well above random. LRMs are also found to be more rational when selecting outcomes with non-negative expected gain, rather than optimal or strictly positive expected gain. Our results suggest that LRMs could potentially be applied to tasks that rely on cognitive decision-making skills, but that more research is necessary before they can robustly make rational decisions.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 71 of its 113 references that are in the map. First posted on arXiv 14 Oct 2022.

Builds on

71 of 113 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Understanding Prior Bias and Choice Paralysis in Transformer-based Language Representation Models through Four Experimental Probes3 Oct

Other references · 70

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
38 more
  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  2. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  9. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  12. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  13. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  14. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  15. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  16. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  17. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  18. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  19. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  20. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  21. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  22. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  23. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  24. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  25. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  26. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  27. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  28. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  29. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  30. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 303
  31. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  32. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  33. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  34. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 157
  35. 2020Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Modelscited by 110
  36. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  37. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  38. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1.

3 papers.

  1. 2023A Pilot Evaluation of ChatGPT and DALL-E 2 on Decision Making and Spatial Reasoningcited by 1
  2. 2023Named Entity Resolution in Personal Knowledge Graphscited by 1
  3. 2024Lifelong learning challenges in the era of artificial intelligence: a computational thinking perspectivecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Benchmarking LLM Competence on Logical Inference over Probability Operators
  2. 2025Reasoning Capabilities and Invariability of Large Language Models
  3. 2025Answer-Centric or Reasoning-Driven? Uncovering the Latent Memory Anchor in LLMs
  4. 2026Mind the (DH) Gap! A Contrast in Risky Choices Between Reasoning and Conversational LLMs
  5. 2025On the Failure of Latent State Persistence in Large Language Models
  6. 2025Geometry of Decision Making in Language Models
  7. 2024Calibrating Reasoning in Language Models with Internal Consistency
  8. 2024Confidence in the Reasoning of Large Language Models