# On Improving Faithfulness of Podcasts from Documents

arXiv: https://arxiv.org/abs/2607.21961 · HTML: https://mlsanity.com/paper/2607.21961/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 24 Jul 2026.

## Builds on (most cited first, 23 of 23)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [Trusting Your Evidence: Hallucinate Less with Context-aware Decoding](https://mlsanity.com/paper/2305.14739/) (2023)
- [Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering](https://mlsanity.com/paper/2307.16877/) (2023)
- [FaithDial: A Faithful Benchmark for Information-Seeking Dialogue](https://mlsanity.com/paper/2204.10757/) (2022)
- [Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding](https://mlsanity.com/paper/2104.08455/) (2021)
- [FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"](https://mlsanity.com/paper/2410.03727/) (2024)
- [RHO (ρ): Reducing Hallucination in Open-domain Dialogues with Knowledge Grounding](https://mlsanity.com/paper/2212.01588/) (2022)
- [Parameters vs. Context: Fine-Grained Control of Knowledge Reliance in Language Models](https://mlsanity.com/paper/2503.15888/) (2025)
- [The Dangers of trusting Stochastic Parrots: Faithfulness and Trust in Open-domain Conversational Question Answering](https://mlsanity.com/paper/2305.16519/) (2023)
- [Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model](https://mlsanity.com/paper/2310.11648/) (2023)
- [SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidence](https://mlsanity.com/paper/2502.08767/) (2025)
- [MoonCast: High-Quality Zero-Shot Podcast Generation](https://mlsanity.com/paper/2503.14345/) (2025)
- [NexusSum: Hierarchical LLM Agents for Long-Form Narrative Summarization](https://mlsanity.com/paper/2505.24575/) (2025)
- [Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language Models](https://mlsanity.com/paper/2501.01059/) (2025)
- [PodAgent: A Comprehensive Framework for Podcast Generation](https://mlsanity.com/paper/2503.00455/) (2025)
- [Beyond Prompting: An Efficient Embedding Framework for Open-Domain Question Answering](https://mlsanity.com/paper/2503.01606/) (2025)
- [PodBench: A Comprehensive Benchmark for Instruction-Aware Audio-Oriented Podcast Script Generation](https://mlsanity.com/paper/2601.14903/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used to generate long-form conversational content such as podcasts from textual sources. While these systems produce fluent and engaging narratives, they often introduce ungrounded information. In this work, we present the first systematic study of faithfulness in document-grounded podcast generation, where grounding must be maintained across conversational turns in long-form, multi-speaker transcripts. We construct a dataset of over 1500 documents spanning five domains and generate podcast transcripts using multiple LLMs. We introduce a turn-level LLM-as-a-judge framework for evaluating whether conversational turns are supported by the source document, and validate its reliability through human studies. Our analysis shows that even state-of-the-art models, including GPT-4o, frequently generate ungrounded content. To mitigate this issue, we propose catch-n-repair, a model-agnostic framework that detects and rewrites unfaithful conversational turns while preserving conversational flow. Experiments demonstrate consistent improvements in faithfulness across both in-domain and out-of-domain settings.