# When Readability and Source Retention Diverge: An Evaluability Gap in AI Translation

arXiv: https://arxiv.org/abs/2608.19083 · HTML: https://mlsanity.com/paper/2608.19083/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 60 references that are in the map.
- First posted on arXiv 19 Aug 2026.

## Builds on (most cited first, 7 of 7)

- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [What Large Language Models Know and What People Think They Know](https://mlsanity.com/paper/2401.13835/) (2024)
- [Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation](https://mlsanity.com/paper/1808.07048/) (2018)
- [Generalization Bias in Large Language Model Summarization of Scientific Research](https://mlsanity.com/paper/2504.00025/) (2025)
- [The Dangers of trusting Stochastic Parrots: Faithfulness and Trust in Open-domain Conversational Question Answering](https://mlsanity.com/paper/2305.16519/) (2023)
- [Fluency Over Adequacy: A Pilot Study in Measuring User Trust in Imperfect MT](https://mlsanity.com/paper/1802.06041/) (2018)
- [What's the Difference Between Professional Human and Machine Translation? A Blind Multi-language Study on Domain-specific MT](https://mlsanity.com/paper/2006.04781/) (2020)

## Abstract (arXiv)

Readable AI output can leave an evaluability gap: even when the source is shown, an overall-quality judgment may not reflect what an output preserves. We investigated how source-text condition and output rendering relate to perceived translation quality, and how output and system appraisals relate to trust and stated disclosure willingness in a plain-text interface. A focal 2 * 2 comparison (N=306) using TransLingo examined simple generated narratives and complex literary-philosophical prose alongside LLM-generated readability-oriented outputs and researcher-revised fidelity-oriented outputs. A descriptive stimulus audit indicated greater source retention in fidelity-oriented outputs in both source-text conditions. Factorial analyses showed a significant rendering-by-source-text-condition interaction in perceived quality. Participants rated fidelity-oriented outputs higher than readability-oriented outputs for the simple narratives, whereas no reliable rendering difference emerged for the complex prose. A corresponding source-condition-dependent pattern was observed for perceived intelligence, agency-oriented anthropomorphic attribution, and task-performance trust. A separate theory-ordered appraisal-structure SEM characterized concurrent associations among perceived quality, perceived intelligence, agency-oriented anthropomorphic attribution, task-performance trust, and stated disclosure willingness across six domains, with task-performance trust as the proximal correlate of stated willingness. The observed rating pattern distinguishes source access from source evaluability: for the complex stimuli, displaying the source did not ensure that one overall-quality rating reflected differences in retained content. It also separates support for evaluating translation output from data-handling support for decisions about what personal text to entrust to a system.