mlsanityData to 7 Oct 2026

Angler: Helping Machine Translation Practitioners Prioritize Model Improvements

arXiv 2304.05967 ↗First posted 12 Apr 2023cs.HC cs.AI cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Machine learning (ML) models can fail in unexpected ways in the real world, but not all model failures are equal. With finite time and resources, ML practitioners are forced to prioritize their model debugging and improvement efforts. Through interviews with 13 ML practitioners at Apple, we found that practitioners construct small targeted test sets to estimate an error's nature, scope, and impact on users. We built on this insight in a case study with machine translation models, and developed Angler, an interactive visual analytics tool to help practitioners prioritize model improvements. In a user study with 7 machine translation experts, we used Angler to understand prioritization practices when the input space is infinite, and obtaining reliable signals of model quality is expensive. Our study revealed that participants could form more interesting and user-focused hypotheses for prioritization by analyzing quantitative summary statistics and qualitatively assessing data by reading sentences.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 61 of its 144 references that are in the map. First posted on arXiv 12 Apr 2023.

Builds on

61 of 144 references.

References · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  5. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  8. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  9. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  10. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  11. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  12. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 779
38 more
  1. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  2. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  3. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  4. 2018Gender Bias in Coreference Resolutioncited by 438
  5. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  6. 2018Improving fairness in machine learning systems: What do industry practitioners need?cited by 262
  7. 2019Evaluating Gender Bias in Machine Translationcited by 247
  8. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  9. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 215
  10. 2018Stress Test Evaluation for Natural Language Inferencecited by 211
  11. 2019The What-If Tool: Interactive Probing of Machine Learning Modelscited by 207
  12. 2021Data Augmentation Can Improve Robustnesscited by 193
  13. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 173
  14. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  15. 2020Characterising Bias in Compressed Modelscited by 162
  16. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  17. 2019Does Object Recognition Work for Everyone?cited by 149
  18. 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 127
  19. 2018Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontierscited by 117
  20. 2019Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Makingcited by 109
  21. 2019FairVis: Visual Analytics for Discovering Intersectional Bias in Machine Learningcited by 93
  22. 2020Saving Face: Investigating the Ethical Concerns of Facial Recognition Auditingcited by 89
  23. 2018Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Modelscited by 69
  24. 2019explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learningcited by 69
  25. 2021The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Modelscited by 63
  26. 2021Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffscited by 48
  27. 2021Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviorscited by 47
  28. 2019compare-mt: A Tool for Holistic Comparison of Language Generation Systemscited by 43
  29. 2017A Challenge Set Approach to Evaluating Machine Translationcited by 37
  30. 2021Discovering and Validating AI Errors With Crowdsourced Failure Reportscited by 28
  31. 2022Understanding Machine Learning Practitioners' Data Documentation Perceptions, Needs, Challenges, and Desideratacited by 28
  32. 2021UDIS: Unsupervised Discovery of Bias in Deep Visual Recognition Modelscited by 23
  33. 2022Unit Testing for Concepts in Neural Networkscited by 23
  34. 2020PipelineProfiler: A Visual Analytics Tool for the Exploration of AutoML Pipelinescited by 19
  35. 2022A Unified Understanding of Deep NLP Models for Text Classificationcited by 18
  36. 2021GENder-IT: An Annotated English-Italian Parallel Challenge Set for Cross-Linguistic Natural Gender Phenomenacited by 16
  37. 2019Automatically Extracting Challenge Sets for Non local Phenomena in Neural Machine Translationcited by 15
  38. 2020Unmet Needs and Opportunities for Mobile Translation AIcited by 14

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 4; 2026: 1.

11 papers.

  1. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 73
  2. 2023WizMap: Scalable Interactive Visualization for Exploring Large Machine Learning Embeddingscited by 28
  3. 2024Farsight: Fostering Responsible AI Awareness During AI Application Prototypingcited by 28
  4. 2024Abstraction Alignment: Comparing Model-Learned and Human-Encoded Conceptual Relationshipscited by 7
  5. 2024AttributionScanner: A Visual Analytics System for Model Validation with Metadata-Free Slice Findingcited by 6
  6. 2025An Interdisciplinary Approach to Human-Centered Machine Translationcited by 6
  7. 2024Compress and Compare: Interactively Evaluating Efficiency and Behavior Across ML Model Compression Experimentscited by 4
  8. 2025RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generationcited by 4
  9. 2025Evalet: Evaluating Large Language Models through Functional Fragmentationcited by 3
  10. 2025Toward Machine Translation Literacy: How Lay Users Perceive and Rely on Imperfect Translationscited by 1
  11. 2026Human-Data Interaction, Exploration, and Visualization in the AI Era: Challenges and Opportunitiescited by 0