Understanding and Being Understood: User Strategies for Identifying and Recovering From Mistranslations in Machine Translation-Mediated Chat
Abstract
Note: Will be adding at least one more reviewer.
Machine translation (MT) is now widely and freely available, and has the potential to greatly improve interlingual communication. However, it can be difficult for users to detect and recover from mistranslations because limited language skills hinder comprehension of either the inputs or the outpus. In order to use MT reliably and safely, end users must be able to assess the quality of system outputs and determine how much they can rely on them to guide their decisions and actions. In this work we collected 19 MT-mediated high-stakes, role-play conversations and in-depth interviews to understand how users identify and recover from translation errors. Participants communicated using four language pairs: English, and one of Spanish, Farsi, Igbo, or Tagalog. We also collected human annotations of translation quality and conducted a mixed-method analysis to understand user challenges, strategies for recovery, and the kinds of translation errors that proved more or less difficult for users to overcome. We found that users broadly lacked relevant and helpful information to guide their assessments of translation quality. Instances where a user erroneously thought they had understood a translation correctly, were rare but held the potential for drastic consequences in the real world. Finally, inaccurate and disfluent translations had social consequences for the participants, because it was difficult to discern when disfluent message was reflective of the other person’s intentions, or an artifact of imperfect MT. We draw on theories of grounding and repair in communication to contextualize these findings, and propose design implications for HCI researchers, MT researchers, and opportunities for greater coherence and collaboration between these efforts.
Machine translation (MT) is now widely and freely available, and has the potential to greatly improve interlingual communication. However, it can be difficult for users to detect and recover from mistranslations because limited language skills hinder comprehension of either the inputs or the outpus. In order to use MT reliably and safely, end users must be able to assess the quality of system outputs and determine how much they can rely on them to guide their decisions and actions. In this work we collected 19 MT-mediated high-stakes, role-play conversations and in-depth interviews to understand how users identify and recover from translation errors. Participants communicated using four language pairs: English, and one of Spanish, Farsi, Igbo, or Tagalog. We also collected human annotations of translation quality and conducted a mixed-method analysis to understand user challenges, strategies for recovery, and the kinds of translation errors that proved more or less difficult for users to overcome. We found that users broadly lacked relevant and helpful information to guide their assessments of translation quality. Instances where a user erroneously thought they had understood a translation correctly, were rare but held the potential for drastic consequences in the real world. Finally, inaccurate and disfluent translations had social consequences for the participants, because it was difficult to discern when disfluent message was reflective of the other person’s intentions, or an artifact of imperfect MT. We draw on theories of grounding and repair in communication to contextualize these findings, and propose design implications for HCI researchers, MT researchers, and opportunities for greater coherence and collaboration between these efforts.
Research Areas
