Cognitive Effort in Post-Editing of Machine Translation: evidence from eye movements, subjective ratings, and think-aloud protocols
DOI:
https://doi.org/10.14456/nvts.2016.16Keywords:
cognitive effort, eye tracking, machine translation, post-editing, subjective ratings, think-aloud protocolsAbstract
Machine translation (MT) post-editing is becoming increasingly popular as a way of coping with the growing demands placed on human translators. However, as a field, we know relatively little about the inner workings of this activity and its cognitive implications. This thesis presents an empirical study that examines MT post-editing by contrasting the cognitive effort required by this activity with a number of its key elements, including characteristics of the source text and of the MT output, post-editors’ individual traits, the different linguistic aspects of the task post-editors attend to, and the quality of the post-edited text, assessed by human evaluators in terms of fluency (linguistic quality) and adequacy (faithfulness to the source text). A combination of eye movements, subjective ratings and concurrent think-aloud protocols was adopted to investigate cognitive effort. Two post-editing tasks were conducted for data collection: one involving eye tracking and a self-report scale of cognitive effort, and another carried out by a different, but comparable, sample of participants, under a think-aloud condition.
A mixture of approaches including mixed-effects modelling and discussion of qualitative examples was exploited to analyse the data collected in the tasks. The results obtained indicate that a number of textual features can act as predictors of cognitive effort in post- editing, including the sentence-level type-token ratio of the source text and an automatic score used for MT evaluation. The level of quality of the raw MT output was found to be the single most important aspect influencing the expenditure of cognitive effort in the activity. The relationship between cognitive effort and post-editors’ individual traits was found to be a complex one, with significant links in this respect only appearing in the context of interactions between variables. A complex relationship was also found between editing behaviour and the quality of the post-edited text: the number of changes implemented was found to have a generally positive association with post-edited fluency, though cognitive effort was found to be negatively correlated with both the fluency and adequacy of the post- edited text, which seemed to suggest that automatic mental processes play a crucial role in the activity. Grammar and lexis were the linguistic aspects of the task most frequently attended to by post-editors. Both these aspects were found to be significantly related to the levels of cognitive effort expended, but no significant difference was found between grammar and lexis themselves.
To the knowledge of the author, this thesis represents the first attempt to date at combining eye-tracking data, subjective ratings, and think-aloud protocols in a study on post-editing. This combination showed the extent to which different methods converge in measuring similar constructs. Despite the criticisms received by the think-aloud method in previous research, results of this methodological triangulation revealed that think-aloud protocols correlate with eye movements and subjective ratings as measures of cognitive effort in post- editing. It is expected that findings in this respect will inform further cognitive investigations not only in post-editing but also in translation and traditional translation revision.