|
|
Modern Linguistics 2021
口译多模态加工中的手势认知效应研究
|
Abstract:
本文使用ELAN多模态话语分析软件对两段中国导演海外影片宣传的口译活动进行了分析。结果发现手势是口译加工中重要的显身特征、信息模态和认知资源，具体来说：1) 认知负荷更大时，手势数量更多，手势位置更高，手势更加快速而频繁，但主要手势类型都是节奏性手势；2) 口译认知加工中的手势、语言和副语言产品具有跨模态统一性；3) 口译质量更高的译员更倾向于模仿讲者手势，但手势类型可能发生变化。同时，手势数量也可能受到现场语境因素以及译员认知资源的影响。本研究构建的初步框架和分析工具有助于弥补口译手势多模态研究的不足，相关结论亦可促进基于询证的口译教学。
This research uses a multi-modal discourse analysis software, ELAN, to analyse two inter-preter-mediated events featuring Chinese film directors promoting their films to an international audience. It is found that gestures represent an important embodied information modality and cognitive resource in interpreting processing. Specifically: 1) With higher cognitive load, more gestures with higher positions are made, with faster frequency and pace, although beats remain the dominant gesture type for all speakers, including interpreters; 2) Gestures align with the interpreter’s linguistic and paralinguistic products during the multimodal cognitive processing of interpreting; 3) Interpreters achieve higher interpreting quality scores through more imitation of speaker gestures, although the imitator may change the gesture type. Meanwhile, contextual factors and availability of the interpreter’s cognitive resources may also impact on the use of gestures. The preliminary framework and analysis tools built in this research could fill some of the gaps in understanding the multi-modal nature of interpreting gestures. The initial findings could also promote evidence-based interpreter education.
| [1] | Feyereisen, P. and De Lannoy, J. (1991) Gestures and Speech: Psychological Investigations. Cambridge University Press, Cambridge. |
| [2] | Kendon, A. (1980) Gesticulation and Speech: Two Aspects of the Process of Utterance. In: Key, M.R., Ed., The Relationship of Verbal and Nonverbal Communication, Mouton, Hague, The Netherlands, 208-227.
https://doi.org/10.1515/9783110813098.207 |
| [3] | Bressem, J. and Ladewig, S. (2011) Rethinking Gesture Phases: Articulatory Features of Gestural Movement. Semiotica, 2011, 53-91. https://doi.org/10.1515/semi.2011.022 |
| [4] | Kendon, A. (2004) Gesture: Visible Action as Utterance. Cambridge University Press, Cambridge.
https://doi.org/10.1017/CBO9780511807572 |
| [5] | McNeill, D. (1992) Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago. |
| [6] | Bruner, J.S. (1990) Acts of Meaning. Harvard University Press, Cambridge. |
| [7] | Alibali, M., Bassok, M., Solomon, K., Syc, S. and Goldin-Meadow, S. (1999) Illuminating Mental Representations through Speech and Gesture. Psychological Science, 10, 327-333. https://doi.org/10.1111/1467-9280.00163 |
| [8] | McNeill, D. (2005) Gestures and Thought. University of Chicago Press, Chicago. |
| [9] | Goldin-Meadow, S. and Wagner, S.M. (2005) How Our Hands Help Us Learn. Trends in Cognitive Sciences, 9, 234-241. https://doi.org/10.1016/j.tics.2005.03.006 |
| [10] | Goldin-Meadow, S. (2011) Learning through Gesture. Wiley Interdisciplinary Reviews: Cognitive Science, 2, 595-607.
https://doi.org/10.1002/wcs.132 |
| [11] | Chu, M., Meyer, A., Foulkes, L. and Kita, S. (2014) Individual Differences in Frequency and Saliency of Speech-Accompanying Gestures: The Role of Cognitive Abilities and Empathy. Journal of Experimental Psychology: General, 143, 694-709. https://doi.org/10.1037/a0033861 |
| [12] | Gillespie, M., James, A.N., Federmeier, K.D. and Watson, D.G. (2014) Verbal Working Memory Predicts Co-speech Gesture: Evidence from Individual Differences. Cognition, 132, 174-180.
https://doi.org/10.1016/j.cognition.2014.03.012 |
| [13] | Pouw, W.T., Mavilidi, M.F., Van Gog, T. and Paas, F. (2016) Gesturing during Mental Problem Solving Reduces Eye Movements, Especially for Individuals with Lower Visual Working Memory Capacity. Cognitive Processing, 17, 269-277. https://doi.org/10.1007/s10339-016-0757-6 |
| [14] | Wu, Y.C. and Coulson, S. (2014) Co-Speech Iconic Gestures and Visuo-spatial Working Memory. Acta psychologica, 153, 39-50. https://doi.org/10.1016/j.actpsy.2014.09.002 |
| [15] | ？zer, D. and G？ksun, T. (2020) Visual-Spatial and Verbal Abilities Differentially Affect Processing of Gestural vs. Spoken Expressions. Language, Cognition and Neuroscience, 35, 896-914.
https://doi.org/10.1080/23273798.2019.1703016 |
| [16] | Kendon, A. (1972) Some Relationships between Body Motion and Speech. In: Seigman, A. and Pope, B., Eds., Studies in Dyadic Communication, Pergamon Press, Elmsford, 177-216. https://doi.org/10.1016/B978-0-08-015867-9.50013-7 |
| [17] | Skipper, J., Goldin-Meadow, S., Nusbaum, H. and Small, S. (2007) Speech-Associated Gestures, Broca’s Area, and the Human Mirror System. Brain and Language, 101, 260-277. https://doi.org/10.1016/j.bandl.2007.02.008 |
| [18] | Hostetter, A.B., Alibali, M.W. and Kita, S. (2007) I See It in My Hands’ Eye: Representational Gestures Reflect Conceptual Demands. Language and Cognitive Processes, 22, 313-336. https://doi.org/10.1080/01690960600632812 |
| [19] | Jenkins, T., Coppola, M. and Coelho, C. (2017) Effects of Gesture Restriction on Quality of Narrative Production. Gesture, 16, 416-431. https://doi.org/10.1075/gest.00003.jen |
| [20] | Rauscher, F.H., Krauss, R.M. and Chen, Y. (1996) Gesture, Speech, and Lexical Access: The Role of Lexical Movements in Speech Production. Psychological Science, 7, 226-231. https://doi.org/10.1111/j.1467-9280.1996.tb00364.x |
| [21] | Morsella, E. and Krauss, R.M. (2004) The Role of Gestures in Spatial Working Memory and Speech. The American Journal of Psychology, 117, 411-424. https://doi.org/10.2307/4149008 |
| [22] | Dargue, N., Sweller, N. and Jones, M.P. (2019) When Our Hands Help Us Understand: A Meta-analysis into the Effects of Gesture on Comprehension. Psychological Bulletin, 145, 765-784. https://doi.org/10.1037/bul0000202 |
| [23] | Goldin-Meadow, S., Cook, S.W. and Mitchell, Z.A. (2009) Gesturing Gives Children New Ideas about Math. Psychological Science, 20, 267-272. https://doi.org/10.1111/j.1467-9280.2009.02297.x |
| [24] | Stieff, M., Lira, M.E. and Scopelitis, S.A. (2016) Gesture Supports Spatial Thinking in STEM. Cognition and Instruction, 34, 80-99. https://doi.org/10.1080/07370008.2016.1145122 |
| [25] | Chu, M. and Kita, S. (2011) The Nature of Gestures’ Beneficial Role in Spatial Problem Solving. Journal of Experimental Psychology: General, 140, 102-116. https://doi.org/10.1037/a0021790 |
| [26] | Eielts, C., Pouw, W., Ouwehand, K., Van Gog, T., Zwaan, R.A. and Paas, F. (2020) Co-Thought Gesturing Supports More Complex Problem Solving in Subjects with Lower Visual Working-memory Capacity. Psychological Research, 84, 502-513. https://doi.org/10.1007/s00426-018-1065-9 |
| [27] | Gluhareva, D and Prieto P. (2017) Training with Rhythmic Beat Gestures Benefits L2 Pronunciation in Discourse-Demanding Situations. Language Teaching Research, 21, 609-631. https://doi.org/10.1177/1362168816651463 |
| [28] | Novack, M. and Goldin-Meadow, S. (2015) Learning from Gesture: How Our Hands Change Our Minds. Educational Psychology Review, 27, 405-412. https://doi.org/10.1007/s10648-015-9325-3 |
| [29] | Furuyama, N., Nobe, S., Someya, Y., Sekine, K. and Hayashi, S. (2005) A Study on Gestures in Simultaneous Interpreters. Interpretation Studies, 5, 111-136. |
| [30] | Rennert, S. (2008) Visual Input in Simultaneous Interpreting. Meta, 53, 204-217. https://doi.org/10.7202/017983ar |
| [31] | Zagar-Galv？o, E. (2009) Speech and Gesture in the Booth—A Descriptive Approach to Multimodality in Simultaneous Interpreting. In: De Crom, D., Ed., Selected Papers of the CETRA Research Seminar in Translation Studies 2008, CETRA, Leuven. |
| [32] | Zagar-Galv？o, E. (2013) Hand Gestures and Speech Production in the Booth: Do Simultaneous Interpreters Imitate the Speaker. In: Carapinha, C. and Santos, I., Eds., Estudos de Linguística, Coimbra University Press, Coimbra, 115-129.
https://doi.org/10.14195/978-989-26-0714-6_7 |
| [33] | Zagar-Galv？o, E. (2020) Gesture Functions and Gestural Style in Simultaneous Interpreting. In: Salaets, H. and Br？ne, G., Eds., Linking up with Video: Perspectives on Interpreting Practice and Research, John Benjamins Publishing Company, Amsterdam/Philadelphia, 151-179. https://doi.org/10.1075/btl.149.07gal |
| [34] | Stachowiak-Szymczak, K. (2019) Eye Movements and Gestures in Simultaneous and Consecutive Interpreting. Springer International Publishing, Cham. https://doi.org/10.1007/978-3-030-19443-7 |
| [35] | Gerwing, J. and Li, S. (2019) Body-Oriented Gestures as a Practitioner’s Window into Interpreted Communication. Social Science & Medicine, 233, 171-180. https://doi.org/10.1016/j.socscimed.2019.05.040 |
| [36] | Gile, D. (2009) Basic Concepts and Models for Interpreter and Translator Training. Revised Edition, John Benjamins Publishing Company, Amsterdam. https://doi.org/10.1075/btl.8 |
| [37] | Cienki, A. and Müller, C. (2008) Metaphor and Gesture. John Benjamins, Amsterdam. https://doi.org/10.1075/gs.3 |
| [38] | 齐涛云, 杨承淑. 多模态同传语料库的开发与建置——以职业译员英汉双向同传语料库为例. 中国翻译, 2020, 41(3): 126-135, 189. |
| [39] | 王斌华. 语料库口译研究——口译产品研究方法的突破. 中国外语, 2012(3): 96-102. |
| [40] | Goldman-Eisler, F. (1958) Speech Production and the Predictability of Words in Context. The Quarterly Journal of Experimental Psychology, 10, 96-109. https://doi.org/10.1080/17470215808416261 |
| [41] | Plevoets, K. and Defrancq, B. (2016) The Effect of Informational Load on Disfluencies in Interpreting. Translation and Interpreting Studies: The Journal of the American Translation and Interpreting Studies Association, 11, 202-224.
https://doi.org/10.1075/tis.11.2.04ple |
| [42] | Plevoets, K. and Defrancq, B. (2018) The Cognitive Load of Interpreters in the European Parliament: A Corpus-based Study of Predictors for the Disfluency uh(m). Interpreting, 20, 1-28. https://doi.org/10.1075/intp.00001.ple |
| [43] | Xiang, X., Zheng, B. and Feng, D. (2020) Interpreting Impoliteness and Over-politeness: An Investigation into Interpreters’ Cognitive Effort, Coping Strategies and Their Effects. Journal of Pragmatics, 169, 231-244.
https://doi.org/10.1016/j.pragma.2020.09.021 |
| [44] | Paas, F. and Sweller, J. (2012) An Evolutionary Upgrade of Cognitive Load Theory: Using the Human Motor System and Collaboration to Support the Learning of Complex Cognitive Tasks. Educational Psychology Review, 24, 27-45.
https://doi.org/10.1007/s10648-011-9179-2 |
| [45] | Gile, D. (1999) Testing the Effort Models’ Tightrope Hypothesis in Simultaneous Interpreting—A Contribution. Hermes, 12, 153-172. https://doi.org/10.7146/hjlcb.v12i23.25553 |
| [46] | Gile, D. (2008) Local Cognitive Load in Simultaneous Interpreting and Its Implications for Empirical Research. Forum, 6, 59-77. https://doi.org/10.1075/forum.6.2.04gil |
| [47] | Xie, B. and Salvendy, G. (2000) Prediction of Mental Workload in Single and Multiple Task Environments. International Journal of Cognitive Ergonomics, 4, 213-242. https://doi.org/10.1207/S15327566IJCE0403_3 |
| [48] | Gile, D. (2011) Errors, Omissions and Infelicities in Broadcast Interpreting. Preliminary Findings from a Case Study. In: Alvstad, C., Hild, A. and Tiselius, E., Eds., Methods and Strategies of Process Research. Integrative Approaches in Translation Studies, John Benjamins, Amsterdam/Philadelphia, 201-218. |
