Журнал высшей нервной деятельности им. И.П. Павлова, 2022, T. 72, № 5, стр. 609-622

Методика айтрекинга в психолингвистике и параллельная регистрация с ЭЭГ

М. Ю. Походай 1*, Б. Бермудес-Маргаретто 4, Ю. Ю. Штыров 12, А. В. Мячиков 13

1 Центр нейроэкономики и когнитивных исследований Института когнитивных нейронаук НИУ ВШЭ
Москва, Россия

2 Центр функционально интегративной нейронауки (CFIN), кафедра клинической медицины, Орхусский университет
Орхус, Дания

3 Университет Нортумбрии в Ньюкасле
Ньюкасл-апон-Тайн, Великобритания

4 Университет Саламанки
Саламанка, Испания

* E-mail: mpokhoday@hse.ru

Поступила в редакцию 15.03.2022
После доработки 22.06.2022
Принята к публикации 27.06.2022

Аннотация

Развитие технологий привело к значительному прогрессу в применяемых методах исследований в когнитивных науках. Среди методов изучения поведения человека и лежащих в его основе мозговых механизмов и когнитивных процессов – айтрекинг (окулография) – метод записи и анализа глазодвигательной активности в реальном времени. В этом обзоре мы рассмотрим использование айтрекинга в когнитивных исследованиях как отдельно, так и в сочетании с электроэнцефалографией, а именно анализом связанных с событиями потенциалов (ССП или ПСС, eng. Event-related potentials (ERP)). Мы также обсудим отслеживание взгляда с точки зрения его использования в языковых исследованиях: от изучения понимания и построения предложений до изучения второго языка и двуязычия. Наконец, речь пойдет о параллельной регистрации глазодвижения и ERP. Своим обзором мы хотим обратить внимание не только на сильные стороны методики айтрекинг, но и на проблемы, которые, как мы считаем, можно решить путем применения параллельной регистрации глазодвигательной активности и ЭЭГ.

Ключевые слова: айтрекинг, окулография, ЭЭГ, ССП, ERP, когнитивные науки, мультимодальные методы, психолингвистика, параллельная регистрация

Список литературы

  1. Анисимов В.Н., Ковалева А.В., Бойко Л.А., Шеденко К.Ю., Галкина Н.В., Латанов А.В. ЭЭГ, вегетативные и глазодвигательные показатели интереса потребителя на примере экскурсионных туров. Вестник психофизиологии. 2019a. 1: 45–54.

  2. Анисимов В.Н., Зинченко Е.М., Бойко Л.А., Кушнир А.Б., Латанов А.В., Галкина Н.В. Методы оценки коммерческих текстов: нейромаркетинговый подход. Практический маркетинг. 2019b. 3(265): 3–14.

  3. Серов И.Е., Ковалева А.В., Казьмина В.Г., Галкина Н.В. Прогнозирование вероятности запоминания логотипов неизвестных брендов по объективным электрофизиологическим показателям. Практический маркетинг. 2019. 5(267): 3–10.

  4. Ярбус А.Л. Роль движений глаз в процессе зрения. М.: “Наука”. 1965.

  5. Allopenna P.D., Magnuson J.S., Tanenhaus M.K. Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of memory and language. (1998). 38(4): 419–439.

  6. Altarriba J., Kroll J.F., Sholl A., Rayner K. The influence of lexical and conceptual constraints on reading mixed-language sentences: Evidence from eye fixations and naming times. Memory & cognition. 1996. 24(4): 477–492.

  7. Altmann G.T., Kamide Y. Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition. 1999. 73(3)d and average readers: Differential effects of frequency and predictability. The Quarterly Journal of Experimental Psychology Section A. 2005. 58(6): 247–264.

  8. Baccino T., Manunta Y. Eye-fixation-related potentials: Insight into parafoveal processing. Journal of Psychophysiology. 2005. 19(3): 204–215.

  9. Balling L.W. Does Good Writing Mean Good Reading?: An Eye-Tracking Investigation of the Effect of Writing Advice on Reading. Fachsprache: International Journal of Specialized Communication. 2013. 35(1–2): 2–23.

  10. Bastiaansen M., Hagoort P. Oscillatory neuronal dynamics during language comprehension. Progress in brain research. 2006. 159: 179–196.

  11. Bentin S., Mouchetant-Rostaing Y., Giard M.H., Echallier J.F., Pernier J. ERP manifestations of processing printed words at different psycholinguistic levels: time course and scalp distribution. Journal of Cognitive Neuroscience. 1999. 11(3): 235–260.

  12. Berg P., Scherg M. Dipole modelling of eye activity and its application to the removal of eye artefacts from the EEG and MEG. Clinical Physics and Physiological Measurement. 1991. 12A (49).

  13. Blythe H.I., Liversedge S.P., Joseph H.S., White S.J., Findlay J.M., Rayner K. The binocular coordination of eye movements during reading in children and adults. Vision Research. 2006. 46(22): 3898-3908.

  14. Bock K., Irwin D.E., Davidson D.J., Levelt W.J. Minding the clock. Journal of Memory and Language. 2003. 48(4): 653–685.

  15. Bolger D.J., Balass M., Landen E., Perfetti C.A. Context variation and definitions in learning the meanings of words: An instance-based learning approach. Discourse processes. 2008. 45(2): 122–159.

  16. Bressler S.L., Richter C.G. Interareal oscillatory synchronization in top-down neocortical processing. Current opinion in neurobiology. 2015. 31: 62–66.

  17. Brysbaert M., Drieghe D., Vitu F. Cognitive processes in eye guidance. 2005.

  18. Buswell G.T. An experimental study of the eye-voice span in reading (No. 17). University of Chicago. 1920.

  19. Chace K.H., Rayner K., Well A.D. Eye movements and phonological parafoveal preview: effects of reading skill. Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale. 2005. 59(3): 209.

  20. Chaffin R., Morris R.K., Seely R.E. Learning new word meanings from context: A study of eye movements. Journal of Experimental Psychology: Learning, Memory, and Cognition. 2001. 27(1): 225.

  21. Chambers C.G., Cooke H. Lexical competition during second-language listening: Sentence context, but not proficiency, constrains interference from the native lexicon. Journal of Experimental Psychology: Learning, Memory, and Cognition. 2009. 35(4): 1029.

  22. Clifton Jr.C., Ferreira F., Henderson J.M., Inhoff A.W., Liversedge S.P., Reichle E.D., Schotter E.R. Eye movements in reading and information processing: Keith Rayner’s 40 year legacy. Journal of Memory and Language. 2016. 86: 1–19.

  23. Clifton Jr.C., Staub A., Clifton C. Syntactic influences on eye movements during reading. Eye. (2011). 3(2).

  24. Cop U., Keuleers E., Drieghe D., Duyck W. Frequency effects in monolingual and bilingual natural reading. Psychonomic bulletin & review. 2015. 22(5): 1216–1234.

  25. Cooper R.M. The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology. 1974.

  26. Conklin K., Pellicer-Sánchez A. Using eye-tracking in applied linguistics and second language research. Second Language Research. 2016. 32(3): 453–467.

  27. Conklin K., Pellicer-Sánchez A., Carrol G. Eye-tracking: A guide for applied linguistics research. Cambridge University Press. 2018.

  28. Coulson S. Electrifying results: ERP data and cognitive linguistics. Methods in cognitive linguistics. 2007. 18.

  29. Croft R.J., Barry R.J. Removal of ocular artifact from the EEG: a review. Neurophysiologie Clinique/Clinical Neurophysiology. 2000. 30(1): 5–19.

  30. Delorme A., Sejnowski T., Makeig S. Enhanced detection of artifacts in EEG data using higher-order statistics and independent component analysis. Neuroimage. 2007. 34(4): 1443–1449.

  31. Dimigen O., Kliegl R., Sommer W. Trans-saccadic parafoveal preview benefits in fluent reading: A study with fixation-related brain potentials. NeuroImage. 2012. 62: 381–393.

  32. Dimigen O., Sommer W., Hohlfeld A., Jacobs A.M., Kliegl R. Coregistration of eye movements and EEG in natural reading: Analyses and review. Journal of Experimental Psychology–General. 2011. 140: 552–572.

  33. Duchowski A.T. Diversity and types of eye tracking applications. In Eye Tracking Methodology. 2017. 247–248.

  34. Eberhard K.M., Spivey-Knowlton M.J., Sedivy J.C., Tanenhaus M.K. Eye movements as a window into real-time spoken language comprehension in natural contexts. Journal of psycholinguistic research. 1995. 24(6): 409–436.

  35. Ehrlich K., Rayner K. Pronoun assignment and semantic integration during reading: Eye movements and immediacy of processing. Journal of verbal learning and verbal behavior. 1983. 22(1): 75–87.

  36. Elgort I., Brysbaert M., Stevens M., Van Assche E. Contextual word learning during reading in a second language: An eye-movement study. Studies in Second Language Acquisition. 2018. 40(2): 341–366.

  37. Fairbanks G. The relation between eye-movements and voice in the oral reading of good and poor silent readers. Psychological Monographs. 1937. 48(3): 78.

  38. Feng G., Miller K., Shu H., Zhang H. Orthography and the development of reading processes: An eye-movement study of Chinese and English. Child development. 2009. 80(3): 720–735.

  39. Fischer T., Graupner S.T., Velichkovsky B.M., Pannasch S. Attentional dynamics during free picture viewing: evidence from oculomotor behavior and electrocortical activity. Frontiers in systems neuroscience. 2013. 7: 17.

  40. Friederici A.D., Weissenborn J. Mapping sentence form onto meaning: The syntax– semantic interface. Brain research. 2007. 1146: 50–58.

  41. Gleitman L.R., January D., Nappa R., Trueswell J.C. On the give and take between event apprehension and utterance formulation. Journal of memory and language. 2007. 57(4): 544–569.

  42. Griffin Z.M. Gaze durations during speech reflect word selection and phonological encoding. Cognition. 2001. 82(1), B1–B14.

  43. Griffin Z.M., Bock K. What the eyes say about speaking. Psychological science. 2000. 11(4): 274–279.

  44. Griffin Z.M., Weinstein-Tull J. Conceptual structure modulates structural priming in the production of complex sentences. Journal of Memory and Language. 2003. 49(4): 537–555.

  45. Godfroid A., Boers F., Housen A. An eye for words: Gauging the role of attention in incidental L2 vocabulary acquisition by means of eye-tracking. Studies in Second Language Acquisition. 2013. 35(3): 483–517.

  46. Godfroid A., Ahn J., Choi I., Ballard L., Cui Y., Johnston S., Yoon H.J. Incidental vocabulary learning in a natural reading context: an eye-tracking study. Bilingualism: Language and Cognition. 2018. 21(3): 563–584.

  47. Henderson J.M., Luke S.G., Schmidt J., Richards J.E. Co-registration of eye movements and event-related potentials in connected-text paragraph reading. Frontiers in systems neuroscience. 2013. 7: 28.

  48. Himmelstoss N.A., Schuster S., Hutzler F., Moran R., Hawelka S. Co-registration of eye movements and neuroimaging for studying contextual predictions in natural reading. Language, Cognition and Neuroscience. 2019. 1–18.

  49. Holmqvist K., Nyström M., Andersson R., Dewhurst R., Jarodzka H., Van de Weijer J. Eye tracking: A comprehensive guide to methods and measures. OUP Oxford. 2011.

  50. Huestegge L., Radach R., Corbic D., Huestegge S.M. Oculomotor and linguistic determinants of reading development: A longitudinal study. Vision research. 2009. 49(24): 2948–2959.

  51. Huey E.B. The psychology and pedagogy of reading. The Macmillan Company. 1908.

  52. Huettig F., McQueen J.M. The tug of war between phonological, semantic and shape information in language-mediated visual search. Journal of Memory and Language. 2007. 57(4): 460- 482.

  53. Huettig F., Rommers J., Meyer A.S. Using the visual world paradigm to study language processing: A review and critical evaluation. Acta psychologica. 2011. 137(2), 151–171.

  54. Hutzler F., Braun M., Võ M.L.H., Engl V., Hofmann M., Dambacher M., et al. Welcome to the real world: validating fixation-related brain potentials for ecologically valid settings. Brain Res. 2007. 1172: 124–129.

  55. Ille N., Berg P., Scherg M. Artifact correction of the ongoing EEG using spatial filters based on artifact and brain signal topographies. Journal of clinical neurophysiology. 2002. 82: 242–253.

  56. Javal E. Essai sur la physiologie de la lecture. Annaels d’o- culistique. 1879. 82: 242–253.

  57. Joseph H.S., Wonnacott E., Forbes P., Nation K. Becoming a written word: Eye movements reveal order of acquisition effects following incidental exposure to new words during silent reading. Cognition. 2014. 133(1): 238–248.

  58. Ju M., Luce P.A. Falling on sensitive ears: Constraints on bilingual lexical activation. Psychological Science. 2004. 15(5), 314–318.

  59. Juhasz B.J., White S.J., Liversedge S.P., Rayner K. Eye movements and the use of parafoveal word length information in reading. Journal of Experimental Psychology: Human Perception and Performance. 2008. Toм 34(6): 1560.

  60. Just M.A., Carpenter P.A. A theory of reading: From eye fixations to comprehension. Psychological review. 1980. 87(4): 329.

  61. Kennedy A. Parafoveal processing in word recognition. The Quarterly Journal of Experimental Psychology Section A. 2000. 53(2): 429–455.

  62. Knoeferle P., Crocker M.W., Scheepers C., Pickering M.J. The influence of the immediate visual context on incremental thematic role-assignment: Evidence from eye- movements in depicted events. Cognition. 2005. 95(1): 95–127.

  63. Knoeferle P., Crocker M.W. Incremental effects of mismatch during picture- sentence integration: Evidence from eye-tracking. In Proceedings of the 26th annual conference of the cognitive science society. 2005. pp. 1166–1171.

  64. Koval N.G. Testing the deficient processing account of the spacing effect in second language vocabulary learning: Evidence from eye tracking. Applied Psycholinguistics. 2019. 40(5): 1103–1139.

  65. Kornrumpf B., Dimigen O., Sommer W. Lateralization of posterior alpha EEG reflects the distribution of spatial attention during saccadic reading. Psychophysiologyю. 2017. 54(6): 809–823.

  66. Kretzschmar F., Bornkessel-Schlesewsky I., Schlesewsky M. Parafoveal versus foveal N400s dissociate spreading activation from contextual fit. NeuroReport. 2009. 20: 1613–1618.

  67. Kretzschmar F., Schlesewsky M., Staub A. Dissociating word frequency and predictability effects in reading: Evidence from coregistration of eye movements and EEG. Journal of Experimental Psychology: Learning, Memory, and Cognition. 2015. 41(6): 1648.

  68. Kutas M., Federmeier K.D. Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). Annual review of psychology. 2011. 62: 621–647.

  69. Kutas M., Van Petten C.K., Kluender R. Psycholinguistics electrified II (1994–2005). In Handbook of psycholinguistics. Academic Press. 2006. pp. 659–724.

  70. Laeng B., Sirois S., Gredebäck G. Pupillometry: A window to the preconscious? Perspectives on psychological science. 2012. 7(1): 18–27.

  71. Lewis A.G., Wang L., Bastiaansen M. Fast oscillatory dynamics during language comprehension: Unification versus maintenance and prediction?. Brain and language. 148: 51–63.

  72. Li N., Niefind F., Wang S., Sommer W., Dimigen O. Parafoveal processing in reading Chinese sentences: Evidence from event-related brain potentials. Psychophysiology. 2015. 52(10): 1361–1374.

  73. Li L., Marinus E., Castles A., Yu L., Wang H.C. Eye-tracking the effect of semantic decoding on orthographic learning in Chinese. 2019.

  74. Libben M.R., Titone D.A. Bilingual lexical access in context: evidence from eye movements during reading. Journal of Experimental Psychology: Learning, Memory, and Cognition. 2009. 35(2): 381.

  75. López-Peréz P.J., Dampuré J., Hernández-Cabrera J.A., Barber H.A. Semantic parafoveal-on-foveal effects and preview benefits in reading: Evidence from Fixation Related Potentials. Brain and language. 2016. 162: 29–34.

  76. Lowell R., Morris R.K. Word length effects on novel words: Evidence from eye movements. Attention, Perception, & Psychophysics. 2014. 76(1): 179–189.

  77. MacGregor L.J., Pulvermüller F., Van Casteren M., Shtyrov Y. Ultra-rapid access to words in the brain. Nature Communications. 2012. 3: 711.

  78. Marian V., Spivey M. Competing activation in bilingual language processing: Within-and between-language competition. Bilingualism: Language and Cognition. 2003. 6(2): 97–115.

  79. Marian V., Spivey M., Hirsch J. Shared and separate systems in bilingual language processing: Converging evidence from eyetracking and brain imaging. Brain and language. 2003. 86(1): 70–82.

  80. Mathôt S. Pupillometry: Psychology, physiology, and function. Journal of Cognition. 2018. 1(1).

  81. McConkie G.W., Zola D., Grimes J., Kerr P.W., Bryant N.R., Wolff P.M. Children’s eye movements during reading. Vision and visual dyslexia. 1991. 13: 251–262.

  82. Meyer A.S., Sleiderink A.M., Levelt W.J. (Viewing and naming objects: Eye movements during noun phrase production. Cognition.1998. 66(2): B25–B33.

  83. Metzner P., von der Malsburg T., Vasishth S., Rösler F. Brain responses to world knowledge violations: A comparison of stimulus-and fixation-triggered event-related potentials and neural oscillations. Journal of Cognitive Neuroscience. 2015. 27(5): 1017–1028.

  84. Miles W.R. Ocular dominance in human adults. The journal of general psychology. 1930. 3(3): 412–430.

  85. Mohamed A.A. Exposure frequency in L2 reading: An eye-movement perspective of incidental vocabulary learning. Studies in Second Language Acquisition. 2018. Toм 40(2): 269–293.

  86. Nikolaev A.R., Jurica P., Nakatani C., Plomp G., van Leeuwen C. Visual encoding and fixation target selection in free viewing: presaccadic brain potentials. Front Syst Neurosci. 2013. 7: 26.

  87. Nikolaev A.R., Meghanathan R.N., van Leeuwen C. Combining EEG and eye movement recording in free viewing: Pitfalls and possibilities. Brain and cognition. 2016. 107: 55–83.

  88. Nikolaev A.R., Nakatani C., Plomp G., Jurica P., van Leeuwen C. Eye fixation-related potentials in free viewing identify encoding failures in change detection. Neuroimage. 2011. 56: 1598–1607.

  89. Nyström M., Andersson R., Holmqvist K., Van De Weijer J. The influence of calibration method and eye physiology on eyetracking data quality. Behavior research methods. 2013. 45(1): 272–288.

  90. Pickering M.J., Frisson S., McElree B., Traxler M.J. Eye movements and semantic composition. The on-line study of sentence comprehension: Eyetracking, ERP, and beyond. 2004. 33–50.

  91. Picton T.W., van Roon P., Armilio M.L., Berg P., Ille N., Scherg M. The correction of ocular artifacts: a topographic perspective. Clinical Neurophysiology. 2000. 111(1): 53–65.

  92. Pellicer-Sánchez A. Incidental L2 vocabulary acquisition from and while reading: An eye-tracking study. Studies in Second Language Acquisition. 2016. 38(1): 97–130.

  93. Raney G.E., Campbell S.J., Bovee J.C. Using eye movements to evaluate the cognitive processes involved in text comprehension. JoVE (Journal of Visualized Experiments). 2014. 83.

  94. Rayner K. Eye movements and the perceptual span in beginning and skilled readers. Journal of experimental child psychology. 1986. 41(2): 211–236.

  95. Rayner K. Eye movements in reading and information processing: 20 years of research. Psychological bulletin. 1998. 124(3): 372.

  96. Rayner K., Duffy S.A. Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity. Memory & cognition. 1986. 14(3): 191–201.

  97. Rayner K., Pollatsek A., Reisberg D. Basic processes in reading. The Oxford handbook of cognitive psychology. Oxford University Press, New York. 2013. 442–461.

  98. Rayner K., Sereno S.C., Raney G.E. Eye movement control in reading: a comparison of two types of models. Journal of Experimental Psychology: Human Perception and Performance. 1996. 22(5): 1188.

  99. Rayner K., Slattery T.J., Drieghe D., Liversedge S.P. Eye movements and word skipping during reading: effects of word length and predictability. Journal of Experimental Psychology: Human Perception and Performance. 2011. 37(2): 514.

  100. Reichle E.D., Perfetti C.A. Morphology in word identification: A word-experience model that accounts for morpheme frequency effects. Scientific Studies of Reading. 2003. 7(3), 219–237.

  101. Sereno S.C., Rayner K. Measuring word recognition in reading: eye movements and event-related potentials. Trends Cogn. Sci. 2003. 7: 489–493.

  102. Sereno S.C., Rayner K., and Posner M.I. Establishing a time-line of word recognition: evidence from eye movements and event-related potentials. Neuroreport. 1998. 9: 2195–2200.

  103. Shtyrov Y., Lenzen M. First-pass neocortical processing of spoken language takes only 30 msec: Electrophysiological evidence. Cognitive neuroscience. 2017. 8(1): 24–38.

  104. Siegel M., Donner T.H., Engel A.K. Spectral fingerprints of large-scale neuronal interactions. Nature Reviews Neuroscience. 2012. 13(2): 121.

  105. Simola J., Holmqvist K., Lindgren M. Right visual field advantage in parafoveal processing: Evidence from eye-fixation-related potentials. Brain and language. 2009. 111(2): 101–113.

  106. Simola J., Le Fevre K., Torniainen J., Baccino T. Affective processing in natural scene viewing: Valence and arousal interactions in eye-fixation-related potentials. NeuroImage. 2015. 106: 21–33.

  107. Simola J., Torniainen J., Moisala M., Kivikangas M., Krause C.M. Eye movement related brain responses to emotional scenes during free viewing. Frontiers in systems neuroscience. 2013. 7, 41.

  108. Singer W. Dynamic formation of functional networks by synchronization. Neuron. 2011. 69(2): 191–193.

  109. Starr M., Inhoff A. Attention allocation to the right and left of a fixated word: Use of orthographic information from multiple words during reading. European Journal of Cognitive Psychology. 2004. 16(1–2): 203–225.

  110. Staub A., Rayner K. Eye movements and on-line comprehension processes. The Oxford handbook of psycholinguistics. 2007. 327–342.

  111. Takeda Y., Sugai M., Yagi A. Eye fixation related potentials in a proof reading task. International Journal of Psychophysiology. 2001. 40(3): 181–186.

  112. Tanenhaus M.K., Trueswell J.C. Eye movements and spoken language comprehension. In Handbook of psycholinguistics. Academic Press. 2006. pp. 863–900.

  113. Tanenhaus M.K., Spivey-Knowlton M.J., Eberhard K.M., Sedivy J.C. Integration of visual and linguistic information in spoken language comprehension. Science. 1995. 268(5217), 1632–1634.

  114. Tanenhaus M.K., Spivey-Knowlton M.J., Eberhard K.M., Sedivy J.C. Using eye movements to study spoken language comprehension: Evidence for visually mediated incremental interpretation. 1996.

  115. Vignali L., Himmelstoss N.A., Hawelka S., Richlan F., Hutzler F. Oscillatory brain dynamics during sentence reading: a fixation-related spectral perturbation analysis. Frontiers in human neuroscience. 2016. 10. 191.

  116. Von Stein A., Chiang C., König P. Top-down processing mediated by interareal synchronization. Proceedings of the National Academy of Sciences. 2000. 97(26): 14748–14753.

  117. Weber A., Melinger A., Lara Tapia L. The mapping of phonetic information to lexical presentations in Spanish: evidence from eye movements. In 16th International Congress of Phonetic Sciences (ICPhS 2007). Pirrot. 2007. pp. 1941–1944.

  118. Wochna K.L., Juhasz B.J. Context length and reading novel words: An eye- movement investigation. British Journal of Psychology. 2013. 104(3): 347–363.

  119. Yarbus A.L. Role of eye movements in the visual processes. 1965.

Дополнительные материалы отсутствуют.