Mobile Eye Tracking During Storybook Listening: Applying the Visual World Paradigm in the Investigation of Preschoolers' Online Discourse Processing

• Author / Creator

  Toth, Abigail

• The current thesis assessed the application of the visual world eye tracking paradigm (VWP) as a tool for investigating online language processing in a naturalistic setting. Furthermore, it investigated how individual differences in vocabulary and working memory influence children’s eye movements within in the visual scene with respect to spoken language input.
  In the VWP an individual’s eye movements are monitored as they receive spoken language input and view a visual scene. It works under the assumption that where the individual is looking indicates where their attention is focused and thus what they are processing at any given moment. The VWP has been used to investigate the online processing of various linguistic phenomena, including the processing of reference, particularly in the realm of online pronoun resolution. These studies have shown that upon hearing an ambiguous third person singular pronoun (i.e., ‘he’), there is an increased proportion of looks to the subject of the preceding clause, suggesting that people are more likely to interpret the pronoun as co-referring with the subject (e.g., Arnold, Eisenband, Brown-Schmidt, & Trueswell, 2000; Järvikivi, Van Gompel, Hyönä, & Bertram 2005; Song & Fisher, 2005; 2007). For example when participants hear utterances such as ‘The panda hit the parrot by the lake. He wanted to go home’ while viewing a scene with both animals, they are more likely to look at the panda than the parrot when they hear ‘he’, indicating that there is a subject bias. However, almost all of the previous studies have looked at a series of isolated items (such as the example above), with only to referents in the visual scene (i.e., the subject and object). We do not know how the VWP works in more naturalistic settings, that is, when there is continuous linguistic discourse and multiple referents in the visual scene, as is the case in storybook listening.
  Both children and adults listened to a five-minute long storybook while wearing eye-tracking glasses. The storybook contained multiple referring expressions, both names (e.g., ‘Bear’) and pronouns (e.g., ‘he’), and was designed to becoming increasingly more complex as it unfolded over time, beginning with just a single character and ending with a total of five characters. Using generalized additive mixed modeling (GAMMs), we analyzed the eye gaze data of 16 children and 12 adults with respect to the mention of 37 names and 10 pronouns embedded throughout the story. Overall we found that eye movements patterns differed for items (names and pronouns) that occurred during the first half of the story compared to items (names and pronouns) that occurred during the second half of the story, for both children and adults. Upon hearing a name during the first half of the story, both children and adults’ looks to the target referent increased. Upon hearing a pronoun during the first half of the story, children’s looks to the subject of the preceding clause increased. Adults, however, had the highest proportion of looks to the subject at the onset of the pronoun, suggesting they were able to use discourse cues to predict that the subject would be referred to. Hearing a name during the second half of the story had no influence on looks to the target referent, for both children and adults. Upon hearing a pronoun during the second half of the story, children’s looks to the subject of the preceding clause increased, however, this took much longer in the time course compared to pronouns that occurred during the first half of the story. Hearing a pronoun during the second half of the story had no influence on adults’ looks to the preceding subject. Furthermore, we found that children’s working memory (WM) capacity influenced their language mediated eye movements.
  The findings of the current thesis demonstrate that there is not a uniform mapping between linguistic input and eye movements within the visual scene. It is likely that individuals only direct their eye gaze towards entities in the visual scene under particular language processing circumstances. As such, these findings call into question whether or not the visual world paradigm is an effective tool for investigating language processing in naturalistic settings. Further research is needed in order to better understand the relationship between eye gaze and spoken language processing in continuous discourse.

• Subjects / Keywords

  • referential processing
  • pronoun resolution
  • eye tracking
  • visual world paradigm

• Graduation date

  Fall 2018

• Type of Item

  Thesis

• Degree

  Master of Science

• DOI

  https://doi.org/10.7939/R3M61C572

• License

  Permission is hereby granted to the University of Alberta Libraries to reproduce single copies of this thesis and to lend or sell such copies for private, scholarly or scientific research purposes only. Where the thesis is converted to, or otherwise made available in digital form, the University of Alberta will advise potential users of the thesis of these terms. The author reserves all other publication and other rights in association with the copyright in the thesis and, except as herein before provided, neither the thesis nor any substantial portion thereof may be printed or otherwise reproduced in any material form whatsoever without the author's prior written permission.