Speaker: Marloes Mak @Marloes_Mak
Affiliation: Radboud University
Title: Different Kinds of Simulation During Literary Reading: Insights from a Combined fMRI and Eye Tracking Study
Abstract (long version below): In a combined eye tracking and fMRI study, we investigated the existence of a common neural locus for different kinds of simulation and investigated individual differences during reading, using a fixation-related analysis for our fMRI data. We found a variety of brain areas activated by simulation-eliciting content, both modality-specific brain areas and a general simulation area. Individual variation in percent signal change in activated areas was related to measures of story appreciation as well as personal characteristics (i.e., transportability, perspective taking). Taken together, these findings suggest that mental simulation is supported by both domain-specific processes, and by higher-order language processing.
Long abstract
Mental simulation is an important aspect of narrative reading. Research has shown that there is evidence for a difference between three kinds of simulation in language processing: perceptual simulation, motor simulation, and the simulation of introspective states (more commonly called “mentalizing”. In a previous study, we found that gaze durations are differentially impacted by these different kinds of mental simulation. Motor simulation, perceptual simulation, and mentalizing as elicited by literary short stories influenced eye movements in distinguishable ways (Mak & Willems, 2019).
In the current combined eye tracking and fMRI study, we investigated the existence of a common neural locus for these different kinds of simulation, using a fixation-related analysis for our fMRI data. We additionally investigated whether individual differences during reading, as indexed by eye movements, are reflected in domain-specific activations in the brain.
We hypothesized that a shared neural locus could be in an area involved in the general process of constructing a coherent representation of the content of narratives, or situation model building (e.g., Martín-Loeches, Casado, Hernández-Tamames, & Álvarez-Linera, 2008; see also Smirnov et al., 2014). Candidate areas for such a process would be the posterior cingulate cortex, anterior cingulate cortex, angular gyrus, precuneus, insula, dorsolateral and medial prefrontal cortex, and the superior frontal gyrus. Another possibility would be that different kinds of simulation are associated with domain-specific processes. Motor simulation would then activate motor areas, perceptual simulation would activate areas involved in the processing of perceptual information, and mentalizing would activate the mentalizing-network.
We studied this in an experiment where 40 participants read two Dutch literary short stories while they were in an MRI scanner and simultaneously had their eye-movements tracked. This allowed us to align the eye-tracking data temporally with the neuroimaging data, within participants. We were interested in the responses of participants at the word level: we measured fixation duration and brain activation as a response to the number of times the fixated words were underlined for being part of motor descriptions, perceptual descriptions, and mental even descriptions. The scoring (underlinings) of motor, perceptual, and mental event content was acquired in a separate pretest with different participants. After scanning, questionnaires regarding reading habits in daily life (directly and indirectly measured), and personal characteristics questionnaires regarding empathy and transportability were administered.
Our eye-tracking data showed comparable results to those found in our earlier eye-tracking study (Mak & Willems, 2019). Motor descriptions were associated with shorter gaze durations (i.e., faster reading), whereas perceptual and mental event descriptions were associated with longer gaze durations on the next word (i.e., slower reading). We found a variety of brain areas activated by simulation-eliciting content, both modality-specific brain areas and a general simulation area. The general simulation area was in the left anterior supramarginal gyrus, and area that is both anatomically and functionally adjacent to the angular gyrus, and has been found to be involved in the integration and regulation of a multitude of neural processes. We propose that it is likely that this area is involved in referential indexing and situation model building.
Individual variation in percent signal change in activated areas was related to measures of story appreciation as well as personal characteristics. For all three kinds of descriptions, we found areas in which individual differences in the percent signal change as a response to these descriptions was associated with measures of story appreciation (story-specific effects) as well as personal characteristics (trait-based measures). Percent signal change was not associated with story world absorption (state-based measures). Apparently, state-based individual differences do not explain the effects of reading on the neural level, whereas trait-based individual differences do. This is in line with results found in multiple previous studies, where trait-based individual differences (as opposed to state-based individual differences) were also more strongly associated with simulation (Faber, Mak, & Willems, 2020; Hartung, Hagoort, & Willems, 2017; Hartung, Wang, Mak, Willems, & Chatterjee, 2021; Mak, De Vries, & Willems, 2020; Van den Hoven, Hartung, Burke, & Willems, 2016). Seemingly, simulation is more strongly associated with stable characteristics than with reading experiences such as absorption and appreciation.
Taken together, these findings suggest that mental simulation is supported by both domain-specific processes grounded in previous experiences, and by the neural mechanisms that underlie higher-order language processing (e.g., situation model building, event indexing, integration).
References
Faber, M., Mak, M., & Willems, R. (2020). Word skipping as an indicator of individual reading style during literary reading. Journal of Eye Movement Research, 13(3), 1–9. https://doi.org/10.16910/jemr.13.3.2
Hartung, F., Hagoort, P., & Willems, R. M. (2017). Readers select a comprehension mode independent of pronoun: Evidence from fMRI during narrative comprehension. Brain and Language, 170, 29–38. Redirecting
Hartung, F., Wang, Y., Mak, M., Willems, R., & Chatterjee, A. (2021). Aesthetic appraisals of literary style and emotional intensity in narrative engagement are neurally dissociable. Communications Biology, 4(1), 1401. Aesthetic appraisals of literary style and emotional intensity in narrative engagement are neurally dissociable | Communications Biology
Mak, M., De Vries, C., & Willems, R. M. (2020). The Influence of Mental Imagery Instructions and Personality Characteristics on Reading Experiences. Collabra: Psychology, 6(1), 43. The Influence of Mental Imagery Instructions and Personality Characteristics on Reading Experiences | Collabra: Psychology | University of California Press
Mak, M., & Willems, R. M. (2019). Mental simulation during literary reading: Individual differences revealed with eye-tracking. Language, Cognition and Neuroscience, 34(4), 511–535. https://doi.org/10.1080/23273798.2018.1552007
Martín-Loeches, M., Casado, P., Hernández-Tamames, J. A., & Álvarez-Linera, J. (2008). Brain activation in discourse comprehension: A 3t fMRI study. NeuroImage, 41, 614–622. Redirecting
Smirnov, D., Glerean, E., Lahnakoski, J. M., Salmi, J., Jääskeläinen, I. P., Sams, M., & Nummenmaa, L. (2014). Fronto-parietal network supports context-dependent speech comprehension. Neuropsychologia, 63, 293–303. Redirecting
Van den Hoven, E., Hartung, F., Burke, M., & Willems, R. M. (2016). Individual Differences in Sensitivity to Style During Literary Reading: Insights from Eye-Tracking. Collabra, 2(1), 25. Individual Differences in Sensitivity to Style During Literary Reading: Insights from Eye-Tracking | Collabra: Psychology | University of California Press