Speaker: Bien Klomberg
Affiliation: Tilburg University
Title: Running through the Who, Where, and When: A Cross-cultural Analysis of Situational Changes in Comics
Abstract (long version below): Understanding visual narratives requires readers to track aspects of time, spatial location, and characters across sequences. This work investigated these dimensions as situational “runs” – uninterrupted sequences of temporal, spatial, and character continuity – in a corpus of 300+ annotated comics from the United States, Europe, and Asia. We compared these runs’ proportion and average lengths. Our results showed semantic information changed frequently, with a distinct organization of time, space, and character information, and with cross-cultural patterns. Altogether, these findings raise questions about comprehenders’ processing strategies across cultures and how general frameworks of visual narrative comprehension account for situational (dis)continuity in storytelling.
Long abstract
Understanding visual narrative sequences, like in comics, requires readers to construct a mental model of the situation depicted in the sequence (Cohn, 2019, 2020b; Dijk & Kintsch, 1983; Huff et al., 2020; Loschky et al., 2020). To build such a mental model, readers track situational dimensions such as time, spatial location, and characters across a sequence (Dijk & Kintsch, 1983; Gernsbacher, 1997; Zwaan & Radvansky, 1998). Changes across these situational dimensions necessitate updating processes, with more discontinuous information sponsoring greater updating (Huff et al., 2014; Hutson et al., 2018; Manfredi et al., 2017). This flow of information across panels appears to follow patterns specific to culture, as previous work has found situational changes across adjacent panels to differ cross-culturally (Cohn, 2020a; McCloud, 1993). However, only few works have yet examined such situational dimensions across extended sequences of panels, rather than across moments of transition. We therefore question how meaningful information shifts across sequencing in visual narratives. This sequencing is expected to reflect prior cross-cultural findings, with Asian comics showing more temporal and character discontinuity than European and American comics, and comparable spatial continuity.
Thus, this study investigated situational “runs” – uninterrupted sequences of the situational dimensions of time, space, and characters, in a corpus of 300+ annotated comics from the United States, Europe, and Asia. The data originated from the Visual Language Research Corpus, VLRC (see http://www.visual languagelab.com/vlrc), which was built up over a five-year period across various subprojects. Books were collected through convenience sampling, including public domain resources and donations from comic companies. Within this corpus, our selected data consisted of 342 books annotated by 12 coders. These books were all annotated for whether panels repeated the same situational information as their previous panel or not. For time, annotators coded whether there was a determinable change in temporal states (a time change, where time appeared to pass) or not. For spatial locations and characters, annotators coded whether the panel showed the same location/character(s) as its previous panel or whether the location or character(s) changed to some extent. As storytelling typically assumes time to progress, time changes were a feature of continuity, while changes in characters and spatial locations were considered discontinuities. We then recorded runs within these three dimensions, which refers to strings of continuous information or of a continuous progression of time. Furthermore, since situational dimensions may also change simultaneously, we considered two interactions: a completely continuous run (with time changes but no spatial/character changes) and a completely discontinuous run (with spatial/character changes, but no time changes). In addition to the three individual categories of time, spatial location, and characters, this resulted in five types of runs total.
We compared these runs’ proportions (how many panels of a book in total were part of a run) and their average lengths (how long continuity persisted in an uninterrupted string of panels). Overall, we found that across books, semantic information changed frequently, and run length correlated with runs’ proportion. Moreover, analysis of the individual situational dimensions suggested a distinct organization for time, spatial location, and characters: Temporal continuities persisted the longest, and both time and spatial location runs involved many panels per book on average, while character runs were the shortest and involved the least number of panels per book. This may imply that character information changes within persistent spaces, and that spatial continuities embed within even longer temporal continuities. These relative manifestations of situational dimensions could potentially inform expectations of readers, who may encode such patterns and account for them during processing. For instance, frequent temporal discontinuous could be more challenging for processing than frequent discontinuities in either spatial location or characters.
Furthermore, cross-cultural patterns arose, with American and European comics using more completely continuous runs than Asian comics. American and European comics also used more and longer temporal and character continuities than Asian comics, while Asian comics used more spatial continuity than the Western narratives. These findings largely align with prior work on situational changes across adjacent panels (Cohn, 2020a; McCloud, 1993) and with work on the attentional framing of panels (Cohn et al., 2012), referring to the amount of information given in a panel.
These findings raise questions about comprehenders’ processing strategies across cultures and how general frameworks of visual narrative comprehension (Cohn, 2020b; Loschky et al., 2020) account for variations in situational (dis)continuity. As these frameworks describe, more incongruous information is more effortful to integrate in a mental representation, so the more discontinuous Asian narratives may be experienced as more demanding to process. However, readers of these stories may potentially habituate to these structured patterns, which could lessen the updating costs associated with discontinuities (see Cohn and Kutas (2015)). These cross-cultural preferences could even interact with the global distinction found for temporal, spatial, and character (dis)continuity, where character discontinuities appear the least demanding, such that these costs may be preferred over the costs of temporal or spatial discontinuities. Future psychological work could investigate how readership of different comics may affect the processing of visual narrative sequences. Overall, if readers and/or authors indeed encode (cross-cultural) situational patterns, our results pose relevant challenges for theories of discourse as to what structures of information would be stored, and how.