Speaker: Simone Rebora
Affiliation: University of Verona; University of Bielefeld
Title: How can I attribute this text? A comparison between stylometry and readers’ perception
Abstract (long version below): The paper discusses the results of an experiment which presents readers with a typical problem in stylometry: the attribution of an anonymous text given the writing samples of candidate authors. Three Harry Potter fanfictions (average length ~ 1,500 words) were used for the experiment. Out of 142 participants, 60.56% were able to correctly attribute the anonymous text, while 87.11% of 12,793 stylometric analyses proved successful. A comparison is established between the motivations provided by readers and the working logic of stylometric methods, highlighting similarities and differences.
Long abstract
In the context of “distant reading” (Moretti 2013), stylometry is currently one of the most advanced techniques, using statistical methods to automatically identify the “stylistic fingerprint” of a writer. Still, discussion is still open as to why it works so efficiently (see Kestemont 2014; Evert et al. 2017).
This paper aims at providing a contribution to this discussion, by setting its groundwork at the intersection between stylistics and reader response studies. Indeed, stylistics is one of the fields of research in humanities that has been most impacted by the success of stylometry (Herrmann, Schöch, and van Dalen-Oskam 2015). At the same time, as Whiteley and Canning recently stated, “stylistics is, and always has been, inherently and inescapably concerned with reader response” (Whiteley and Canning 2017).
The experiment was set up to emulate a typical authorship attribution task in stylometry: participants had to read two texts by two different authors and then attribute a third anonymous text to one of the two authors. Texts were downloaded from Archive of Our Own, a fanfiction platform already used for authorship attribution tasks (see Kestemont et al. 2018).
A fictional frame was devised to present the experiment as a possible real case scenario, with two Harry Potter fanfiction writers (Anne and Rose) proposing a challenge to the participant (see Figure 1).
Participants had to answer three control questions to verify if they actually read the stories, they had to vote for Anne or Rose as the author of the “Anonymous fanfic”, and were allowed to leave a comment to motivate their choice. Table 1 reveals the correct answer to the central question, being Rose the actual author of the “Anonymous fanfic”.
A total of 162 participants took part in the experiment, which was repeated in five rounds. Participants were both Bachelor’s students (two rounds, n = 56), Master’s students (two rounds, n = 73), PhD students and young researchers (one round, n = 33). As Table 2 shows, 60.56% of the participants gave the correct answer to the attribution question. Overall, it can thus be stated that the text was correctly attributed by a slight (but substantial) majority of the readers. It is also interesting to notice how the level of education does not seem to correlate with attribution accuracy. Percentages do not even increase substantially when considering just the 83 participants who left a comment. Only specific aspects highlighted in the comments (see Figure 2 for an overview) drive a correct attribution.
To get an overview of the efficiency of stylometric methods on this case study, a series of analyses was performed using the Stylo R package (Eder, Kestemont, and Rybicki 2016), by combining the eleven distance measures available in the package with all the possible lists of most frequent words (from one to 1,163 MFW). This resulted in a total of 12,793 different analyses. Overall, 87.11% of the analyses provided the correct attribution.
A deeper understanding of the phenomena that caused such a result is allowed by Figure 3, which shows the frequency of the ten MFW in the corpus. The grey bars represent the frequency of each word in each text, while the red dashed lines represent their mean frequency in the whole corpus. When comparing the ten words, it becomes evident that in the majority of the cases there is a clear pattern connecting Rose’s text with the Anonymous text. See for example the word “he”: both Rose’s and the Anonymous fanfiction use it less than the average, while Anne’s fanfiction uses it more than the average. Stylometric distances like Burrows’s Delta automatize such a reasoning in a purely mathematical manner, by repeating this operation for all words in a text.
In Figure 3, the case of the word “he” is mirrored by the case of the word “I”, which is clearly overused in both Rose’s and the Anonymous fanfiction. It seems reasonable to suppose that such a correspondence was caused by a comparatively higher use of dialogues in the two texts: and while readers caught this overuse through a structural analysis of the text (see the “dialogues” category in Figure 2), stylometry catches it by just looking at the frequency of single words.
In terms of modelling (Flanders and Jannidis 2019), it can thus be stated that, by representing style as similarity in patterns of word frequencies, stylometry also measures a wide range of phenomena that are commonly identified by readers, but it does it indirectly, by looking at how they affect the use of words.