Speaker: Federico Pianzola, Srishti Sharma, Frank Tsiwah
Affiliation: University of Groningen, Netherlands
Title: A Computational Analysis Linking the Emotion Arcs of Books and Reader Response
Abstract (long version below): We explore the association between emotions expressed in fictional stories and emotions experienced by their readers. We use 428 English books from 9 different genres (mostly 19-20th Century) and their corresponding reviews from Goodreads. We use sentiment analysis, calculating two different types of values: book sentiment (including the sentiment story arc of each book) and review sentiment. We test our hypotheses using both linear regression models and generalized additive models (GAM) to analyze the association of the sentiment values of books and reviews. Our results show that book sentiment can predict review sentiment, with the ending part of the book being our best predictor, but we also observed different non-linear patterns dependent on genre.
Preregistration
Sentiment classifier fine-tuned on fiction and book reviews
Long abstract
Measuring the long-term effects of reading fiction is a complex endeavour that has outcomes whose reliability is controversial (Kidd and Castano 2013; 2019; Samur, Tops, and Koole 2018). We can be more confident about the impact of fiction in the short term, namely with respect to the various subjective-phenomenal experiences undergone by readers while they engage with captivating fictional characters and their stories. Psychological states like ânarrative absorptionâ (Hakemulder et al. 2017), âreading flowâ (Thissen, Menninghaus, and Schlotz 2018), and empathy with characters (Burke et al. 2016) are being measured and associated with discursive features of literary texts with increasing accuracy, and improvements of both theories and methods are constantly achieved. Recently, digital media and computational methods have offered new â and more ecologically valid â ways to look at how readers respond to fiction. Book reviews posted on websites (e.g. Goodreads) or comments written in the margins of digital texts read on mobile apps (e.g. Wattpad) enable a new kind of research â based on statistically reliable and generalizable evidence (cf. Piper 2020) â that can investigate how thousands of readers spontaneously share their thoughts and emotions about books and stories (Rehberg Sedo 2011; Rebora et al. 2019; Bourrier and Thelwall 2020; Pianzola 2021).
In this study, we aim at leveraging the availability of a huge quantity of book reviews (from Goodreads) to evaluate the emotional impact of fiction on readers. Namely, we will assess whether the emotion represented and expressed in fiction can predict the emotions perceived and expressed by readers in their reviews. For instance, do tragic stories prompt more negative wordings in reviews? The underlying hypothesis is that emotions in stories affect readers and their engagement with and evaluation of the story.
The plausibility of a direct link between textual emotions and reader response is supported by research that looked at the close relationship between a storyâs sentiment on the level of paragraph and the sentiment found in readersâ comments in the margin of such paragraphs (Pianzola, Rebora, and Lauer 2020). Similarly, Jacobs and Kinder (2019) found that computationally calculated affect scores of words can predict the human liking of small text chunks (~105 words) of a short story. In brief, existing research showed that there is a link between emotions in texts and the emotional response of readers, but we still do not know the details of this kind of relationship in the case of book reviews.
Hypotheses
Based on evidence presented in existing research, we formulated the following hypotheses (H) regarding the relationship between emotional patterns in books and their corresponding reviews:
H1. The sentiment of a book has a positive linear relation with the sentiment of its reviews. That is, if we take two books, the one with a more positive story sentiment will also have a more positive reviewsâ sentiment.
H2. Romance is the genre with the strongest association between story sentiment and reviewsâ sentiment.
H3. The amplitude of sentiment change in the ending part of a story has a stronger linear relation with the reviewsâ sentiment than that existing between the overall story sentiment and reviewsâ sentiment (H1). That is, the plot influences the readersâ appraisal of a book: i.e. stories ending with a growing positive sentiment elicit reviews with a more positive sentiment than stories ending in a decreasing sentiment.
Data
We use 428 English books from 9 different genres (mostly 19-20th Century) and their corresponding reviews from Goodreads.
To train a Deep Learning classifier (RoBERTa), we used:
- the Dataset for Emotions of Narrative Sequences (DENS) (Liu, Osama, and de Andrade 2019), which contains a combination of 9,710 manually annotated paragraphs (between 40 and 200 tokens long) taken from English literature (Project Gutenberg) and modern narratives by amateur writers (Wattpad, post 2006).
- a dataset of 300 manually annotated (at sentence-level) book reviews from Amazon and LibraryThing (Ălvarez-LĂłpez et al. 2017).
Methods
We preregistered our analysis specifying a linear regression model and the covariates to be included in the model.
Additionally, we used a Generalized Additive Model (GAM) for a more fine-grained analysis of the potential interaction between genre and the unfolding of sentiment throughout the book, and at what point(s) in the book this potential interaction is significant.
Results
- H1. Accepted: The relation between the sentiment of a book and the sentiment of its reviews can be approximated by a linear fit.
- H2. Rejected: We didnât find any genre with an association between story sentiment and reviewsâ sentiment stronger than that of other genres.
- H3. Unclear: The amplitude of sentiment change in the ending part of a story has a relation with the reviewsâ sentiment similar to that existing between the overall story sentiment and reviewsâ sentiment (H1).
There is a significant association between book sentiment and reviewsâ sentiment, meaning that the emotions portrayed in a book have an impact on how people evaluate the book. But this association follows different non-linear patterns for different genres.
Fig. 1. Pairwise comparison between book sentiment of various genres.
Fig. 2. Pairwise comparison between reviewsâ sentiment of various genres.
Fig. 3. Example of association between sentiment in different parts of books and reviewsâ sentiment.
There is a significant difference (marked in red on the x-axis) between the association of positive and negative book sentences with reviews for romance. A curve above the horizontal line intersecting zero means that positive sentences are positively associated with review sentiment significantly more than negative sentences.
In brief, in books with more positive reviews, the romance goes on throughout the whole book and thereâs a happy ending, but a negative situation often precedes it.