Online search trends and word-related emotional response during COVID-19 lockdown in Italy: a cross-sectional online study

Selection of experimental stimuli

We used Google Trends (https://trends.google.com/trends/) to assess internet activity related to the COVID-19 epidemic in Italy in the first four months of 2019 and 2020. The period before Italy’s first confirmed COVID-19 patient (February 21st, 2020) was included as a baseline to assess the COVID-related change in the temporal pattern of online searches. Indeed, Google Trends determines the normalized proportion of searches for user-specified terms among all searches performed using Google for a given location and time period, expressed as the relative search volume (RSV) with a datapoint for each day, scaled on a [0, 100] range where 100 is the maximum search interest for the time and location selected. Moreover, data from 2019 were used to control for potential unspecific seasonal trends or idiosyncratic temporal patterns in RSV data (for example, the word “freedom” -libertà in Italian- shows a peak on April 25th, the Liberation Day in Italy).

The following terms were used: “coronavirus”, “COVID”, “COVID-19”, and “virus”. We also extracted the RSV for the 1121 words included in the Italian adaptation of the English affective norms (ANEW; Montefinese et al., 2014) by using the gtrends R package (Massicotte & Eddelbuettel, 2016) for R (R Core Team, 2019). RSV data for one word (mildew) were not available. We retrieved RSV data from January 1st to April 27th (most current data available at the time of data retrieval), for both 2019 and 2020 years.

The experimental stimuli were selected among the Italian ANEW words by assessing to what degree the temporal dynamics in their search trends was specifically related to that of the search trend for the COVID-19 terms. We aimed to identify the Italian ANEW words that consistently showed the greater change in internet activity due to the COVID-19 epidemic, while controlling for unspecific RSV trends. This was done by taking four different analytical approaches based on a multiverse analysis (Steegen et al., 2016). 

First, for each year, a COVID-related RSV time series (COVID-RSV) was computed by averaging the RSV time series for the four COVID-related terms. Pearson’s correlation coefficients (r) were then computed between the COVID-RSV and those for the Italian ANEW words (ANEW-RSV). These r values thus reflect the strength of the association between the COVID-RSV and each ANEW-RSVs for both the 2019 and 2020. Next, we compared the r values for 2020 and 2019 by performing Steiger’s Z tests for non-overlapping correlations based on dependent groups (Steiger, 1980), thus obtaining a Z value Z_Pears for each ANEW word. Second, we computed differential RSV time series by subtracting the COVID-RSV and ANEW-RSVs for 2019 from those for 2020 and computed their Pearson’s correlation coefficients (r_diff). Both Z and r_diff values reflect the 2020-specific change in the strength of the association between the COVID-RSV and each ANEW-RSVs.

The same procedure described in the previous paragraph was performed after rank-transformation of all original RSV data to compute non-parametric Spearman’s correlation coefficients ρ and ρ_diff, as well as the Z_Spear value from the Steiger’s tests comparing ρ values for 2020 and 2019. This was done to control for both non-normality of our data and potential outlier observations. We thus obtained four differential correlation measures (r_diff, ρ_diff, Z _Pears, and Z_Spear) reflecting the (signed) degree of the specific impact of the COVID-related interest on the search trends for each ANEW word.

Based on these correlational measures, we selected three groups of stimuli, each composed by 20 words, as described below. This number of stimuli was the largest that can be reliably rated by each participant during a single online session in a reasonable amount of time (based on pilot testing and Montefinese et al.’s normative study (2014), which used 56–57 stimuli for each participant), ensuring the reliability of the ratings and yielding the maximum possible power. The first group (REL+) consisted in the words showing the largest positive relation between their search trends and the search trend for the COVID-related terms. By contrast, the second group (REL-) consisted in the words showing the largest negative relation between their search trends and the search trend for the COVID-related terms. In other words, the COVID-19 epidemic in Italy, and the consequent increase in interest for the COVID-related terms, was related to a similar increase of interest for the REL+ words and a decrease of interest for the REL- words. The third group (UNREL) consisted in the words for which the search trend was unrelated to the search trend for the COVID-related terms.

The REL+ and REL- words were selected as those consistently showing, respectively, the highest and the lowest r_diff, ρ_diff, Z _Pears, and Z_Spear values. Specifically, we first selected the words that were in the top (or bottom, respectively) 2.5% of the distribution for at least three out of the four differential correlation values, and then selected the words with the largest differential correlation values that were in the top (or bottom, respectively) 2.5% of the distribution for at least two out of the four differential correlation values. The UNREL words were selected as those showing the smallest differential correlation values. For all the three groups, the selection was limited to nouns and verbs.

Figure 1 shows the differential RSV time series for the COVID-related terms and one exemplar stimuli for each of the REL+ (fever, febbre in Italian), REL- (hotel), and UNREL ([to] disturb, disturbare in Italian) groups. This figure illustrates the clear COVID-related increase of online searches for the REL+ word “fever”, likely due to health concerns, as well as the clear COVID-related decrease of online searches for the REL- word “hotel”, likely due to a more limited mobility suddenly imposed to Italians during the COVID-19 lockdown.

The selected experimental stimuli are available at Open Science Framework (see https://osf.io/2mc3k/, Table S1).

Procedure

An online survey was conducted using Google Forms (https://www.google.com/forms/about/) to collect affective ratings during the lockdown caused by the COVID-19 epidemic in Italy. The first section of the form consisted of the informed consent, including a basic description of the study, followed by a section asking participants to specify their gender, age, and education level. The next sections of the form consisted in the Positive and Negative Affect Schedule (PANAS, Terraciano, McCrae & Costa, 2003) and Fear of COVID-19 Scale (FCV-19S, Ahorsu et al., 2020) questionnaires to evaluate participants’ positive and negative affective state (assessed, respectively, with the PANAS+ and PANAS- subscales of the PANAS) and fear of COVID-19, which we expected to modulate participants’ affective ratings. Finally, in the last section of the form participants were asked to provide their affective ratings for the 60 experimental stimuli, which were presented in a random order. Specifically, participants were instructed to rate how they felt when reading each word along the three affective dimensions of valence, arousal, and dominance by using the 9-point self-assessment manikin (Lang, 1980). The format and instructions for the affective rating task were the same as those used in our previous work (Montefinese et al., 2014).

Data were collected in the period from May 4th to May 17th, 2020, the last day of full lockdown in Italy, from 71 adult native Italian speakers (56 females and 13 males; mean (SD) age = 26.2 (7.9) years; mean (SD) education = 15.3 (3.2) years). There were no other specific eligibility criteria. Participants consisted of a convenience sample recruited via online advertisements through social networks or identified via researchers’ personal networks. It is important to note that in the present study affective ratings were provided for each word by twice as many participants, on average, as compared not only to the Italian ANEW norms (Montefinese et al., 2014), but also to affective norms in general (e.g., Warriner, Kuperman & Brysbaert, 2013), so assuring an adequate reliability and generalizability of our affective ratings. It is also important to note that most of our research questions involved by-items statistical analyses, so the number of participants did not directly impact on the statistical power of our analysis.

A first sensitivity power analysis was conducted in G*Power for a mixed ANOVA on current and normative ratings with three groups of 20 words, assuming a correlation between repeated measures of .80 (as estimated conservatively from our previous study, Montefinese et al., 2014). This analysis revealed that our sample size (60 words) was large enough to detect a small effect size (Cohen’s d = .12, corresponding to η²_p = .014) with a power of .80. We also used the method introduced by Westfall and colleagues (2014) to perform a sensitivity power analysis for a stimuli-within-condition linear mixed-effects model, assuming participants, stimuli, and residual variance partitioning coefficients of .1, .15, and .75, respectively (as estimated conservatively from some recent unpublished studies with a similar design from our research group). This analysis revealed that our sample size (71 participants and 60 words) was large enough to detect a small-medium effect size (d = .30) with a power of .80.

The procedure used in the study is in accordance with the ethical standards of the 2013 Declaration of Helsinki for human studies of the World Medical Association. The project has been approved by the Ethical Committee for the Psychological Research of the University of Padova (approved protocol reference number: 3563).

Data analysis

We performed a series of analysis to investigate (1) the relation between the lexical and affective variables for the words we used and the COVID-dependent changes in their online searches; (2) the reliability of the present affective ratings; (3) the impact of the COVID-19 lockdown in Italy on affective ratings; (4) the effect of participants’ emotional profile on affective ratings.

A first set of analyses was conducted to investigate whether the magnitude of the specific impact of the COVID-related interest on the search trends for Italian ANEW words could be explained by their lexical and affective variables taken from Italian ANEW norms (Montefinese et al., 2014). To this aim, we first computed the zero-order parametric and non-parametric correlations between the four differential correlation measures (r_diff, ρ_diff, Z_Pears, and Z_Spear), on the one side, and the valence, arousal, dominance, familiarity, concreteness, and word frequency, on the other side. The differential correlation values were first transformed to improve the normality of their distribution by performing a natural log-transformation on their absolute values. We also performed stepwise multiple regression analyses with each transformed differential correlation value as the dependent variable and the lexical and affective variables as predictors. We used a tolerance cutoff of .6 to minimize multicollinearity and maximize precision of the regression parameter estimates.

We also assessed the reliability of the present affective ratings by correlating them with those of the Italian ANEW norms (Montefinese et al., 2014) and by computing split-half correlations corrected with the Spearman-Brown formula after 10,000 randomizations.

We then investigated the impact of the lockdown imposed by the COVID-19 epidemic on affective ratings. To this aim, we compared the affective ratings collected in the present sample with those collected in the normative sample for the same stimuli. First, for each affective dimension, we performed a two-tailed paired t-test contrasting the mean ratings from the present and normative samples; we also performed two-tailed Welch’s t-tests contrasting the individual ratings for each word, followed by an internal meta-analysis to estimate combined effect sizes. Moreover, we investigated whether the lockdown-dependent differences in affective ratings were modulated by the specific impact of the COVID-related interest on the search trends for our stimuli. For each affective dimension, we performed a by-items Welch’s ANOVA on the raw difference in the affective ratings between the present and the normative samples, with Stimulus Type (REL+, REL-, UNREL) as a between-items factor. Post-hoc comparisons were performed using Welch’s t tests.

Lastly, we investigated whether participants’ affective state and fear of COVID-19 modulated their affective ratings. To this aim, for each affective dimension we performed three linear mixed-effects model (LMM) analyses with the raw difference in the affective ratings as the dependent variable, three parameters for (1) the fixed effects of Stimulus Type, (2) either the PANAS-, PANAS+, or FCV-19S (centered), and (3) their interaction, and by-subjects and by-items random intercepts.

Impact of lexical and affective variables on COVID-dependent changes of ANEW search trends

All the correlations were significant (p < .001), but (at best) moderate in size (for all the results, see Table S3, https://osf.io/2mc3k).

The final model for the multiple regression analysis included four predictors (F (4, 1108) = 52.6, p < .001, R ² = 15.95%; seven cases were not included due to missing word frequency data; see Supplemental material, https://osf.io/8hpek). Results showed that the specific impact of the COVID-related interest on the search trends was greater for the Italian ANEW words with higher word frequency (b = 0.067, 95% confidence interval (95% CI) = [0.050–0.085]; t = 7.46; p < .001), concreteness (b = 0.095, 95% CI = [0.073–0.012]; t = 8.48; p < .001), and valence (b = 0.041, 95% CI = [0.023–0.059]; t = 4.53; p < .001), as well as for the Italian ANEW words with lower arousal (b =  − 0.054, 95% CI = [−0.095, −0.012]; t =  − 2.52; p = .012).

Reliability analysis on affective ratings

The reliability analysis showed very high correlations between the Italian ANEW norms (Montefinese et al., 2014) and the affective ratings collected in the present sample, especially for the valence (.98, .81, and .79 for valence, arousal, and dominance, respectively), and the median split-half correlations were even higher (.99, .93, and .97, for valence, arousal, and dominance, respectively; range = [.97, .99], [.74, .97], and [.93, .99]).

Lockdown impact on affective ratings

The analyses revealed that the lockdown imposed by the COVID-19 epidemic affected participants’ affective ratings. Indeed, as compared to the normative sample, our participants rated the experimental stimuli with lower valence (mean difference M_diff = −0.625, 95% CI = [−0.746, −0.503]; t (59) = −10.27; p < .001; d =  − 1.325, 95% CI = [−1.670, −0.974]), arousal (M_diff = −0.220, 95% CI = [−0.363, −0.077]; t (59) = −3.08; p = .003; d =  − 0.397, 95% CI = [−0.659, −0.133]), and dominance (M_diff = −0.635, 95% CI = [−0.808, −0.461]; t (59) = −7.32; p < .001; d =  − 0.945, 95% CI = [−1.247, −0.638]). These results were confirmed by the Welch’s t-tests performed on each word, which revealed significant differences for 26, 7, and 27 words (corresponding to 43.33%, 11.67%, and 45% of the words) for valence, arousal and dominance, respectively (https://osf.io/2mc3k, Table S4; see also Fig. S1), as also suggested by the results of the internal meta-analysis. Most of these significant differences reflected lower affective ratings in the current sample, with an apparent difference in their distribution across the three types of stimuli (REL+, REL-, UNREL; see https://osf.io/2mc3k, Fig. S1). Indeed, for the valence, the combined effect sizes d for REL+, REL-, and UNREL words were, respectively, −0.340 (95% CI = [−0.403, −0.277]), −0.354 (95% CI = [−0.471, −0.237]), and −0.178 (95% CI = [−0.265, −0.092]), with a significant difference across Stimulus Types (Q*(2) = 7.93, p = .019). For the arousal, the combined effect sizes d for REL+, REL-, and UNREL words were, respectively, .001 (95% CI = [−0.068, .069]), −0.063 (95% CI = [−0.159, 0.034]), and −0.176 (95% CI = [−0.248, −0.105]), with a significant difference across Stimulus Types (Q*(2) = 7.37, p = .025). For the dominance, the combined effect sizes d for REL+, REL-, and UNREL words were, respectively, −0.333 (95% CI = [−0.445, −0.221]), −0.205 (95% CI = [−0.330, −0.079]), and −0.175 (95% CI = [−0.267, −0.083]), with no significant difference across Stimulus Types (Q*(2) = 4.82, p = .090).

These results were confirmed by the Welch’s ANOVAs (see Table 2; see also https://osf.io/prx4s). Indeed, the COVID-related decrease in valence was significantly different across Stimulus Types, with a smaller decrease for UNREL words as compared to both REL- (t (36.7) = −2.71; p = .010; d =  − 0.858, 95% CI = [−1.525, −0.172]) and REL+ (t (34.8) = −2.30; p = .028; d =  − 0.723, 95% CI = [−1.378, −0.057]) ones, which in turn did not differ between each other (t (31.2) = 0.23; p = .818; d = .073, 95% CI = [−0.548, 0.693]). Moreover, the COVID-related decrease in arousal was significantly different across Stimulus Types, but this time with a significantly larger decrease for UNREL words as compared to REL- (t (37.6) = 3.09; p = .004; d = 0.977, 95% CI = [0.274, 1.166]), but not REL+ words (t (33.6) = 1.61; p = .116; d = 0.510, 95% CI = [−0.137, 1.144]) ones, and no significant differences between REL- and REL+ words (t (35.5) = 0.96; p = .342; d = 0.305, 95% CI = [−0.327, 0.928]). Finally, the COVID-related decrease in dominance did not significantly differ across Stimulus Types, with similar decreases (all —t—s < 1.72; p s >.096; —d—s < 0.541).

To sum up the results of these analysis, they provided converging evidence revealing COVID-dependent changes of affective ratings, with lower valence especially for REL- and REL+ words, lower arousal especially for UNREL words, and lower dominance regardless of the word group.

Effect of participants’ emotional profile on affective ratings

The results of LMM analyses for the three affective dimensions are shown in Table 3 (see also Table S5, https://osf.io/2mc3k).

For the valence, the LMM analyses (see https://osf.io/hbnuc) confirmed that the decrease in valence ratings was significantly different across Stimulus Types and revealed that this effect was modulated by participants’ PANAS- and FCV-19S scores. Indeed, the decrease in valence ratings was larger for participants with higher PANAS- scores (F(1, 69) = 17.51, p < .001) and this effect was significantly modulated by Stimulus Type (F(2, 4128) = 6.42, p = .002): the impact of PANAS- on the decrease in valence was smaller for both REL- and REL+ words, for which the decrease in valence was evident also for participants’ with lower PANAS- scores; by contrast, the decrease in valence for UNREL words was evident in participants with higher PANAS- scores only (Fig. 2A). A similar pattern was observed for the model assessing the impact of participants’ FCV-19S scores, with a significant two-way interaction (F(2, 4128) = 15.06, p < .001) as shown in Fig. 2C.

For the arousal, the LMM analyses (see https://osf.io/j4kym) revealed that participants’ ratings were positively related to both their PANAS- (F(1, 69) = 12.32, p < .001; Fig. 2D) and FCV-19S (F(1, 69) = 5.40, p = .023) scores, and this latter effect was significantly modulated by Stimulus Type (F(2, 4128) = 3.20, p = .041): the impact of FCV-19S on arousal ratings was larger for both REL- and REL+ words, for which participants with higher FCV-19S scores tended to show an increase in arousal ratings, as compared to UNREL ones (Fig. 2F). By contrast, participants’ arousal ratings were negatively related to their PANAS+ scores (F(1, 69) = 4.40, p = .040), especially for the REL- words as compared to the UNREL ones (F(2, 4128) = 3.86, p = .021; Fig. 2E).

For the dominance, the LMM analyses (see https://osf.io/5w7pc) revealed that participants’ ratings were related positively to their PANAS+ scores (F(1, 69) = 10.98, p < .001; Fig. 2H), but negatively to both their FCV-19S (F(1, 69) = 7.07, p = .010; Fig. 2I) and PANAS- (F(1, 69) = 10.72, p = .002) scores, and this latter effect was significantly modulated by Stimulus Types (F(2, 4128) = 7.45, p < .001): the impact of PANAS- on the decrease in dominance was smaller for both REL- and REL+ words, for which the decrease in dominance was evident also for participants’ with lower PANAS- scores; by contrast, the decrease in dominance for UNREL words was evident in participants with higher PANAS- scores only (Figs. 2G–2I). A similar pattern was observed for the model assessing the impact of participants’ FCV-19S scores, as shown in Fig. 2I, but the two-way interaction did not reach the significance level (F(2, 4128) = 2.26, p = .105).