The role of orthographic neighbourhood effects in lateralized lexical decision: a replication study and meta-analysis

Participants

A priori power analysis using pilot data and the SimR package (version 1.0.5; Green & MacLeod, 2016) in R (R Development Core Team, 2016) suggested that 30 participants would provide sufficient power (power >80%) to detect the critical N by visual field interaction in reaction time. The critical effect size used in these simulations was 15 ms, which is more conservative than the effect size reported by Perea, Acha & Fraga (2008). To reach this sample size, 45 native English speakers were recruited through Prolific Academic (https://www.prolific.co/); a web-based platform for recruiting and screening paid research participants according to experimenter-specified criteria. Participants were right-handed, had normal or corrected-to-normal vision, no significant hearing loss, history of neurological disease, head injury, epilepsy, or diagnosis of language and/or reading impairment. Thirteen participants were removed from the study as their score on the Edinburgh Handedness Inventory was below 80 (Oldfield, 1971). This cut-off was chosen to match Perea, Acha & Fraga (2008). Two further participants were removed from the study as they scored below chance when completing the lexical decision task in at least one of the three visual field conditions. The remaining 30 participants (18 female) had a mean age of 29.0 years (SD_age = 8.12 years).

This study was approved by the University of Oxford’s Medical Sciences Inter-divisional Research Ethics Committee (reference number: R67385/RRE001) and conducted in accordance with the principles of the Declaration of Helsinki. Participants gave informed written consent and received compensation at a rate of £10/hour for their participation.

Materials

Stimuli were selected using the same restrictions as Perea, Acha & Fraga (2008). We selected 120 word and 120 non-word targets. All targets had two syllables and were 5 letters in length. For word targets, 60 had few substitution neighbours (M_N = 0.7, range_N: 0–1) and 60 had many substitution neighbours (M_N = 9.5, range_N: 7–12). N was acquired using the English Lexicon Project (Balota et al., 2007). Word targets did not differ in zipf frequency scores (low N: M_zipf = 3.1, SD _zipf = 0.82; high N: M_zipf = 3.1, SD_zipf = 0.77; Mann Whitney U = 1768.00, p = .869), obtained from the SUBTLEX-UK database (van Heuven et al., 2014). One hundred and twenty non-word targets were selected from the English Lexicon Project (Balota et al., 2007) and had either no substitution neighbours or one substitution neighbour.

Design and procedure

The experiment consisted of two tasks: (1) the Edinburgh Handedness Inventory, (2) the visual half-field task. The visual half-field task was within-subjects and utilised a 2 (lexicality: word, non-word) ×2 (N: low, high) ×3 (visual field: RVF, LVF, CVF) design.

The study was administered via Gorilla (https://gorilla.sc/; Anwyl-Irvine et al., 2020). Participants were instructed to sit approximately 50 cm from the screen. Each trial began with a fixation cross presented centrally for 400 ms. Participants were instructed to keep their eyes on the central fixation point. Then, a target was presented for 150 ms to the left, right, or at the position of the fixation point. Previous attempts at using Gorilla to run similar visual half-field tasks involving rapid presentation of stimuli yielded good test-retest reliability (Parker et al., 2020).

Stimuli were presented in black lowercase Tahoma font on a white background. Gorilla’s scaling tool was used to ensure that stimulus size and position were kept consistent despite differences in participants’ screen sizes. Font was scaled to 250% equating to 28pt font, and stimuli were presented at a displacement of 2.5° of visual angle when presented to the left or right of fixation. Participants were instructed to press one of two buttons on the keyboard to indicate whether the string was a legitimate English word or not (“S” for word or “K” for non-word). They were instructed to do so as quickly and accurately as possible. The next trial started contingent on the participant’s response. Stimuli were divided into three counterbalanced lists with each participant responding to 120 words and 120 non-words in a random order. The whole session took approximately 12 minutes.

Data analysis

Error rates were analysed using a generalized linear mixed-effects model (GLMM) fitted to binomial data using the glmer() function and logit link from the lme4 package (version 1.1-21; Bates et al., 2019). Reaction times were analysed with linear mixed-effects models (LMM) using the lmer() function from the lme4 package. Both models adopted an identical structure: dv ∼ VF + N + VF:N + (1 ∣ participant) + (1 | word). The contr.sum function was used to implement summed-to-zero contrasts for N, constituting a main N effect. The contr.helmert() function was used to implement Helmert contrasts for visual-field. This enabled us to contrast means for the LVF and RVF directly before contrasting the combined mean of LVF and RVF with the mean for the CVF. This series of contrasts resulted in the intercept corresponding to the grand mean. For each model, we report regression coefficients (b), 95% confidence intervals (95% CI), standard errors (SE), and t/z values. We consider as statistically significant those cases where —t/z—>1.96.

While we considered all trials for error rate analysis, only correct responses were considered for reaction time analysis. Trials were excluded if reaction times were less than 250 ms or greater than 1,200 ms.³ Following the removal of these cases, we identified and removed outliers using Hoaglin & Iglewicz’s (1987) procedure. For each participant (across all conditions) we calculated the difference between first and third quartiles of the response time data. Outliers were then defined as those where reaction times were 1.65 times this difference above the third and below the first quartiles. This enabled us to exclude trials with particularly long responses which may have resulted from delayed loading times. These procedures led to the removal of 12.3% of trials.

Error rates

Error rates are shown in Table 1. The GLMM fitted to error rates indicated that there were fewer errors in the CVF relative to the mean of the RVF and LVF (see Table 2). There were fewer errors in the RVF than the LVF. The remaining fixed effects did not significantly impact error rates.

Reaction times

The LMM fitted to log-transformed reaction times indicated that reaction times were faster in the CVF relative to the mean of the RVF and LVF (meeting our first criterion for successful replication). Furthermore, reaction times were faster in the RVF than the LVF (our second criterion for replication). While the main N effect was not significant, the interaction between N and visual field (LVF –RVF) indicated that the N effect differed in the RVF and LVF. In the LVF, high N targets were responded to 24.7 ms faster than low N targets. In contrast, in the RVF high N targets were responded to 10.5 ms slower than low N targets (see Figs. 2A–2C). This pattern of effects was near identical to that reported by Perea, Acha & Fraga (2008; see Figs. 2D–2F). The second N by visual field interaction (CVF –LVF and RVF) did not significantly impact reaction time. This indicates that when N effects were summated across the LVF and RVF, the N effect did not statistically differ from the N effect in the CVF.

Exploratory analyses

In addition to our pre-registered analyses, we conducted exploratory analyses looking at the N effect at the participant level and how lateralisation on the visual half-field task correlated with the magnitude of the N effect.

To date, no study investigating the N effect during lateralized lexical decision has examined the extent to which individual participants show facilitatory or inhibitory effects of N in each visual field. Out of the 30 participants in the current study, 20 showed a facilitatory effect in the LVF, 14 showed a facilitatory effect in the CVF, and 11 showed a facilitatory effect in the RVF. The reaction time differences for individual participants are shown in Fig. 3A, with the numerical difference for each participant reported in the (Table S1). Furthermore, only 12 out of the 30 participants showed both a facilitatory N effect in the LVF and an inhibitory effect in the RVF, indicating that while this pattern of results is consistent at the population level, there are likely to be subtle individual differences in N effects across hemispheres.

One potential explanation for the observed differences in the magnitude of the N effect across each visual field is that the N effect is influenced by the extent to which participants are lateralized for language. It seems plausible that those who are left hemisphere language dominant may show one pattern of N effects while those who show bilateral, or even right hemisphere dominance, may show weaker effects of N or even a reversed pattern. To explore this potential explanation, we calculated laterality indices (LIs) using correct responses to both high and low N words in either the LVF or RVF: we assume that participants who are left-hemisphere dominant for language should show a RVF advantage (and therefore would respond faster to RVF words), whereas participants who are right hemisphere dominant would show a LVF advantage.

LIs were calculated using the formula: 100 ×(Right - Left)/(Right + Left), where Right refers to the mean reaction time in the RVF and Left refers to the mean reaction time in the LVF. A negative LI on this scale indicates lateralisation to the left hemisphere and a positive LI indicates lateralisation to the right hemisphere. The mean LI was −2.8 (SD = 4.27) and a one-sample t-test indicated that there was significant lateralisation to the left hemisphere at the population level, t (29) = −3.26, 95% CI [−4.42, −1.23], p = 0.001. A series of Pearson’s product-moment correlations indicated that the N effect was not correlated with LIs in the LVF, r = .04, 95% CI [−.32, .40], p = .818, CVF, r =  − .02, 95% CI [−.38–.34], p = .908, or RVF, r = .07, 95% CI [−.30, .42], p = .727.⁴ Figure 3D shows scatterplots of these results.

Literature search

The search of the literature followed the PRISMA guidelines (Moher et al., 2009) and covered all articles published between 1st January 1997 and 1st June 2020. Searches were conducted on the 18th June 2020 using several databases containing published and unpublished literature. The following bibliographic databases of published literature were searched: ProQuest, ScienceDirect, and Web of Science. The grey/unpublished literature were searched using: Ethos, OpenGrey.eu, Open Science Framework Pre-Prints, PsychArXiv, and DissOnline.de. Specifically, we searched the following terms in the title, topic, and abstract fields of each database: (“orthographic neighbo*” AND “hemisphere”), (“orthographic neighbo*” AND “hemifield”), (“orthographic neighbo*” AND “laterali*”), (“orthographic neighbo*” AND “visual field”), (“orthographic neighbo*” AND “visual-field”), (“orthographic neighb*” AND “Visual half field”), and (“orthographic neighb*” AND “visual half-field”). The reference list of key papers were also checked (Andrews, 1997; Ellis, 2004; Lavidor & Ellis, 2001; Lavidor & Ellis, 2002; Perea, Acha & Fraga, 2008; Whitney, 2004).

The study selection process is illustrated in Fig. 4. The abstracts of articles identified through database searches were screened for duplicates. All duplicates were then removed. Six reviewers then screened titles and abstracts of 232 unique articles using the metagear package (version 0.6; Lajeunesse, 2016), which was also used to store and manage bibliographic data throughout the review process. Reviewers assessed whether the article looked potentially eligible for inclusion and responded with yes, no, or maybe. Each article was screened twice by a different reviewer. The agreement rate was 94.4%. In the case of no agreement, a review was conducted by a third reviewer. From the 232 unique articles, seven were considered eligible for further screening. Four articles were rated as ’maybe’ relevant by reviewers, which prompted screening of the methods sections. Of these four, one article was considered eligible for inclusion. Thus, there were eight eligible articles in total.

Each of the eight articles was then evaluated against the inclusion criteria presented in the Supplemental Materials. In short, the studies had to examine the N effect during a lateralized lexical decision and include reaction time as a dependent variable. At this stage, one article was removed as it was a doctoral thesis where the relevant chapters had been published and the corresponding article had already been deemed eligible for inclusion. In addition to reviewing the full text, we checked the Retraction Watch Database (http://retractiondatabase.org/RetractionSearch.aspx?) to ensure these articles had not been retracted. None of the articles that passed a full-text review were indicated to have been retracted.

For the 7 included papers we conducted a forward search of the cited literature. In total, 237 studies were identified. After removing duplicates and articles that had previously been screened, 41 additional unique articles were identified. Each article was screened twice by a set of four reviewers. The agreement rate was 93%. For 3 articles where there was disagreement, a third screen was conducted. None of the 41 articles were rated as eligible for inclusion in the meta-analysis.

To summarise, our initial search and screening strategies led to seven articles being identified for use in the current meta-analysis (Fiset & Arguin, 1999; Lavidor & Ellis, 2001; Lavidor & Ellis, 2002; Lavidor, Johnston & Snowling, 2006; Mano et al., 2010; Perea, Acha & Fraga, 2008; Whitney & Lavidor, 2005).

Data extraction and effect size calculation

Of these seven articles, two reported multiple experiments (Lavidor & Ellis, 2002; Whitney & Lavidor, 2005). One of these (Lavidor & Ellis, 2002) included a between items manipulation of word length. Rather than aggregate across this manipulation, we treated each level of the manipulation as a separate experiment. Thus, data extraction was conducted for 11 experiments by two reviewers. In addition to extracting values such as sample size, language of stimuli, etc., reviewers extracted mean reaction times for low- and high-N words in both the LVF and RVF. This could be completed for all but one experiment (Fiset & Arguin, 1999). We contacted the authors of the experiment in an attempt to gain summary statistics and/or raw data. However, the authors could not locate the data, thus the experiment was removed from the meta-analysis. For the remaining experiments, both reviewers extracted identical mean reaction times across conditions and experiments.

As each experiment adopted a within-subjects design, it was necessary to obtain an estimate of the correlation between reaction times for high- and low-N words in each visual field. As none of the experiments included in the meta-analysis reported correlations between reaction times for low- and high-N words across visual fields, we contacted the first, last, and corresponding authors for each of the six articles requesting both the full and aggregate data. The authors indicated that they no longer had access to the data or we received no reply within one month. Thus, we could not obtain an estimate of the correlation between mean reaction times in the high- and low-N condition in either visual field from the published literature. As such, we decided to use our replication study to derive estimates of these correlations. From this dataset, the correlation between reaction times in the low- and high-N condition was r = .90 in the LVF and r = .79 in the RVF. It is important to note that these correlations are stronger in magnitude than those typically observed in psychological science. This will of course have implications for the outcome of our meta-analysis; as the correlation coefficient increases, the estimate of the variance for each study decreases and precision increases. As such, we felt it necessary to repeat the meta-analysis assuming a number of values for r and report these in the Supplemental Materials.

To estimate the size of the N effect across visual fields, reaction times in the low N condition were subtracted from scores in the high N condition. The variance of the effect size was calculated using Formulas 12.8–12.9 from Borenstein et al. (2009). The variance of each study (V) was calculated as: ${\displaystyle V=\frac{S_{dif}^2}{n}}$ where S_diff is the standard deviation of the mean difference for that study and n is its sample size. S_diff was calculated as: ${\displaystyle S_{dif}=\sqrt{S_1^2+S_2^2-2\times r\times S_1\times S_2}}$ where S₁ is the standard deviation of the high N condition, S₂ is the standard deviation of the low N condition, and r is the correlation between the means in the high and low N conditions. A summary of the effect sizes for the included experiments are shown in Table 3.

Publication bias

We assessed publication bias by presenting effect sizes against the inverse of their standard error in a funnel plot (Sterne et al., 2011). Symmetrical funnel plots indicate an absence of bias while asymmetrical funnels plots indicate the presence of bias. For the LVF, the funnel plot indicated no clear evidence of asymmetry/publication bias (see Fig. S1). To statistically assess asymmetry we conducted Egger’s regression (Egger et al., 1997), which indicated no significant evidence for funnel plot asymmetry in the LVF (t (9) = −.23, p = .821). Inspection of the funnel plots for the RVF indicated potential asymmetry (see Fig. S2). However, the Egger’s regression for the RVF indicated no significant evidence for funnel plot asymmetry (t (9) = −2.06, p = .070).

Quality assessment

The quality of each study was assessed by two reviewers via the Appraisal Tool for Cross-Sectional Studies (AXIS; Downes et al., 2016). The AXIS tool contains 20 questions grouped under introduction, method, results, discussion, and other. As some of the questions appeared more relevant to clinical work, we used only 15 items here. Ratings for each of the six published articles appear in the S2. Scores ranged from 6–11 for Reviewer 1 and from 7–11 for Reviewer 2.

It is noteworthy that each study used an incredibly similar methodology. All studies relied on right-handers drawn from a university population. Furthermore, the experimental setup was highly consistent across studies. For instance, the presentation time and displacement of stimuli did not vary considerably between studies. Most studies presented stimuli for 150 ms at 2.5° eccentricity.

The studies differed in how they operationalise high- and low-N. Lavidor & Ellis (2002) reported that words had a mean N of 9.5 and 1.0 in their high- and low-N condition. Perea, Acha & Fraga (2008) reported that the mean N of words in the high- and low-N conditions were 8.4 and 0.3 respectively. It may, therefore, be of interest to examine how the between-study differences in N thresholds influence the magnitude of the effect size across visual fields.

Data analysis

Meta-analysis

The results from the meta-analysis are shown in Table 4 and Fig. 5. The estimated N effect in the LVF was −31.8 ms, indicating that the N effect is facilitatory in the right hemisphere. In contrast, the estimated effect in the RVF was 10.8 ms, indicating that N is inhibitory in the left hemisphere. Not only was the direction of the effect different across visual fields, so was the size of the effect. Scrutiny of the I² statistic indicated moderate heterogeneity between studies comparing N effects in the LVF (Higgins et al., 2003). The I² statistic was larger for studies comparing the N effect in the RVF, indicating increased variation in outcomes for the RVF relative to the LVF. This is an issue we return to in our meta-regression.

Meta-regression

A series of exploratory meta-regressions were conducted to examine how the strength of the N manipulation influenced the N effect in each visual field. The strength of the N size manipulation was calculated by subtracting the mean low N rating for stimuli from the high N rating. For example, Lavidor & Ellis (2002) report that words had a mean N of 9.5 and 1.0 in their high- and low-N condition. Thus, the strength of the N manipulation was defined as 9.5–1.0 = 8.5. We fit a meta-regression model to the LVF data which indicated that the strength of the N manipulation did not influence the N effect in the LVF, b = .78, 95% CI [−3.28–4.85], SE = 2.07, p = .706. An identical model was fit to the RVF data, which yielded a similar pattern of results where the strength of the N manipulation did not influence the N effect, b =  − 3.34, 95% CI [−7.13–.46], SE = 1.94, p = .085.