Replicability of sight word training and phonics training in poor readers: a randomised controlled trial

Trial design

All children completed screening and outcome measures at Test 1 (see Fig. 1). After 8 weeks of no training, they returned to do the outcome measures (Test 2) to index “non-treatment gains” (i.e., due to test-retest effects, test situation familiarity, regression to the mean, maturation, and a “test-related Hawthorne effect” resulting from an awareness of being tested multiple times on similar outcome measures). Group 1 did 8 weeks of phonics training (and then Test 3) followed by 8 weeks of sight word training (and then Test 4). Group 2 did the same training in the reverse order. In the analysis, we controlled for non-treatment gains when comparing phonics training to sight word training, and when comparing phonics-then-sight word training versus sight word-then-phonics training. (Note: unlike McArthur et al. (2013a), we did not include a “mixed” group since this group showed no advantage over groups 1 and 2 in McArthur et al.)

Participants

In line with McArthur et al. (2013a), this study recruited a typical “mixed” sample of poor readers from the community. Children were aged from 7 to 12; scored below the average range for their age (i.e., had a z score lower than −1.0, which represents the lowest 16% of readers) on the Castles and Coltheart 2 (CC2) irregular-word reading test and/or nonword reading test (Castles et al., 2009; see below); had no history of neurological or sensory impairment as indicated on a background questionnaire; and used English as their primary language at school and at home (see Screening Tests below). This resulted in a sample that was very similar in age, nonverbal IQ, and irregular word reading as McArthur et al. (see Table 2). However, the mean CC2 nonword reading scores for groups 1 and 2 in McArthur et al. (−1.50 and −1.27, respectively) were higher than in the current study (−1.66 and −1.62, respectively). Similarly, the mean CC2 regular word reading scores for groups 1 and 2 in McArthur et al. (−1.41 and −1.29, respectively) were higher than in the current study (−1.61 and −1.57, respectively). Thus, on average, children in the current study had slightly poorer explicit phonological decoding abilities than children in McArthur et al. (2013a).

Interventions

Screening tests

Primary outcomes

Sample size

A flow diagram of the number of participants in each stage of the study is shown in Fig. 2. At the end of the study, there were 41 children in Group 1 and 44 children in Group 2.

Sequence generation

Children were allocated to groups using minimisation randomization (balanced 1:1 for age, CC2 nonword reading, CC2 irregular word reading; executed using MINIMPY; Saghaei, 2011), which is considered the most appropriate sequence allocation procedure for trials comprising fewer than 100 participants. It is considered methodologically equivalent to randomization by CONSORT (Schulz, Altman & Moher, 2010; Note: McArthur et al. (2013a)) used a quasi-randomised allocation procedure).

Allocation concealment and implementation

The lead research assistant on the project allocated children to each group and arranged their training. They concealed group allocation from research assistants who conducted the test session. All training was done online at home. All instructions to parents were provided via written documents. Parents contacted the lead research assistant if unclear about any aspect of the training.

Blinding

Unlike drug trials, it is difficult to guarantee double blinding in cognitive treatment studies. However, parents and children were not told their group allocation, and all children received exactly the same type of training (in different orders). Most parents and children lack the expertise to discriminate between different types of reading. In addition, no tester assessed the same child twice, and no tester was aware of the child’s group allocation (i.e., the tester was blind to group allocation). Thus, it is highly likely this study used a double-blind procedure.

Participant flow

A flow diagram of the number of participants in each stage of the study is shown in Fig. 2. 41 successfully completed the phonics-then-sight word training (Group 1), and 44 successfully completed the sight word-then-phonics training (Group 2). We included all children in the final analysis who completed their training, bar one child whose mother admitted at the end of the study that her child had been participating in another reading intervention. Participants who withdrew from the training did so for various personal reasons. Thus, the drop out rate in this study was low, and reasons for drop out appeared random.

Baseline data

Between groups t-tests revealed that the two training groups did not differ significantly on the screening and outcome measures prior to training (i.e., see Table 2).

Training fidelity

Based on McArthur et al. (2013a), we predicted that by asking children to train for five 30-minute sessions per week for 8 weeks (20 h in total), at a minimum they would manage four 20-minute session per week for 7 weeks (due to illness, holidays, and the occasional “bad day”; a minimum of 9 h and 20 min). Two children from each group failed to reach this minimum, and were excluded from the final analysis. On average, the final sample completed around 14 h for each program (see Table 2). There was no significant difference between groups in training times.

Numbers analysed

The analyses included 41 children in Group 1 and 44 children in Group 2. We analysed the data of participants in the groups to which they were originally allocated. In line with McArthur et al. (2013a), we conducted an available case analysis on the data (i.e., based on participants with complete data).

Outcomes

Figure 3 shows each group’s mean and 95% confidence interval (CI) for gains in raw scores (i.e., difference scores) for each outcome measure. The first three CIs in each graph represent Group 1, and the last three CIs represent Group 2. Within each group, the first CI (T1T2) represents gains in raw scores from Test 1 (before training) to Test 2 (after 8 weeks of no training) due to non-training effects. The second CI (T1T3) reflects gains in raw scores between Test 1 (before training) and Test 3 (after the first 8 weeks of training). The third CI (T1T4) reflects gains in raw scores between Test 1 (before training) and Test 4 (after 16 weeks of training).

Any T1T2 CI marked with * represents a statistically significant gain due to non-training effects. Any T1T3 or T1T4 CI marked with ** represents a statistically significant gain that is significantly larger than non-training effects. Only gains marked ** were considered “valid training effects”. For each effect, we calculated Cohen’s d effect sizes calculated from the difference scores (i.e., mean group difference score/SD group difference score). Cohen’s d scores of 0.3, 0.5, and 0.8 were considered to represent small, medium, and large effect sizes, respectively. Effect sizes for each outcome measure are compared to McArthur et al. (2013a) in the Table 1.

To determine if there was a reliable difference between 8 weeks of phonics and sight word training, we used a between-group ANCOVA (controlling for each group’s corresponding non-training gains measured over the T1T2 no-training period) to compare T1T3 gains for Group 1 and Group 2. To determine if different orders of training had different effects on each outcome, we used a between-groups ANCOVA (controlling for non-training gains) to compare T1T4 gains for each group.

Main findings

The aim of the current study was to test the replicability of the sight word and phonics training effects in poor readers reported by McArthur et al. (2013a). Regarding sight word training, McArthur et al. found that specific sight word training had (1) large and significant valid treatment effects on trained irregular words (replicated in this study: d = 1.0), untrained irregular words (replicated in this study: d = 0.9), word reading fluency (replicated in this study: d = 1.4), and word reading comprehension (not replicated in this study: non-significant d = 0.6); (2) a moderate and significant valid treatment effect on nonword reading accuracy (not replicated in this study: d = − 0.1); and (3) no valid treatment effect on nonword reading fluency (replicated in this study: non-significant d = 0.2). Thus, the current study replicated all bar two of the sight word training effects found by McArthur et al. (2013a).

In the light of McArthur et al.’s significant and valid sight word treatment effects on reading nonwords and reading comprehension, our non-significant sight word training effects on these skills were somewhat puzzling. However, in light of dual route and triangle models of reading, these outcomes made sense. Our specific sight word training used irregular words to maximize training the lexical/semantic pathway, and minimize training the sublexical/phonological pathway. This, in turn, minimized the training of cognitive skills that underpin the ability to read nonwords (i.e., explicit phonological decoding). In addition, our sight word training, which trained the ability to read and spell irregular words by sight, did not train the types of words (typically regular words) that were used as stimuli in the reading comprehension test. Thus, the puzzle is not so much why the current study failed to find a valid sight word training effect on nonword reading and reading comprehension, but why McArthur et al. did, since they used very similar methods.

Given that McArthur et al. (2013a) and the current study represent the only two group controlled trials of specific sight word training (i.e., using irregular words to train the ability to recognise words from orthographic memory) in poor readers, we must turn to a single case study by Broom & Doctor (1995) for insight. This study examined the effect of specific sight word training (i.e., training irregular words) on reading comprehension in an 11-year-old child with developmental surface dyslexia. Like McArthur et al. (2013a) and the current study, specific sight word training had a significant effect on both trained and untrained irregular words. In accord with the current study, but not McArthur et al., it did not have an effect on reading comprehension. The authors suggest, and we concur, that their specific sight word training did not generalise to reading comprehension because their training did not provide the explicit opportunity to apply newfound word reading skills in a reading-comprehension context.

Moving onto phonics training, which trained explicit phonological decoding (reading) and encoding (spelling), McArthur et al. (2013a) found that specific phonics training had (1) large and significant effects on trained irregular words (replicated in this study: d = 0.7), untrained irregular words (replicated in this study: d = 0.8), nonword reading fluency (not replicated in this study: d = 0.2), word reading fluency (replicated in this study: d = 0.9) and reading comprehension (replicated in this study: d = 0.9); and (2) a moderate-to-large effect on nonword reading accuracy (not replicated in this study: d = 0.3). Thus, like the sight word training, the current study replicated all bar two of the phonics training effects found by McArthur et al. (2013a).

While the current study did not replicate the moderate-to-large phonics training effects on nonword reading found by McArthur et al. (2013a), it is not the case that phonics had no effect on nonword reading at all. Figure 3 shows that Group 1 made gains in their nonword accuracy and fluency over their 8 weeks of phonics training that were clearly larger than their non-training gains. However, these gains just failed to reach statistical significance. After a further 8 weeks of training, Group 1’s gains became statistically significant due to minor additional gains made over 8 weeks of sight word training. Group 2’s data show exactly the same pattern of results but in the reverse order (i.e., because they did phonics training after sight word training). Thus, the outcomes of the current study suggest that phonics training did have an effect on nonword reading accuracy and fluency, but this effect was certainly smaller than the effect found by McArthur et al.

Why might this be the case? Since the current study used very similar methods to McArthur et al. (2013a), the answer most likely lies with our sample. As noted under Participants, groups 1 and 2 in the current study had slightly weaker explicit phonological decoding abilities than groups 1 and 2 in McArthur et al. Such children may respond less well to phonics instruction (Galuschka et al., 2014), which would explain why the current study found smaller effects of phonics training on tests that tax phonics-related skills such as nonword reading accuracy and fluency.

Finally, in terms of order training, McArthur et al. (2013a) found that order of sight word and phonics training only had an effect on untrained irregular word reading, which was significantly better after phonics-then-sight word training than sight word-then-phonics training. This was not observed in the current study. The closest thing we found to an order effect was for word reading fluency, which was markedly higher after phonics-then-sight word training than the reverse. However, this order effect just failed to reach statistical significance. Combined with the outcomes of McArthur et al., this finding suggests that order of phonics and sight word training may not matter in poor readers aged from 7 to 12 years who have some phonics related skills (i.e., who can read at least a few nonwords). To a certain extent, this makes sense in terms of the dual route and triangle models of word reading, which make no predictions about the effect of training one pathway (e.g., the sublexical/phonological pathway) before another (e.g., the lexical/semantic pathway). However, this finding does not align with the assumption that poor readers should be taught explicit phonological decoding prior to sight word reading (Chall, 1967). Whether or not an order effect might apply to poor readers with no phonics-related skills at all remains an empirical investigation at this point in time.

Limitations

Because the current study is a replication of McArthur et al. (2013a), it necessarily shares some of its limitations. One was the use of a “within-subjects” control group to index non-treatment gains (i.e., from Test 1 to Test 2) rather than a separate “between-subjects” untrained group (i.e., from Test 1 to Test 2 to Test 3 to Test 4). McArthur et al. (2013a) chose to use a within-samples control group for three reasons. First, children in a between-subjects control group may produce different (e.g., smaller) non-treatment gains than children in a within-subjects control group, which may lead to over-estimations of a treatment effect. Second, recruiting a between-subjects group would delay the administration of potentially effective treatment for poor readers for 6 months during a critical period of their reading development. And third, it is more difficult to recruit poor readers for a study in which there is a high chance of being allocated to an untreated control group.

The use of a within-subjects control group in both McArthur et al. (2013a) and the current study allowed the explicit measurement of non-training effects from Test 1 to 2 (T1T2), but not from Test 1 to 3 (T1T3), or from Test 1 to 4 (T1T4). Thus, the use of T1T2 gains to represent non-training gains may have underestimated true T1T3 and T1T4 non-training gains. According to previous research, non-training gains on cognitive tests over no-training periods decrease in size across test sessions (e.g., Bartels et al., 2010; Collie et al., 2003; Kohnen, Nickels & Coltheart, 2010). Thus, if T1T3 and T1T4 non-training gains were solely responsible for any “valid training gains” found in this study (i.e., gains marked ** in Fig. 3 that are both significantly larger than 0 and significantly larger than T1T2 gains) then (1) T1T3 gains should be less than double T1T2 gains, (2) T1T4 gains should be less then triple T1T2 gains, and (3) both groups should show very similar-sized gains (since type of training should have no effect). Examination of Fig. 3 reveals that these criteria did not apply to gains in trained irregular word accuracy, nonword reading accuracy, nonword reading fluency, or word reading fluency. This reinforces the conclusion that these gains reflect valid training gains. However, these criteria did apply to untrained irregular word accuracy and reading comprehension, which questions whether the gains in these outcomes were valid training gains, as defined by the criteria used by McArthur et al. (2013a).

Given the apparently reliable effects of sight word and phonics training on trained irregular word accuracy, nonword reading accuracy, nonword reading fluency, and word reading fluency, but the questionable effects of this training on untrained irregular word accuracy and reading comprehension, it is clear that a randomised controlled trial is now needed to compare the effect of phonics and sight word training to an untrained control group and a trained control group (e.g., maths training). Since McArthur et al. and the current study both found that order of training phonics and sight word reading had a limited effect on outcomes in 7- to 12-year-old poor readers, such studies could focus on training phonics and sight words in isolation. This would reduce the length of the experiment from 6 months (i.e., including a test-retest period, and two training periods) down to just 2 months (comprising a single training period). Unlike the current study, such a randomised control trial that included both an untrained control group and a trained control group would allow the explicit tracking of non-training effects across all test sessions. In the case of a trained control group, such non-training effects would include training-related Hawthorne effects, which are improvements in reading and spelling outcomes arising from an awareness of being involved in training. This new randomised controlled study may also benefit from including a passage reading test as an outcome measure to extend our understanding of the effects of phonics and sight word training in poor readers.

A second limitation of both the current study and McArthur et al. (2013b) was that the reading gains made by poor readers—though statistically significant, reliable, and large in effect size—did not “propel” children’s reading into the average range. This does not represent a failure of phonics or sight word training. Instead, it represents the degree of difficulty of treating reading in children who are, by definition, “reading resistant.” Now that we have established that phonics and sight word training both have reliable effects on heterogeneous groups of children with poor reading we can start to focus on how such effects can be maximised in children with different patterns of reading impairment.

A third limitation of this study, which was not considered by McArthur et al. (2013a), and in some ways may be considered a strength, is the highly mixed educational backgrounds of the poor readers in both studies. The country in which both studies were conducted (i.e., Australia) has a highly unregulated approach to reading. The national curriculum is too vague to provide clear advice to teachers about how much time should be spent on different reading strategies, and the National Inquiry of the Teaching of Reading in Australia revealed that tertiary teaching courses allocate less than 5–10% of course time to teaching student teachers how to teach reading (Rowe, 2005). Adding to this confusion is the fact that over the last 5–10 years, evidence-based schools have been moving away from a strictly “whole language” approach to more mixed approaches that include phonics instruction. This means that the children in this study were receiving very different “mixes” of reading instruction at their various schools. On the one hand, this is problematic because it means that this study cannot provide any insight into how phonics training paired with sight word training might interact with different types of instruction at school. However, on the other hand, the fact that two studies (i.e., the original study and the current study) have found similar effects of phonics and sight training in groups of poor readers from very different educational backgrounds attests to the usefulness of these instructional strategies English-speaking countries where the regulation of the teaching of reading is poor.