Effectiveness and adequacy of blinding in the moderation of pain outcomes: Systematic review and meta-analyses of dry needling trials

Disclosure of deviations from prospectively registered protocol

Following original registration on March 2, 2016, two changes were made to the protocol of this review: (1) the data extraction template was pilot tested using an iterative process rather than a sample of 10 included trials, and percentage agreement was used to determine agreement, rather than an intra-class correlation coefficient (ICC); and (2) the time-point that was used to investigate the influence of blinding effectiveness on pain outcomes was the time-point at which blinding was assessed (instead of the pre-defined time-points of immediate, short-term, and long-term pain assessments), because the authors agreed that this time-point would most accurately reflect intervention beliefs (i.e., blinding effectiveness) as beliefs can change over time (Bang et al., 2010; Rees et al., 2005). The second change was updated in PROSPERO prior to data analyses (revision posted on February 5, 2017). This review presents only review questions 1 and 2 of the protocol; review questions 3 and 4 will be reported elsewhere.

Information sources and search strategy

One reviewer (FAB) executed the search strategy. Databases (MEDLINE, EMBASE, AMED, Scopus, CINAHL, PEDro, The Cochrane Library) were searched from inception to February 2016. The general search terms were (needl* OR acupuncture OR intramuscular stimulation) AND (sham OR placebo*), and Medical Subject Headings (MeSH) were used where possible. The full electronic search strategy for MEDLINE is presented in Table 1. Searches were modified to suit the functionality of each database. Thesis databases (Trove, ProQuest) and clinical trial registries (Australian New Zealand Clinical Trials Registry (ANZCTR), Clinicaltrials.gov, World Health Organization International Clinical Trials Registry Platform (WHO ICTRP)) were crosschecked with database searches to identify further potential trials. The reference lists of systematic reviews identified by the search were examined to locate additional or unpublished trials. There were no limits on year, language, or publication status.

Eligibility and study selection

Trials were eligible for inclusion in this review if they (1) were prospective experimental designs (e.g., randomised, non/quasi-randomised trials, pre-post, n-of-1) of any duration, which included a ‘real’ dry needling intervention (referred to as ‘active’ dry needling in this review) and a placebo/sham dry needling intervention; (2) included human adults (≥18 years of age) who were asymptomatic or with symptomatic health conditions; (3) involved a recognised dry needling approach with needle insertion sites based on anatomical or clinical rationales; (4) assessed and reported an outcome for pain [visual analogue scale (VAS) or numeric rating scale (NRS)]. Trials were also eligible for inclusion if they reported blinding assessment data, without reporting on pain, but the results from these trials are not presented in this review. Trials were ineligible for inclusion if the needling therapy involved pre-designated needle insertion sites (e.g., traditional acupuncture points) or involved injection of a substance (wet needling).

Records identified from the search strategy were exported to Endnote, duplicates were removed, and the remaining records were imported into the online screening tool ‘Covidence systematic review software’ (Anonymous, 2018). Titles and abstracts were screened against the eligibility criteria by three independent reviewers in teams of two (FAB and MPM or JLW), and trials potentially meeting the criteria were progressed to full text review. The same three reviewers independently screened the full-text articles in teams of two. Discrepancies were resolved through discussion, with an independent third reviewer (MPM, JLW, or LSKL) consulted where necessary. Where full-text was unavailable, authors were contacted to clarify eligibility and/or to provide full-texts. Non-English publications were translated using Google Translate; the extracted data were then checked with fluent speakers of each language.

Data extraction and Risk of Bias (RoB) assessment

A prospectively designed data extraction template was developed based on the Standards for Reporting Interventions in Controlled Trials of Acupuncture (STRICTA) (MacPherson et al., 2010) and the Cochrane Handbook ‘Checklist of items to consider in data collection or data extraction’ (Higgins & Green, 2011). The domains of data extraction were: source details, trial demographics, trial design, participant details, therapist details, intervention details, outcomes (pain and blinding assessment), blinding strategies, sample size and dropouts, results (pain and blinding assessment), and key conclusions of the authors.

The provisional data extraction template was pilot tested for inter-rater agreement by two reviewers (FAB and JLW) using an iterative process (two randomly selected included trials in each iteration). Once the pre-specified level of inter-rater agreement was established (≥75% agreement of items within an individual trial), two independent reviewers performed the remaining data extraction (FAB and LSKL, JLW, or MPM), with a third reviewer consulted to resolve disagreements as required.

Only data from the first phase of crossover trials were extracted due to the risk of carry-over intervention effects. Where necessary (i.e., where no text or table data were provided), graphical data were extracted using a ruler; if there were differences in these values between the two extracting reviewers, the average value was calculated. Pain intensity data were converted to a 100-point continuous scale where required (e.g., if collected using a 10 cm VAS or an NRS).

Risk of Bias (RoB) of individual trials was assessed using the Cochrane RoB assessment tool for randomised trials (because all included studies were randomised trials) (Higgins et al., 2011). Three key domains (allocation concealment, performance bias, detection bias) were determined a priori based on relevance to the review questions. The key domains were informed by empirical evidence for the likelihood and magnitude of these biases influencing trial outcomes (Higgins et al., 2011; Hróbjartsson et al., 2014; Hróbjartsson et al., 2013; Savović et al., 2012; Wood et al., 2008). The overall RoB for individual trials was determined using the three key domains (low = low RoB for all key domains, unclear = low or unclear RoB for all key domains, high = high RoB for one or more key domains) (Higgins et al., 2011). Two independent reviewers appraised each trial (FAB and MPM, JLW, or LSKL), with a third reviewer consulted to resolve disagreements as required.

Publication bias for each meta-analysis was assessed by visual inspection of asymmetry of funnel plots, which were contour-enhanced to allow consideration of the potential influence of the statistical significance of trial outcomes on publication bias (Peters et al., 2008). A statistical test for asymmetry was also computed for funnel plots containing ≥10 trials using the method specified in Egger et al. (1997) at a significance level of p < 0.10 (Higgins & Green, 2011; Sterne et al., 2011).

Data syntheses

For both review questions, meta-analyses used generic inverse variance and random-effects models. Restricted Maximum Likelihood (REML) was used to estimate between-trial variance. Stata statistical software (version 15.1) (StataCorp, 2017) was used to compute inferential statistics and create plots. The x² test and I² statistic were used to assess statistical heterogeneity; p < 0.10 was interpreted as statistically significant heterogeneity and I ²>50% was interpreted as substantial heterogeneity (Higgins & Green, 2011). Intervention effects were interpreted as statistically significant when p < 0.05, and between-group effect sizes [Standardised Mean Difference (SMD)] were considered large if >0.80, moderate if between 0.20 and 0.80, and small if <0.20 as defined by Cohen (1988).

A blinding index (BI) (Bang, Ni & Davis, 2004) was used to quantify the effectiveness of blinding (participant belief about group allocation relative to actual group allocation), where blinding assessments were sufficiently reported. The BI estimates the degree of unblinding (i.e., correct identification of group allocation) beyond random chance (Bang, Ni & Davis, 2004). To assist with interpretation of blinding effectiveness, groups within included trials were classified based on the BI cut-offs proposed by Moroz et al. (2013) (Table 2). Trials were then categorised based on paired classifications for the active and sham groups, termed a ‘blinding scenario’ (e.g., ‘Correct/Incorrect’, which means that the active group was classified as ‘Correct’ and the sham group was classified as ‘Incorrect’) (Bang et al., 2010). Using this classification method, a total of nine blinding scenarios were possible (Bang et al., 2010). The ‘R’ software package (version 3.4.3) (R Core Team, 2017) was used to compute BIs and their 95% Confidence Intervals (CIs).

Outcome of search strategy

The outcome of the search strategy is presented in Fig. 1. The search identified 11835 records. Four additional publications were identified by searching personal records (Itoh, Katsumi & Kitakoji, 2004) and through hand searching reference lists of 199 systematic reviews (Itoh & Katsumi, 2005; Itoh et al., 2006b; Katsumi et al., 2004). Following removal of duplicates, 4894 potentially relevant publications were screened. Title and abstract screening excluded 4280 publications. Of the remaining 614 publications, 588 were excluded following full-text review, leaving 26 publications (Fig. 1). The exclusion of two research questions from the current review resulted in the exclusion of three trials (within one publication) from this review because they did not report a pain outcome (Braithwaite, 2014) (this publication is included in Fig. 1 because the two omitted review questions that did include results from this publication are reported elsewhere). The 25 relevant publications included one trial that presented results over two publications (Tough et al., 2010; Tough et al., 2009), and two single publications with two eligible sham groups (Itoh & Katsumi, 2005; Itoh et al., 2007); therefore, 25 publications (with 26 group comparisons from 24 trials) are presented in the current review. Of these 25 publications, 24 publications (with 25 group comparisons from 23 trials) provided sufficient data for inclusion in the current meta-analyses. For the meta-analyses, in the two trials with two eligible sham groups (Itoh & Katsumi, 2005; Itoh et al., 2007) the active group data were used twice.

Five non-English publications were included in the current three Japanese publications (Itoh & Katsumi, 2005; Itoh et al., 2006b; Katsumi et al., 2004) and two Spanish publications (Espejo Antúnez et al., 2014; García-Gallego et al., 2011).

Of 22 authors who were contacted to clarify eligibility and/or to provide full-texts, 12 replied confirming ineligibility and 10 did not reply. For eight further records, author contact details could not be found. Nine authors of included trials were contacted to clarify trial details or request data; one author replied stating they no longer had access to the data, and the remaining eight authors did not reply.

Risk of Bias (RoB) assessment

A summary of results for the RoB assessment is presented in Table 3. Overall RoB was high in one trial, unclear in 18 trials, and low in five trials (Table 3). The areas with least RoB were participant blinding and reporting bias (low RoB in all included trials). The areas with greatest potential for bias were blinding of therapists and research personnel (high or unclear RoB in all included trials), allocation concealment, and attrition bias (Table 3).

Assessment of publication bias

Visual inspection of asymmetry of contour-enhanced funnel plots suggested that publication bias may be present (Fig. 2) (Peters et al., 2008). A statistical test for asymmetry was computed for funnel plots containing ≥10 group comparisons (Figs. 2A, 2C and 2D) and a statistically significant result was found for all three plots (p < 0.001, p = 0.083, and p = 0.061, respectively), which further supports the presence of publication bias (Egger et al., 1997).

Description of included trials

Table 4 presents a summary of trial characteristics and results of the 24 trials involving 26 group comparisons. Non-penetrating (NP) shams were used in 16 group comparisons, penetrating (P) shams were used in nine group comparisons, and one group comparison used anaesthesia (general or spinal) to blind participants (Mayoral et al., 2013) (Table 4). The 16 NP devices were guide-tubes alone (N = 3 group comparisons), custom-made blunted/retracting needles (N = 12 group comparisons), and one commercial device (the Park sham Park et al., 1999). Of the nine group comparisons that used penetrating shams, six inserted needles subcutaneously only (i.e., superficial dry needling above trigger points (TrP SDN) or away from trigger points (Non-TrP SDN)), and three inserted needles into muscle but away from trigger points (Non-TrP DN) (Table 4).

Fourteen trials (16 group comparisons) assessed blinding effectiveness (N = 13 trials) or intervention credibility ( N = 1 trial). To assess blinding effectiveness, participants were asked whether they thought a needle had been inserted (N = 9 trials), if they felt a ‘needling sensation’ (N = 1 trial), or which group they thought they were in (N = 1 trial). The remaining two trials did not report how blinding was assessed (Huguenin et al., 2005; Sterling et al., 2015). To assess intervention credibility, participants completed the Credibility/Expectancy Questionnaire (CEQ) (Devilly & Borkovec, 2000) (N = 1 trial). Of the 13 trials that assessed blinding effectiveness, 10 trials (12 group comparisons) presented blinding data in a way that the BI could be calculated for active and sham groups (Table 4). To evaluate blinding effectiveness, nine trials used inferential statistics to determine if there was a difference in proportions of guesses, and four trials described, but did not statistically analyse, the blinding data (Table 4). For pain outcome assessments, 22 trials used a VAS (N = 11 trials used a 100 mm scale and N = 11 trials used a 10 cm scale) and two trials used an NRS (NRS 0-10).

Data syntheses

Review question 1: Does blinding effectiveness moderate intervention effect on pain?

Figure 3 presents a bubble plot (meta-regression) of the influence of the summary BI on effect size (pain). There was no evidence of a moderating effect of the summary BI on effect size (meta-regression coefficient −1.87 (95% CI [−5.63–1.88]); p = 0.292; N = 12 group comparisons; n = 248) (Fig. 3). There was evidence of statistically significant and substantial statistical heterogeneity (I² = 79.0%; p < 0.001) (Higgins & Green, 2011).

Immediate intervention effect (<24 h after the first/only intervention)

There were seven group comparisons where immediate pain outcomes were collected (Fig. 4). One group comparison (n = 31) met the requirements for adequate blinding (Inoue et al., 2006), and intervention effects were statistically significant in favour of active dry needling (SMD −0.76 (95% CI [−1.49–−0.03])). For inadequately blinded group comparisons (N = 6; n = 206), there was no evidence of a difference in intervention effects between active and sham groups [pooled SMD -0.47 (95% CI -0.95 to 0.02)]. There was evidence of significant and substantial heterogeneity in the pooled group comparisons (Higgins & Green, 2011) (Fig. 4).

Short-term intervention effect (24 h to one month, closest assessment to one week)

There were 20 group comparisons where short-term pain outcomes were collected (Fig.  5). For adequately blinded group comparisons (N = 3; n = 122) there was no evidence of a difference in intervention effects between active and sham groups (pooled SMD −0.40 (95% CI [−0.96–0.15])), whereas inadequately blinded group comparisons (N = 17; n = 504) had statistically significant intervention effects that favoured active dry needling (pooled SMD -0.71 (95% CI [−1.05–−0.38])). There was evidence of statistically significant and substantial heterogeneity of pooled group comparisons for the inadequately blinded subgroup, whereas the adequately blinded subgroup had moderate heterogeneity that was not significant (Higgins & Green, 2011) (Fig.  5).

Long-term intervention effect (one to six months, closest assessment to three months)

There were 16 group comparisons where long-term pain outcomes were collected (Fig. 6). For adequately blinded group comparisons (N = 4; n = 202) there was no evidence of a difference in intervention effects between active and sham groups (pooled SMD −0.30 (95% CI [−0.62–0.02])), whereas inadequately blinded group comparisons (N = 12; n = 284) had statistically significant intervention effects that favoured active dry needling (pooled SMD -1.14 (95% CI [−1.64–−0.65])). There was evidence of statistically significant and substantial heterogeneity of pooled group comparisons for the inadequately blinded subgroup, whereas the adequately blinded subgroup had low heterogeneity that was not significant (Higgins & Green, 2011) (Fig. 6).

Key findings

This review aimed to determine whether blinding effectiveness and/or blinding adequacy moderated pain outcomes in dry needling trials. Of the 23 trials included in the meta-analyses, only 10 (43.5%) reported data that were sufficient to calculate the BI, and only five (21.7%) reported adequate blinding procedures. The small number and size of included trials meant that there was insufficient evidence to determine if a moderating effect of blinding effectiveness or adequacy existed (Button et al., 2013; Higgins & Green, 2011).

Strengths and limitations

The strengths of this review included prospective peer review and registration of the protocol (PROSPERO), adherence to the PRISMA statement for reporting (Moher et al., 2009), and independent screening for trial eligibility, data extraction, and RoB assessments by two reviewers. The search strategy was comprehensive and trials were not limited to the English language. Despite attempts to limit the impact of publication bias on the current results by searching trial registrations and thesis databases, asymmetry of funnel plots suggests publication bias was present (Fig. 2). The small number of trials in one of the funnel plots (<10; Fig. 2B) meant that it could not be confidently interpreted (Egger et al., 1997; Higgins & Green, 2011).

The current findings should be interpreted with caution. The meta-analytical techniques used in this review are not randomised comparisons and are therefore observational in nature (Higgins & Green, 2011). The strength of inferences is therefore limited by potential confounding by uncontrolled covariates, and subgroups may have differed in capacity to detect effects (Higgins & Green, 2011). However, for review question 2, the a priori hypothesis, the statistical significance of the findings, and the consistency of the difference across comparisons strengthen the validity of the inferences (Oxman & Guyatt, 1992). That no studies to date have made head-to-head comparisons of blinded versus non-blinded dry needling interventions precludes any further analysis, aside from indirect comparisons. The meta-analyses used random effects modelling to allow for statistical heterogeneity between trials (Higgins & Green, 2011). Homogeneity was improved for meta-analyses investigating blinding adequacy (I ² values <75%; Figs. 4–6), in which comparisons were grouped based on four RoB domains, for three pre-defined time periods, so these analyses may be more reliable (Higgins & Green, 2011).

The included trials were methodologically heterogeneous, and many were likely to have had a high risk of null findings in the presence of small to moderate effects due to insufficient power (N = 20 group comparisons with n < 50 participants, with power clearly achieved for the pain outcome in only three trials, and zero for blinding assessment outcomes) (Button et al., 2013). Trials were also clinically diverse in terms of participant health condition, pain chronicity, age, and intervention dose, which may have confounded results, in particular because the aetiology of pain may influence the specific effects of dry needling (Cagnie et al., 2013; Dommerholt, 2011), as well as non-specific effects (Tracey, 2010). The limited number of trials precluded investigation of potential covariates (i.e., sensitivity or multivariable meta-regression analyses) (Higgins & Green, 2011).

Contrary to best practice, active group data were used twice in several meta-analyses that included trials with two eligible sham groups (Itoh & Katsumi, 2005; Itoh et al., 2007) (Figs. 3, 5 and 6), which may have caused unit-of-analysis errors due to correlations between the non-independent comparisons (Higgins & Green, 2011). However, due to extremely small sample sizes (n = 8 to 10 participants in the relevant active groups) and potential differences in physiological effects of the sham interventions (i.e., penetrating versus non-penetrating) (Lund, Näslund & Lundeberg, 2009), it was decided that the active group data could not be split, nor could the sham groups be combined as recommended by Higgins & Green (2011).

To determine whether the current review required updating (original search completed in February 2016), a citation search was undertaken for trials included in the current systematic review (292 citations since January 2016 as at 18th of September 2017, with reference lists of 39 potentially relevant systematic reviews also reviewed). This search revealed 47 new prospective primary studies of dry needling; of these, only one was blinded using sham dry needling and this trial did not report an assessment of blinding effectiveness (Mason et al., 2016). Addition of one trial to the current review was unlikely to significantly alter the results, therefore the review was not updated (Elkins, 2018).