The rule-based insensitivity effect: a systematic review

Protocol and registration

The review protocol was designed in line with the PRISMA guidelines (Moher et al., 2009) and registered in PROSPERO (CRD42018088210).

Information sources and search strategy

To identify as many relevant records as possible, multiple electronic databases were searched (i.e., “Web of Science”, “PsychINFO”, “PsychArticles”, and “PubMed [Medline]”) using the search terms: “rule governed behavior”, “rule-governed behavior”, “rule governed behaviour”, “rule-governed behaviour”, “verbal regulation”, “instructional control”, “verbal rule”, “instructed behavior”, “instructed behaviour”, “instructed learning”, “instruction following”, “instruction-following”, “rule following”, and “rule-following.” These search terms were iteratively developed with experts on systematic reviews and rule-governed behavior, and were subsequently presented to other experts on systematic reviews and rule-governed behavior who were not associated with the project. All searches were conducted on 4∕10∕2017 by the first author (i.e., Ama Kissi) and yielded 1459 records. Five novel records were additionally retrieved by contacting experts in the field, which resulted in a final set of 1464 records that were assessed for eligibility.

Eligibility criteria

There were several general criteria that a record had to meet before being included in the current review: (1) it had to be a peer-reviewed journal article, (2) it had to be written in English, (3) it had to include a study that examined the RBIE by first asking participants to follow socially -or self-generated rules that initially corresponded with a set of contingencies but then became inaccurate after a contingency change, and (4) this study had to have an overall sample age of at least 18 years, (5) and at least 10 participants within each experimental group (see Van Ryckeghem et al., 2018 for similar eligibility criteria).

Furthermore, depending on the research objective under scrutiny, the individual studies reported in these records had to meet an additional number of criteria to be deemed eligible for inclusion. For instance, when addressing our first research question (“Is there evidence for the rule-based insensitivity effect in adults?”), we only included studies that did not focus upon individuals with clinical problems. That is, only studies which used convenience samples (e.g., students), samples taken from the general population, or those that were not diagnosed with clinical problems, or reported sub-clinical problems were included. Studies were deemed eligible for answering our second research question (“Do adults suffering from psychological problems display a larger RBIE compared to their non-clinical counterparts?”), if they used the following samples: individuals diagnosed with psychological problems (clinical group) or those who scored high on instruments measuring psychological problems but were not formally diagnosed with a clinical problem (sub-clinical group), and a comparison group consisting of individuals that did not suffer from the above problems or were recruited via convenience sampling.

Study selection process

Out of the 1,464 records that were assessed for eligibility, 1,446 were excluded because they were not published in English (n = 123), were not peer-reviewed journal articles (n = 207) (e.g., book chapters, dissertations, or conference papers) or dealt with a topic that did not meet our inclusion criteria (n = 1044). Three journal articles were, furthermore, omitted because they did not provide sufficient information to assess their eligibility. An additional 69 journal articles were excluded that were on the RBIE but were non-experimental (n = 6), relied on non-adult samples (n = 14), used samples with less than 10 participants per experimental condition (n = 41), or did not include a contingency change or manipulate accurate rules (n = 8). This resulted in a remaining total of 18 records consisting of 22 individual studies. One of these studies was subsequently omitted because it did not have at least 10 participants within each experimental group. As such, 21 studies were finally included in the systematic review. The eligibility of all studies were independently assessed by the first two reviewers (i.e., Ama Kissi and Colin Harte) initial agreement = 99% [kappa = .98], agreement after discussion = 100% [kappa = 1.00]). See Fig. 1 for the flow diagram of the study selection process.

Qualitative synthesis: coding procedure and items

Certain characteristics of each of the 21 studies were independently coded by the first two reviewers (i.e., Ama Kissi and Colin Harte) (initial inter-reviewer agreement = 96%, inter-reviewer agreement after discussion = 100%). These characteristics involved the source, study, task, and sample characteristics. The source characteristics entailed the year in which the first author published the study and the country where s/he worked in when the paper was published. The study characteristics referred to the type of task, experimental design, procedure, and analytic method that were used to examine the RBIE. Furthermore, the task characteristics entailed whether a study reported the exact instructions or rules that were used, how these instructions or rules were delivered (orally versus written) or generated (self [i.e., by the rule-follower]-versus socially [i.e., by another person than the rule-follower]), the reinforcement schedules that were used, the required behavioral responses, the type of consequential stimuli that were used, whether the contingency change was (un)signaled, whether a description was provided of who the experimenter was, and whether the experimenter was present. Finally, the sample characteristics that were evaluated were the size and mean age of the sample, the ratio of males:females, and whether the sample was selected (i.e., from either a healthy, clinical or sub-clinical population, or the general population) or non-selected (i.e., a convenience sample). These characteristics were evaluated for each experimental group.

Quantitative synthesis: vote counting

To synthesize the quantitative results of the included studies, we used the vote-counting method. This method was chosen because not all studies reported effect sizes or information that could be used to calculate such estimates. According to the Cochrane Collaboration guidelines for systematic reviews, the best way to use the vote-counting method is by assessing whether the results of the empirical studies fall into one of two categories: “positive” or “negative” effects (see Deeks, Higgins & Altman, 2008). Positive effects refer to results that are in favor of the predicted relationship between the independent and dependent variable(s), whereas negative effects refer to outcomes that are in the opposite direction of what is expected. We only judged (or voted) whether a study had positive or negative effects if it included a comparison group (i.e., a no-instructions group). That is, a group that received the same treatment as the rule groups but was not asked to follow the instructions or rules that these groups had to follow. We applied this restriction because we argued that such a comparison group is necessary if a study wishes to draw conclusions about the extent to which certain rules or instructions are responsible for the observed effects. In doing so, performances in the comparison group would serve as a baseline of how people behave in the absence of these types of rules or instructions. As such, if a study did not include such a comparison group, we argued that its effects were unclear (i.e., there was insufficient information to cast votes).

The outcome data that were preferably used to cast votes were measures of the central tendency (e.g., mean, mode, or median) of participants’ responses, during all blocks after the contingency change. If a study, however, did not report participants’ performances during all blocks following the contingency change, but only during a fraction of the trials after this change, we limited our analysis to that data. In the unfortunate event that no data was provided that could be used to draw conclusions about the central tendency of participants’ responding after the contingency change, we relied on the conclusions that the authors formulated themselves (Cerutti, 1991; Torgrud, Holborn & Zak, 2006) [Experiments 1 and 2]). Finally, in all of the above cases, if there were multiple contingency changes we only considered participants’ responding after the first change. This was, specifically, done to prevent carry-over effects from influencing the interpretation of the results.

All votes were independently cast by the first two reviewers (i.e., Ama Kissi and Colin Harte) in the following manner (inter-reviewer agreement = 100%, kappa = 1.00). For the first research question (“Is there evidence for the rule-based insensitivity effect in adults”), study results were considered positive if evidence was found for the RBIE. That is, if participants did not adapt to a novel task-contingency or rule (i.e., if their behavior was still in line with the self-generated or socially-provided rule that was in place before the contingency change). Furthermore, study results were considered negative if one of three conditions were met. First, if a task-contingency was changed and participants’ behavior was now always in line with this novel contingency. Second, if a self-generated or socially-provided rule was altered, and participants’ behavior was now always in accordance with this novel rule. Third, if both a task-contingency and rule was changed, and participants’ behavior was now always in line with this novel contingency and rule.

To cast votes for the second research question (“Do adults suffering from psychological problems display a larger RBIE compared to their non-clinical counterparts?”), we first assessed whether there was evidence supporting the RBIE. This was achieved in the same way as outlined above. If evidence for the effect was found, we subsequently examined if it was larger (in absolute terms) in the (sub-)clinical groups, compared to their non-clinical counterparts. If this was the case, then the study results would be categorized as positive. If these results were in the opposite direction, we would categorize them as negative.

Assessment of risks of bias

We, additionally, scrutinized the internal validity of the included studies. This examination involved assessing risks of bias using the Cochrane Collaboration tool for assessing risks of bias (Higgins & Altman, 2008) and the Office of Health Assessment and Translation (OHAT) Risk of Bias Rating Tool (NTP, 2015). Risks of bias can be defined as those aspects of a study design that can distort the conclusions that can be drawn from it. For the present review, we evaluated five potential risks of bias: selection, exclusion, performance, detection, and reporting bias. Note that these biases do not cover all risks of bias that are described in the Cochrane Collaboration and OHAT risks of bias tools. Indeed, given that these tools were not originally developed for assessing risks of bias in experimental-behavioral research, we selected and reformulated those risks of bias that we deemed relevant for evaluating such work.

For each of the studies, judgments of risks of bias (coded in terms of ‘high’, ‘low’, or ‘unclear’ risk of bias) were made in the following ways. To examine the possibility that there were systematic differences between the baseline characteristics of the groups that were compared (i.e., a selection bias), we examined: (1) the adequateness of a study’s sequence generation procedure, (2) whether the experimental group to which participants were allocated to was concealed, (3) participants’ past experiences with the experiment, and (4) the possibility that they were misclassified to experimental groups. Furthermore, to assess the likelihood of an exclusion bias (i.e., systematic differences in the exclusion of participants from a study) we evaluated the possibility that there were systematic differences between groups with regard to the amount, nature, and handling of missing outcome data. To determine the risk of a detection bias (i.e., systematic differences between groups in how outcomes are determined) we evaluated: (1) the validity and reliability of the outcome assessment methods, (2) the adequateness of the outcome assessments, (3) the adequateness of the methods that were used to determine sample sizes and (4) the adequateness of the methods used to analyze the results. Judgments concerning performance biases (i.e., systematic differences between groups in how they were treated or exposed to factors other than the manipulation of interest) were made by examining whether: (1) the experimental contexts were standardized, (2) participants were informed about the study objectives, and (3) researchers and/or participants were informed about the experimental group to which participants were allocated to. Finally, to assess the possibility of a reporting bias (i.e., systematic differences between reported and unreported findings) we assessed potential discrepancies between the outcomes that were specified prior to the study and those that were eventually reported.

Assessment of external validity

To determine the external validity of each of the included studies, we examined whether a study adequately described its eligibility criteria (in terms of age, sex, and diagnosis), the demographics of its sample, its study setting, its recruitment procedure, and the experimental manipulations that it used per experimental group.

Summaries of included studies

For more information about the included studies, see Appendix S1 which contains summaries of all the included studies. These summaries are structured according to those studies that were deemed eligible to address the first (k = 20) and second research question (k = 1). There are two points worth noting about these summaries. First, they only include descriptions of those results that were relevant for the current research questions. As such, these summaries may contain less information than provided in the original study reports. Second, whenever it is mentioned that there is a difference between groups, this denotes an absolute and not a statistically significant difference.

Qualitative synthesis: source, study, task and sample characteristics

Quantitative synthesis: vote counting

To address Research Question 1 (“Is there evidence for the rule-based insensitivity effect in adults?”) votes were only cast for the 11 out of the 20 studies that included a no-instructions group as a comparison group. These votes indicated that the results of each of these 11 studies were positive. No judgments could, however, be made for the one study that was relevant for addressing Research Question 2 (“Do adults suffering from psychological problems demonstrate larger levels of the RBIE compared to their non-clinical counterparts?”), because this study did not include a no-instructions group. For an overview of the vote-counting results for both research questions see Table 4.

Assessments of risks of bias

Most of the included studies did not report the necessary information to assess all relevant domains of risks of selection, performance, exclusion, and detection bias. Nevertheless, the following can be said about those study aspects that we could draw conclusions about. Of the eleven out of the 21 studies that used a no-instructions group as a comparison group, none assessed the possibility that this group followed similar rules as the rule groups during the experiment. As a result, it could be that in these studies participants were misclassified to experimental groups. That is, there remains a possibility that participants were inaccurately thought to belong to a comparison group while in fact their behavior was actually governed by rules similar to those manipulated in the experimental groups. Furthermore, for the remaining domains, we argued that there were low risks of bias. Indeed, we argued that there was a low risk of reporting bias, seeing as there was a correspondence between the outcomes that were specified prior to the study and those that were actually reported. With respect to standardization of the experimental contexts, we argued that there was a low probability that the experimental groups were treated differently (performance bias). We also argued that there was a low probability that the methods that were used to assess the study outcomes were invalid or unreliable, and that the experimental groups differed with respect to how these outcomes were assessed (detection bias). See Appendix S2 and S3 for an overview of the judgments that were made for each aspect or domain of a study that could lead to a risk of bias.

Assessment of external validity

The majority of those included studies that were relevant for examining our first research question (“Is there evidence for the rule-based insensitivity effect in adults?”) (k = 20) did not report all relevant demographics (i.e., mean age, sex, and education level) of their samples (k = 13) nor their recruitment procedure (k = 13). Most of these studies (k = 16), however, explicitly described the setting in which the experiment took place, and all of them provided a detailed description of the experimental manipulations per group. The one study that was relevant for examining our second research question (“Do adults suffering from psychological problems display a larger RBIE compared to their non-clinical counterparts?”), selected participants based on the presence or absence of sub-clinical symptoms of depression, reported the eligibility criteria that they used, the demographics of their sample, and the experimental manipulations per group. Nevertheless, this study did not provide information about the experimental setting nor the procedure used to recruit participants.

Evidence for the RBIE

Remarkably, only 11 out of the 20 studies that were deemed relevant for addressing our first research question (“Is there evidence for the rule-based insensitivity effect in adults?”) were eligible for vote-counting, because they included a no-instructions group (as a comparison group). Of these studies, the results showed that after the contingency change, the rule groups were more inclined to demonstrate behavior that was reinforced before the change, compared to their non-instructed counterparts. At first glance, this seems to suggest that when adults are asked to follow initially accurate rules, they experience more difficulties adapting to changes in contingencies (compared to when they are not asked to follow such rules). Nevertheless, the risk of bias assessments showed that such a conclusion may be premature because none of the 11 included studies assessed whether their no-instructions groups functioned as adequate comparison groups. That is, none of these studies examined if, during the experiment, participants in their comparison group did not follow rules about the task-contingencies that were similar to those followed by the rule groups. As a result, this made it difficult to determine whether the effects that were observed in the rule groups could be attributed to the rules or instructions that were manipulated in those experiments.

Despite the fact that we found preliminary evidence for the RBIE in all 11 studies, it is important to acknowledge that there might be variables that increase or decrease the likelihood of observing this effect. For instance, according to past work, the RBIE might be less likely to occur if the experimenter is not physically present (e.g., Kroger-Costa & Abreu-Rodrigues, 2012), participants are provided with inaccurate as opposed to accurate instructions before a contingency change occurs (e.g., Hojo, 2002), and if the consequences for behaving in line with the actual task-contingencies outweigh those of following the rule (Donadeli & Strapasson, 2015). Unfortunately, a systematic examination of potential moderators of the RBIE (besides the moderating impact of the absence/presence of psychological problems) was beyond the scope of this systematic review. Nonetheless, we deem such an examination vital as it might further our understanding of the robustness of this effect. As such, we recommend that future work systematically examines those variables that might decrease or increase the RBIE.

Psychological problems and the RBIE

Despite the key role that the RBIE has been argued to play in psychological problems, only one of the included studies was deemed relevant for examining this idea. However, given that this study did not include a no-instructions group, no judgments could be made about the extent to which evidence was found for the RBIE and whether psychological problems moderated this effect. This suggests that there is currently no evidence available to draw firm conclusions about the relationship between psychological problems and the RBIE in adults. Furthermore, even if we evaluated the peer-reviewed journal articles (n = 69) which examined the RBIE but were omitted because they: (a) used samples smaller than 10, (b) samples from non-adult populations, (c) used non-experimental designs, and/or (d) did not manipulate rules or include a contingency change, we still failed to identify many relevant studies. Indeed, such a revised search only resulted in an additional four studies: two studies that investigated the impact of sub-clinical depressive symptoms in adolescents (McAuliffe, Hughes & Barnes-Holmes, 2014) [Experiments 1 and 2]), one study that examined that of ADHD in children (Kollins, Lane & Shapiro, 1997) and another study that examined that of schizophrenia in samples smaller than 10 (Monestès et al., 2014). We, therefore, strongly recommend that more work is conducted on the relationship between the RBIE and psychological problems to better inform clinical theory and treatment.

When carrying out such work, researchers should also explore certain variables that could moderate this effect in clinical groups. For instance, it might be that clinical groups (e.g., arachnophobics) are more insensitive to contingency changes if they follow pathology-relevant (e.g., “If you want to remain alive, always avoid places where there could be spiders”) but not pathology-irrelevant rules (e.g., “to gain points press the blue button”). Likewise, it is possible that different clinical groups (e.g., people suffering from psychosis vs. depression) demonstrate different levels of the RBIE because of differences in the origins (generated by imaginary agents vs. self-generated) of the rules they follow. Another possibility is that variations in the elements of the rules (i.e., the described stimuli [all spiders vs. tarantulas], responses [avoiding spiders vs. attacking them], and contexts [all spider habitats vs. the basement]), might contribute to differences in how people suffering from similar conditions (e.g., arachnophobia) adapt to contingency changes. We believe that such an endeavor would be useful because it could aid clinicians in developing more targeted treatments.

Operationalization of the RBIE

Our coding of task and study characteristics revealed that although most of the included studies used similar tasks and procedures, the precise parameters that were involved often differed. Specifically, many studies used conditional discrimination tasks during which participants could initially gain points if they followed the rules they received from the experimenter. In most of these studies, the task-contingencies were subsequently altered after a number of trials so that the previously effective rules were rendered ineffective. To illustrate, consider Kissi et al.’s (2018) Matching-To-Sample (MTS) task. This task consisted of two experimental phases. On every trial, participants were presented with four images. One image—called the ‘sample stimulus’—was presented at the top of the screen and always consisted of three identical symbols or letters (e.g., TTT). Three other images—called the comparison stimuli—were presented at the bottom of the screen. One of these images had two symbols or letters that were identical to the sample stimulus (e.g., TT%; most-like comparison stimulus), another had one symbol or letter identical to the sample stimulus (e.g., T%%; moderate-like comparison stimulus), while the third had no symbols or letters in common with the sample stimulus (e.g., %%%; least-like comparison stimulus). During the first phase of the experiment, participants could obtain points if they selected the comparison stimulus that was most-like the sample stimulus. However, during the second phase of the experiment, the task-contingencies were changed. Now, participants gained points whenever they selected the comparison stimulus that was least-like the sample stimulus. To examine the RBIE, some participants were given instructions telling them how to gain points in the task, whereas others had to learn about the task-contingencies via trial-and-error. This task is a conditional discrimination task because reinforcement for responses was conditional upon the characteristics of the sample stimulus.

Critically, despite the fact that most included studies used similar tasks, the precise stimuli (e.g., tones vs. images) that were used, the point in time in which the contingency change occurred (e.g., after two vs. three blocks), and the study outcomes (e.g., latencies vs. rate or accuracy of responding) often differed between studies. Generally speaking, if reliable evidence is found for a phenomenon, such variations are often viewed as a potential advantage because they enhance the generalizability of a study’s findings. Yet given that, in our opinion, it is unclear whether the RBIE was adequately assessed in any of the included studies in this review, we believe that this idea cannot be applied to our findings (see the previous sections “Evidence for the RBIE” and “Psychological Problems and the RBIE”).

External validity and risks of bias

The results revealed that many studies did not report all relevant demographics of their samples, how they were recruited, if the contingency changes were announced, and if the experimenter was present during the experiment. In addition, no study provided sufficient information to assess all domains of potential risks of bias. Taken together, this suggests that the reports of the included studies did not provide sufficient information to evaluate all coding items assessing their external and internal validity. The lack of such information is particularly problematic in the context of systematic reviews because it limits the conclusions that can be drawn from it. As such, we strongly recommend that, in future work, researchers report all information about their study that may enable readers to more readily draw conclusions about its external and internal validity (see Schulz, Altman & Moher, 2010 for guidelines).

Other considerations

In many of the studies, there was the implicit assumption that when people were asked to follow accurate rules, their behavior would be exclusively governed by those rules, and that if this was not the case, their actions would be exclusively guided by the task-contingencies. We would argue that such a reasoning might be problematic for two reasons (for similar arguments see Hayes et al., 1986). First, previous work suggests that when humans are not provided with rules they rarely demonstrate purely contingency-shaped behavior. Instead, they often generate and use their own rules about how they should behave in a particular context, based on their (trial-and-error) experiences in that context (Rosenfarb et al., 1992; Shimoff, Matthews & Catania, 1986). Second, such an interplay between environmental contingencies and rules may have also impacted the behavior of the rule groups that were described in the reviewed studies. Indeed, a closer look at the results of these studies showed that when behavior was considered rule-governed, it was rarely ever the case that participants consistently stuck to the rules they were told to follow. Rather, the results suggest that participants sometimes engaged with the task in ways that were not specified by these rules. There could be two possible explanations for this finding. A first possibility is that these deviations from the rules were unintentional and as such reflected erroneous responding. A second possibility is that instances in which participants discarded the rules that they were told to follow, actually constituted intentional attempts to explore instead of exploit the task-contingencies (Berger-Tal et al., 2014).

If the latter possibility is valid as well as the possibility that rules governed the behavior of the no-instructions groups, then this might suggest that comparisons between instructed and non-instructed groups might not inform us about the effects of rule-governed vs. contingency-shaped behavior per se. Indeed, such comparisons might then rather inform us about the relative degree to which socially-provided rules vs. environmental contingencies and self-generated rules vs. environmental contingencies influenced the behavior of the instructed and non-instructed groups, respectively. Yet given that we could not assess the plausibility of this assertion in the current study, this idea remains speculative. We, therefore, recommend that future work examines its validity so that we can gain a better understanding of how the RBIE should be conceptualized (e.g., as an insensitivity of behavior to other contingencies due to a stronger reliance on socially-generated rules than environmental contingencies).

Finally, to the best of our knowledge, there is currently no consensus about how contingency insensitive and sensitive behavior should be measured. Indeed, if anything, the implicit assumption is that behavior is contingency insensitive if it is not in line with a contingency, whereas it is contingency sensitive if it corresponds with a contingency. We believe that although such operational definitions can be useful in some respects, they lack the precision that is needed to measure these behaviors in a uniform and unambiguous manner. Indeed, given the broad and descriptive nature of these definitions, much variation can exist between studies in how they measure contingency sensitive and insensitive behavior. We believe that, although this is not an issue per se, it can become problematic when one wants to draw general conclusions across studies. We, therefore, recommend that future work offers more precise operational definitions of contingency sensitive and insensitive behavior.

Limitations

Several factors should be taken into account when interpreting our results. First, to determine whether or not behavior was in line with a previously effective rule and/or a novel contingency we used a liberal criterion. That is, we considered participants’ behavior to be in line: (1) with a previously effective rule if they demonstrated behavior that corresponded with this rule on at least a few trials, and (2) with a novel contingency and/or rule if they always behaved in line with this contingency and/or rule. As a consequence, it possible that if a different criterion were used, other findings would have emerged. Second, we opted for vote-counting for our quantitative research synthesis, which unlike the standard meta-analytic approach does not provide information about the magnitude of the observed effects (Koricheva & Gurevitch, 2013). Nevertheless, to gain some insight into these effects, we conducted a random effects model meta-analysis using those studies that reported sufficient statistical information. This analysis was based on six studies including a total of 377 participants (i.e., Haas & Hayes, 2006; Harte et al., 2017 [Experiment 2], Kissi et al., 2018; Kudadjie-Gyamfi & Rachlin, 2002; Monestès, Greville & Hooper, 2017; Monestès et al., 2014). It revealed a significant effect size of .76 (Cohen’s d for independent samples; 95% CI [.41–1.12]; p < .001) indicating that participants had far more difficulties adapting to a contingency change if, prior to the change, they received a rule as opposed to no rule. Third, across all studies that were deemed eligible for vote-counting, preliminary evidence was found for the RBIE. This was surprising, given that, in general, the likelihood of observing the same effect across all studies in a systematic review is rather low (Thornton & Lee, 2002). Usually, when such an overrepresentation of positive effects is observed, it is assumed that this might be due to publication bias, i.e., journals’ preference for publishing positive over negative findings (Joober et al., 2012; Thornton & Lee, 2002). Publication bias is particularly problematic in the context of systematic reviews, because it can lead to an overestimation of the existence of a particular effect. Therefore, we recommend the reader to take this bias into account when interpreting the findings of our systematic review. Finally, we adopted pre-defined inclusion and exclusion criteria which inevitably limited the scope of the review and as such the potential conclusions that can be drawn from it. For instance, we only considered peer-reviewed journal articles that examined one instance of the RBIE and one potential moderator of this effect in adult populations. Similarly, we only included experiments with groups that contained at least 10 participants, which led us to discard naturalistic studies and studies that adopted a single-subject methodology.