Internal structure of the action and acceptance questionnaire II (AAQ-II): evidence for a three-factor and bifactor model in two samples of university students

Introduction

Experiential avoidance (EA) refers to the recurring attempts to suppress unpleasant thoughts, emotions, memories, or bodily sensations, even when this leads to actions that are inconsistent with personal values and goals (Hayes et al., 1996, 2004). Although this regulatory strategy provides short-term relief (Nielsen, Sayal & Townsend, 2016), it is regarded as a transdiagnostic process associated with increased psychological distress, anxiety, depression, substance abuse (Mellick et al., 2019; Ruiz et al., 2013; Spinhoven et al., 2016), suicide ideation (Angelakis & Gooding, 2021; Chou, Yen & Liu, 2018), lower quality of life (Hayes et al., 2004), lower levels of wellbeing (Machell, Goodman & Kashdan, 2014), and interpersonal problems in clinical (Akbari et al., 2022) and general populations (Brereton & McGlinchey, 2019; Martínez-Rubio et al., 2023). This illustrates the importance of having access to measurement tools that can reliably and validly assess EA for research and clinical purposes, which is particularly relevant for Acceptance and Commitment Therapy (ACT; Hayes, Strosahl & Wilson, 1999).

The Acceptance and Action Questionnaire (AAQ; Hayes et al., 2004) has been one of the more often used tools for assessing EA, being adapted for multiple contexts and populations. Its psychometric properties, however, have been widely criticized (Rochefort, Baldwin & Chmielewski, 2017), showing inconsistencies concerning its factor structure and reliability indices when applied in different samples or contexts (Bond et al., 2011). To address the shortcomings of the original questionnaire, a seven-item revised version was developed by Bond et al. (2011), known as the Acceptance and Action Questionnaire II (AAQ-II).

In recent years, the AAQ-II has been regarded as the main instrument for measuring EA (Rochefort, Baldwin & Chmielewski, 2017; Mellin & Padrós, 2021), showing good psychometric properties and good stability across different groups (Bond et al., 2011; Ong et al., 2019a). This has contributed to its use in a wide variety of countries and settings; being translated and adapted for use in Spanish (Ruiz et al., 2013), Portuguese (Pinto-Gouveia et al., 2012), Greek (Karekla & Michaelides, 2017), and Chinese (Zhang et al., 2014) populations, among others. However, the evidence on both its construct and discriminant validity is not clear (Kashdan et al., 2020). Rochefort, Baldwin & Chmielewski (2017) found that the AAQ-II functions as a measure of neuroticism/negative affect, while Tyndall et al. (2019) reported that the AAQ-II seemed to assess a construct somewhere between experiential avoidance and distress, which limited its ability to properly discriminate. This highlights the current need for further exploration of the AAQ-II as a valid and accurate measure for assessing the underlying processes of ACT (McLoughlin & Roche, 2022).

In terms of its internal structure, one of the main attributes of the AAQ-II is its unidimensionality. The internal structure of the test suggests the existence of a single underlying factor, which has been consistently reported in several studies (Mellin & Padrós, 2021). However, these studies possess some methodological issues that need to be addressed. First, studies exploring the underlying factor structure of the measure have usually performed principal components analysis (PCA) instead of exploratory factor analysis (EFA). Although researchers commonly tend to use these analyses for the same purpose, both are conceptually distinct methods and thus have different aims. While the objective of PCA is to reduce data to the smallest possible set of components, EFA aims to identify underlying latent factors within the data (Alavi et al., 2020). Using PCA and EFA indistinctly or assuming that PCA is a method within EFA are common misconceptions that methodologists have addressed many times in the literature (Bandalos & Boehm-Kaufman, 2009; Costello & Osborne, 2005; Fabrigar et al., 1999; Schreiber, 2021).

Second, the recurring use of the Kaiser criterion as the method for determining the number of factors to be retained is currently viewed as an outdated strategy (Howard, 2015; Rogers, 2022). Under this method (Kaiser, 1960), all potential factors are examined and only those with eigenvalues equal or greater to 1 are preserved (this meaning that the retained factors should account for the equivalent variance of at least 1 item each). While this strategy is widely used for its simplicity, it is no longer recommended (Fabrigar et al., 1999; Howard, 2015; Rogers, 2022). Also, it has proven to have several weaknesses, such as over and underestimation of factors in presence of large or reduced sample sizes, number of items or factor loadings (Fabrigar et al., 1999; Horn, 1965; Ruscio & Roche, 2012; Zwick & Velicer, 1986). Due to this, alternative methods have been proposed for determining the number of factors in EFA.

One of the most robust and recommended methods is parallel analysis, which compares the observed eigenvalues with eigenvalues from multiple randomly generated datasets of the same size, aiming to retain only meaningful factors that explain more variance that would be expected from chance alone (Horn, 1965; Silverstein, 1987). This addresses the two main problems with the Kaiser criterion: Parallel analysis is less sensitive to sample size and number of items, as its comparison with random data reduces the likelihood of retaining factors based on mere chance or sampling errors, thus helping to identify factors which reflect more structure that would be expected under random conditions (Hayton, Allen & Scarpello, 2004; Zwick & Velicer, 1986).

Nevertheless, factor retention methods such as Kaiser criterion are still being widely used by researchers. This has also been the case for validation studies of the AAQ-II, where this technique has been consistently employed. Table 1 presents a brief review summarizing previous studies examining the internal structure of the AAQ-II and their methods. We conducted two studies. The first was aimed at exploring the underlying factor structure of the measure, also seeking to address the methodological inconsistencies existing in the literature by employing more robust and updated statistical techniques along with replicating findings from prior studies. The second study was aimed to confirm the validity, reliability, and overall psychometric properties of the AAQ-II. Both studies were conducted in two different samples of Spanish-speaking university students. We discuss our results in terms of their methodological and practical implications for the use of the AAQ-II as an assessment tool in future research and clinical practice.

Materials and methods study 1

Results study 1

Materials and methods study 2

Results study 2

Confirmatory factor analysis

Our findings indicate that, when comparing both, the three-factor structure demonstrated a better fit all-round when directly compared to the single-factor structure in terms of absolute, relative and parsimony indices. However, neither of the structures showed an ideal fit in terms of the χ²/df ratio or RMSEA. The bifactor model (Fig. 1) showed excellent fit indices overall when compared with the three- and single-factor structures. It also showed better model parsimony overall and good χ²/df ratio and RMSEA. Results for all the models tested are detailed in Table 3.

Table 3:

Confirmatory factor analysis and bifactor analysis results.

	Absolute fit					Relative fit			Parsimony
	χ²	df	χ²/df	RMSEA	SRMR	CFI	TLI	GFI	AIC	BIC
Three-factor	97.350	11	8.850	0.097	0.019	0.980	0.961	0.987	19,390.632	19,503.947
Single-factor	472.335	14	33.738	0.199	0.046	0.892	0.839	0.944	19,760.069	19,792.531
Bifactor	77.898	21	3.709	0.110	0.018	0.983	0.950	0.975	19,372.776	19,471.926

DOI: 10.7717/peerj.19620/table-3

Note:

RMSEA, Root mean square error of approximation; SRMR, standardized root mean square; CFI, comparative fit index; TLI, Tucker-Lewis index; GFI, goodness of fit index; AIC, Akaike information criteria; BIC, Bayesian information criteria.

As previously stated, traditional fit indices may tend to overestimate the fit of bifactor models due to the increased number of estimated parameters, thus providing a potentially biased representation of the model’s true performance. Calculation of Omega total and ECV indices for the general factor (GF) supported this, with high values suggesting a dominance of this factor within the proposed model (ω_Total = 0.950; ECV_GF = 0.810). Calculation of specific bifactor indices also follow this line, with the Omega hierarchical coefficient suggesting that the GF captures most of the variance (ωh_GF = 0.888), and S-ECV values indicating that the independent factors explain a reduced amount of variance overall (S-ECV_PM = 0.292; S-ECV_ED = 0.128; S-ECV_AC = 0.172). In terms of determinacy and construct replicability, the three independent factors show moderate values, reflecting that the items are properly associated with the underlying constructs measured, and these three independent factors may not be as consistent or stable as expected (FD_PM = 0.770; H_PM = 0.384; FD_ED = 0.723; H_ED = 0.265; FD_AC = 0.594; H_AC = 0.219).

Discussion

We conducted two studies to provide evidence about psychometric properties of the AAQ-II in two samples of university students in Chile. Two factor structures for the AAQ-II were examined through CFA: The three-factor structure based on our analysis and the single-factor structure based on replication of methods by previous studies. In addition, a bifactor model was tested including a general factor (experiential avoidance) within the three-factor model. We found that a three-factor structure yielded the best fit and structural validity, also showing good reliability and convergent validity. These findings contrast prior research reporting an unidimensional structure of this measure (Bond et al., 2011, Pinto-Gouveia et al., 2012; Ruiz et al., 2013) but are in line with recent studies that have examined the underlying factor structure through currently recommended statistical procedures (Pennato et al., 2013; Spencer et al., 2022; Arias, Barraca & García, 2023). Additionally, the bifactor model showed better fit indices overall when compared with the three- and single-factor structures.

Based on our results, the bifactor structure provided a better representation of the underlying latent construct measured by the AAQ-II. This proposed model explained a higher amount of variance and had a better fit all-round when compared to the commonly assumed unidimensional model. Three-factor structures have been also found by Pennato et al. (2013), and Spencer et al. (2022). The former reported a three-factor model accounted for 64% of the variance, compared with the 42% accounted for by the unidimensional solution. The latter briefly examined a three-factor structure in which one factor was composed by a single item. Similarly, Arias, Barraca & García (2023) observed a three-factor configuration that accounted for 69.3% of the variance, compared with the 25.5% accounted for in the unidimensional model. In addition, studies performing confirmatory analyses for the single-factor structure tended to report poor fit indices overall (See Table 1). While these structures tend to slightly differ, a factor composed by items 1 and 4 remains constant. It is also worth mentioning that when using currently unsuitable methods to estimate the number of underlying factors (Gaskin & Happell, 2014; Watkins, 2018), we were able to replicate the unidimensional configuration in our sample.

The three-factor structure found by Pennato et al. (2013) and Arias, Barraca & García (2023) showed almost the same item distribution observed in our sample. The only difference was item 5 (“emotions cause problems in my life”), which the authors found to belong to the second factor, along with items 2 (“I’m afraid of my feelings”) and 3 (“I worry about not being able to control my worries and feelings”). Regarding this, Pennato et al. (2013) reported this particular item to show low loadings on all factors. Our analysis also revealed that it showed the lowest factor loading, thus it is possible that it possesses certain ambiguity and may require further examination. A complementary examination through CFA (see Table S2) allowed us to confirm two things: First, that the three-factor structure we propose in this study demonstrates better fit when compared to previous approaches. Second, that any of the three-factor structures shows better fit when compared to the unidimensional model. Our results from the bifactor analysis also demonstrated better fit overall, also suggesting the importance of understanding the AAQ-II not simply as a unidimensional solution, but a more complex structure with a general factor able to encompass the construct of experiential avoidance.

It should be noted that, while the unidimensional structure was initially reported and proposed by Bond et al. (2011), who followed the same statistical procedures we used, their findings were mainly based on a 10-item version that showed to have two-factors. Their second factor, which was composed of three items, was removed due to explaining only 4.94% of the variance. However, the authors did not examine the underlying structure based on these seven items only. On the other hand, most subsequent studies did not follow the same procedures as the original authors, and either employed different data analysis strategies, or simply assumed a single factor structure without further examination.

Our findings add new insights for a more nuanced description of the construct and suggest that EA, as assessed by the AAQ-II, might not be unidimensional but a construct encompassing multiple processes, as previously stated (Hekmati et al., 2021). The content of each factor was quite similar to those reported by Arias, Barraca & García (2023) in adults from Ecuador. Although all factors might serve the same psychological function (i.e., experiential avoidance), they seem to reflect slightly different facets of the latent trait. The first factor was linked to painful memories, which emphasize the potential negative impact of prior stressful events, experienced as life-affecting memories in the present. This could reflect the associations shown between EA and traumatic-based events (Orcutt, Reffi & Ellis, 2020). Interestingly, this factor contains item 4. When examined under the Item Response Theory—a psychometric approach that complements Classical Test Theory by assessing each item’s capacity to discriminate levels of the latent trait (Reise & Waller, 2008), it has been proven to be the most significant item for clinical purposes, given that it needs a higher level of the latent trait to be endorsed by participants (Langer et al., 2024b; Ong et al., 2019b). This means that this item can work as an indicator within the construct, showing a greater discriminative ability to identify participants exhibiting past memories that impact its present state. The second factor (composed by items 5, 6 and 7) was related to the interference of emotions in the desired life when compared to oneself or others. This factor contains items that reflect a general evaluation of life (item 6: “It seems like most people are handling their lives better than I am”) or general and unspecific consequences of emotions on it (item 5: “emotions cause problems in my life”). Moreover, item 6 has consistently shown the lowest discriminative capacity of the scale (Menéndez-Aller et al., 2021; Ong et al., 2019b), while item 5 has presented a good ability to discriminate EA (Menéndez-Aller et al., 2021).

The third factor, composed by items 2 and 3, refers to difficulties in self-regulation and explicitly addresses the fear of experiencing feelings and not being able to handle them, suggesting that coming into contact with some internal events can be an aversive experience that should be controlled. As stated by Ong et al. (2019b), items reflecting broad language, such as feelings or emotions have shown less discriminative capability of the latent trait than those using specific terms like worries or memories. Thus, these items might not be directly assessing the specific nature of EA and might be capturing variance from other related constructs. These results must be interpreted in the context of the existing debate in the literature about whether the AAQ-II does assess EA or other related constructs such as neuroticism, psychological distress or negative affect (Wolgast, 2014; Ruiz et al., 2013; Rochefort, Baldwin & Chmielewski, 2017; Tyndall et al., 2019). While our results showed that under recommended statistical methods the measure shows a replicable and coherent structure, further research is still needed to fully clarify its construct validity across both clinical and research contexts.

Concerning the convergent validity of the AAQ-II, our results support recent research showing associations between EA, depressive and anxiety symptoms (Akbari et al., 2022) and suicide ideation in university students (Chou, Yen & Liu, 2018; Hekmati et al., 2021). In addition, the negative associations between EA and perceived well-being fit with prior research revealing associations between EA and lack of meaning in life (Pavlacic, Schulenberg & Buchanan, 2021), negative affect and less enjoyment of life events (Machell, Goodman & Kashdan, 2014), and poorer quality of life in college students (Coutinho, Trindade & Ferreira, 2019). Overall, this evidence suggests good convergent validity for both the total scale and each subscale. Nevertheless, considering the evidence suggesting that EA could be a context-specific regulatory strategy (Machell, Goodman & Kashdan, 2014), further examination on the interactions with other psychological constructs is still needed.

Our finding revealing a three-factor and bifactor configuration allows for a better understanding of different facets of EA through a very brief assessment. This is in line with literature suggesting a more comprehensive assessment of EA (Spinhoven et al., 2016). This has potential clinical and research implications. An accurate distinction of which avoidance facet is prominent could be helpful to define specific therapeutic targets, objectives, and strategies. Moreover, this provides a parsimonious measure to assess associations among different facets of EA and psychopathological and wellbeing domains in different contexts and populations. In terms of its practical use, clinicians and counselors could continue to evaluate EA through a single score based on the general factor, keeping in mind that it encompasses three underlying dimensions rather than being the result of a unidimensional solution. We are aware that the AAQ-II has been used both as a measure of psychological inflexibility and experiential avoidance rather than experiential avoidance only. This is a current and relevant issue regarding its validity to accurately assess ACT psychological process (Kashdan et al., 2020; McLoughlin & Roche, 2022; Ruiz et al., 2024). Our findings provide additional insights to contribute to this scientific controversy.

Some limitations deserve mention. First, because of our cross-sectional design, we did not address the temporal stability of the scale. Second, we did not assess other symptoms, emotional regulatory processes and psychological distress potentially associated with EA. Third, we only included first year students and used a non-probabilistic sampling method. Therefore, our findings can be interpreted with caution in terms of their replicability to other populations. In addition, it should also be noted that as the present study was conducted in a Spanish-speaking population, culturally shaped response styles could be a relevant factor to consider. Research has shown that some culturally shaped response patterns linked to emotion expression can have an influence in factor loadings in self-report measures (Harzing, 2006; van de Vijver & Leung, 2001). Studies on the cultural invariance of the AAQ-II considering countries from different continents are still needed to address this.

Furthermore, it may rise as a concern that two factors of the proposed structure are composed by two items only, contrary to the usually recommended three-items per factor rule of thumb. However, researchers should be aware that this derives from a comment in Tabachnick & Fidell (2013), who only suggested that two item factors may struggle to accurately represent the underlying construct it aims to measure. If two items are deemed sufficient for this purpose, then two items may suffice (Worthington & Whittaker, 2006). Thus, the theoretical interpretability and meaningfulness of the factor should always be the most important retention criteria, regardless of the number of items included in each factor.

Conclusions

In conclusion, our results show a bifactor model for the AAQ-II, which contrasts with prior evidence of a unidimensional structure. According to the intended use of the scale, both a single score or split into its three dimensions are suitable for clinical and research purposes. We identified items that consistently have shown loading and interpretability issues. The scale seems to be composed of items with different levels of proximity to EA/psychological inflexibility, which needs to be taken into account when the construct is measured through this tool.

Supplemental Information