Psychometric properties of the Valuing Questionnaire in a Spaniard sample and factorial equivalence with a Colombian sample

Procedure

Participants in both samples responded to an anonymous online survey distributed through social media (e.g., institutional webpages, Facebook, etc.) through a snowball sampling procedure. Specifically, the researchers asked their contacts and participants to share the publication of the survey to reach more people. The research publication was not posted in specific groups (e.g., students’ or mental health groups).

Participants first provided informed consent by accepting the conditions described at the beginning of the survey that included that they were adults (i.e., the only exclusion criterion to participate being younger than 18 years). Afterward, the participants in Sample 1 completed the instruments presented above. Participants in Sample 2 responded to a similar survey, but only the VQ data are relevant for this study. Their respective instructions preceded each questionnaire. Participants were also emphasized that the survey was anonymous and that they could stop their participation anytime they wanted. The median time for the completion of surveys was approximately 15 min. When data collection was finished, global reports of the results of the surveys were sent to the participants who indicated that they were interested in them. Participants were not compensated for their participation.

The survey conducted to recruit Sample 1 was available for one year until at least 200 participants of both genders responded. As we planned to analyze measurement invariance across gender, this number was established because it is the minimum suggested, as a rule of thumb, for conducting confirmatory factor analyses (Kline, 2016).

Participants

Sample 1. This sample consisted of 846 Spanish participants (75.7% females) with an age range between 18 and 72 (M = 35.40, SD = 11.39). The participants’ relative education level was 0.1% no studies, 33% primary studies (i.e., compulsory education) or mid-level study graduates (i.e., high school or vocational training), and 65.6% were undergraduates or college graduates (1.3% did not indicate the educational level). Almost half of the participants (44.6%) reported having received psychological or psychiatric treatment in the past, but only 12.8% were currently in treatment. Also, 12.9% of participants reported using psychotropic medication.

Sample 2. This sample consisted of 724 Colombian participants (74.4% females) whose ages ranged between 18 and 88 (M = 26.11, SD = 8.93). The participants’ relative education level was 17.8% primary studies or mid-level study, and 82.2% were undergraduates or college graduates. Forty-five percent reported having received psychological or psychiatric treatment at some time, but only 8.4% were currently in treatment. Also, 5.4% of participants reported using psychotropic medication.

Instruments

Valuing Questionnaire (VQ; Smout et al., 2014; Spanish translation by Ruiz et al., 2021). The VQ is a 10-item, 7-point Likert (6 = completely true; 0 = not at all true), self-report instrument designed to assess general valued living over the last week. It assesses valued living in everyday language without referring to specific life domains. The VQ has two 5-item factors: Progress (i.e., enactment of values, including clear awareness of what is personally meaningful and perseverance) and Obstruction (i.e., disruption of valued living due to avoidance of unwanted experiences and distraction from values). The Spanish version of the VQ has shown good psychometric properties in Colombian samples, with alphas of .83 and .82 for Progress and Obstruction, respectively.

Acceptance and Action Questionnaire –II (AAQ-II; Bond et al., 2011; Spanish version by Ruiz et al., 2013). The AAQ-II consists of 7 items that measure experiential avoidance on a 7-point Likert-type scale (7 = always true; 1 = never true). The Spanish version of the AAQ-II has shown good psychometric properties (mean alpha of .88) and a one-factor structure in Spanish samples (Ruiz et al., 2013). In this study, the AAQ-II showed an alpha of .91. Strong positive correlations were expected between AAQ-II and VQ-Obstruction, whereas at least medium negative correlations were expected with VQ-Progress.

Cognitive Fusion Questionnaire (Gillanders et al., 2014; Spanish version by Ruiz et al. (2017a). The CFQ is a 7-item, 7-point Likert-type scale (7 = always true; 1 = never true) of general cognitive fusion. The Spanish version by Ruiz et al. (2017a) has shown excellent internal consistency (alpha of .92) and a one-factor structure. In this study, the AAQ-II showed an alpha of .93. As with the AAQ-II, strong positive correlations were expected between CFQ and VQ-Obstruction, whereas at least medium negative correlations were expected with VQ-Progress.

Depression, Anxiety, and Stress Scales –21 (DASS-21; Lovibond & Lovibond, 1995; Spanish version by Daza et al., 2002). The DASS-21 is a 21-item, 4-point Likert-type scale (3 = applied to me very much, or most of the time; 0 = did not apply to me at all) consisting of sentences describing negative emotional states: Depression, Anxiety, and Stress. The Spanish version of the DASS-21 has shown good internal consistency for all the subscales and a hierarchical factor structure with a second-order factor (Ruiz et al., 2017b). In this study, the DASS-21 obtained alphas of .95, .92, .87, and .86 for the total scale, Depression, Anxiety, and Stress, respectively. Strong positive correlations were expected between the DASS-21 and VQ-Obstruction, whereas at least medium negative correlations were expected with VQ-Progress.

Satisfaction with Life Survey (SWLS; Diener et al., 1985; Spanish translation by Atienza et al., 2000). The SWLS is a 5-item survey that evaluates self-perceived well-being through a 7-point Likert-type scale (7 = strongly agree; 1 = strongly disagree). Ruiz et al. (2019) found that the Spanish version of the SWLS showed good psychometric properties and convergent validity in a Spaniard sample. In this study, the SWLS showed an alpha of .89. Strong positive correlations were expected between the SWLS and VQ-Progress, and medium negative correlations were expected with VQ-Obstruction.

Statistical and psychometric analysis

Firstly, we analyzed the fit of the two-factor model of the VQ by computing confirmatory factor analyses (CFAs) with Sample 1 (i.e., the Spaniard sample). In so doing, we followed the procedure used in the Spanish validation of the VQ in Colombia (Ruiz et al., 2021). As previous CFAs of the VQ have shown a method effect in Items 5 and 7 responses (Smout et al., 2014), we decided to compare the fit of the two-factor model where the error terms between these items were allowed to correlate versus the two-factor model with no error correlations. The software LISREL© (version 8.71, Jöreskog & Sörbom, 1999) was used to conduct the CFAs. No missing values were found due to the collection method (i.e., online survey). Given the lack of multivariate normality in the data (multivariate Mardias’ test of skewness and kurtosis = 610.954; p < .001), we selected the robust maximum likelihood (MLR) estimation method with the covariance matrix and the asymptotic variance–covariance matrix. The robust estimation methods in structural equation modeling are known for overcoming to a great extent the problem of the presence of outliers (Yuan & Zong, 2013). Even so, we have also used z-scores and Mahalanobis distance and obtained virtually the same results with or without the few identified outliers.

We computed the Satorra-Bentler chi-square test and the following goodness-of-fit indexes for the two-factor model: (a) the comparative fit index (CFI), (b) the non-normed fit index (NNFI), (c) the root mean square error of approximation (RMSEA), (d) the parsimony normed fit index (PNFI), and (e) the standardized root mean square residual (SRMR). According to authoritative guidelines (e.g., Hu & Bentler, 1999), CFI and NNFI values above .90 represent acceptable models, and above .95 indicate a good fit to the data. Regarding the RMSEA, values below 0.08 represent an acceptable fit, and values below 0.05 constitute a good fit (Browne & Cudeck, 1992; Browne & Cudeck, 1993; Hooper, Coughlan, & Mullen, 2008). Concerning the SRMR, values of 0.08 represent a good fit, and values below 0.05 represent a very good fit to the data. Furthermore, higher PNFI values indicate a more parsimonious model. Lastly, we also calculated the p-value for test of close fit (PCLOSE), which computes a one-sided test of the null hypothesis that RMSEA is equal or lower than 0.05.

Secondly, we analyzed construct reliability and evidence of convergent and discriminant validity of the measurement model following the suggestions by Brown (2015) and Fornell & Larcker (1981). The composite reliability coefficient (CR) was computed to analyze construct reliability. CR values higher than 0.70 can be considered as high construct reliability and adequate internal consistency. The convergent validity of the measurement model was analyzed according to three criteria: (a) factor loadings should be statistically significant (standardized loadings estimates should be 0.40 or higher), (b) CR should be higher than 0.70, and (c) the average variance extracted (AVE) should be ≥ 0.50 for each subscale of the VQ. Lastly, discriminant validity was estimated according to the following criteria: (a) inter-construct correlations should be lower than 0.80, and (b) the square root of AVE ($\sqrt{AVE}$) of both factors of the VQ should be greater than the inter-construct correlations with any other factor.

Thirdly, we analyzed the measurement invariance across countries (i.e., Spain and Colombia) and gender. In so doing, we followed also the procedure used by Ruiz et al. (2021) in the Spanish validation of the VQ in Colombia. Specifically, we adopted the guidelines proposed by Jöreskog (2005) and Millsap & Yun-Tein (2004). Through additional CFAs, we tested metric, scalar, and strict invariances by analyzing whether the item factor loadings, items intercepts, and the variance of error of the items were invariant across countries and gender. The relative fit of four progressively more restrictive models was compared. Firstly, the multiple-group baseline model allowed the unstandardized factor loadings to vary across groups. Conversely, the pattern of item-factor loadings and the number of factors were the same across groups (configural invariance). Secondly, the metric invariance model was nested within the previous model and placed equality of factor loadings across groups (i.e., weak factorial invariance). Thirdly, the scalar invariance model was nested within the metric invariance model and confined the factor loadings and the items intercepts to be the same across groups (i.e., strong factorial invariance). Lastly, the strict invariance model was nested within the scalar invariance and assumed the variance of errors (i.e., indicator residuals) to be equal across groups. Following Kline (2005) indications, we did not place equality constraints on estimates of the factor variances because they differ across groups even when the indicators are quantifying the same construct in an equivalent manner. For the model comparison, we weighed the CFI, NNFI, and RMSEA indices between nested models. We chose the more constrained model (i.e., second model versus the first model, and third model versus the second model) following the criteria advocated by Cheung & Rensvold (2002) and Chen (2007): (a) the difference in RMSEA (ΔRMSEA) was lower than .01; (b) the differences in NNFI (ΔNNFI) and CFI (ΔCFI) were equal to or higher than −.01.

Fourthly, we analyzed the internal consistency of the VQ in Sample 1 with the MBESS package in R (Kelley & Lai, 2012; Kelley & Pornprasertmanit, 2016). We computed corrected item-total correlations to identify items that should be removed because of showing a low discrimination item index (i.e., values below .30). Then, we calculated coefficient alphas and McDonald’s omegas and provided percentile bootstrap 95% confidence intervals (CI). According to Nunnally & Bernstein (1994), values higher than .70 were considered acceptable, whereas above .80 were considered good. To determine what label would apply to the VQ subscales, we observed the 95% CI. Fifthly, we calculated Pearson correlations between the VQ and the remaining scales to assess evidence based on relationships with other variables using SPSS 25©. Correlations were interpreted according to the guidelines provided by Lenhard & Lenhard (2016): small correlation between .10 and .20, medium between .21 and .36, and strong correlations >.36. Lastly, we calculated descriptive data and explored differences in VQ scores across country and gender by computing independent sample t-tests.

Validity evidence based on internal structure

Dimensionality

Table 1 shows the results of the CFAs conducted in Sample 1. The fit of the two-factor model of the VQ was good according to the values of CFI (.98), NNFI (.97), and SRMR (0.053). The RMSEA value was 0.073, which indicates an acceptable fit. However, the test of close fit (PCLOSE) showed that the RMSEA value was significantly higher than 0.05 (p < .01). Thus, overall, the two-factor model showed an acceptable fit according to this goodness-of-fit index. The alternative two-factor model with correlated error terms in Items 5 and 7 did not significantly improve the fit. In conclusion, we chose the two-factor model due to its greater parsimony according to the PNFI (see Table 1). This was also the factor model selected in Ruiz et al. (2021) with Colombian participants. Figure 1 depicts the results of the completely standardized solution of the two-factor model in Sample 1.

Table 1:

Goodness-of-fit indexes of the two-factor model and the two-factor model with error terms allowed to correlate for items 5 and 7 in sample 1.

Goodness-of-fit indicators	Two factor-model	Two factor-model with error terms allowed to correlate (items 5 and 7)
RMSEA [90% CI]	0.073[0.063, 0.083]	0.073 [0.063, 0.084]
CFI	0.98	0.98
NNFI	0.97	0.97
SRMR	0.053	0.052
PNFI	0.74	0.71
S-Bχ2 (df)	185.790 (34)	182.686 (33)

DOI: 10.7717/peerj.12670/table-1

Notes:

CFI

Comparative Fit Index

VQ

Valuing Questionnaire

NNFI

Non-Normed Fit Index

RMSEA

Root Mean Square Error of Approximation

S-Bχ²

Satorra-Bentler Chi-Square Test

SRMR

Standardized Root Mean Square Residual

Psychometric quality of the items

Table 3 shows that all corrected item-total correlations of the VQ in Sample 1 were high. For VQ-Progress, they ranged from .52 to .71 in Sample 1, whereas for VQ-Obstruction from .51 to .70. Cronbach’s alpha coefficients were adequate for both subscales (VQ-Progress = .85, 95% CI [.83–86]; VQ-Obstruction = .84, 95% CI [.82–.86]). The values of McDonald’s omega coefficients were virtually the same (VQ-Progress = .85, 95% CI [.83–87]; VQ-Obstruction = .84, 95% CI [.82–.86]).

Validity evidence based on relationships with other variables

Table 4 presents the Pearson correlations between the VQ and the assessed constructs. Overall, the VQ showed correlations in the expected directions and size. Specifically, VQ-Progress showed strong negative correlations with experiential avoidance and cognitive fusion. Medium-size correlations were found between VQ-Progress and emotional symptoms measured with the DASS-21. Lastly, VQ-Progress correlated positively and strongly with life satisfaction. As expected, VQ-Obstruction obtained the opposite pattern of correlations. Specifically, VQ-Obstruction showed strong and positive correlations with experiential avoidance, cognitive fusion, and emotional symptoms. VQ-Obstruction correlated negatively and strongly with life satisfaction.

Scores on the VQ across countries and gender

No statistically significant differences were found on VQ scores across countries on VQ-Progress (Spain: M = 19.08, SD = 6.08; Colombia: M = 19.50, SD = 6.43; t(1568) = −1.35, p = .18) and VQ-Obstruction (Spain: M = 11.31, SD = 6.63; Colombia: M = 11.70, SD = 6.88; t(1568) = −1.16, p = .25). Regarding gender in Spaniard participants (i.e., Sample 1), there were no statistically significant differences neither for VQ-Progress (Men: M = 18.72, SD = 6.13; Women: M = 19.14, SD = 6.08; t(811) = −0.80, p = .43) or VQ-Obstruction (Men: M = 11.14, SD = 6.60; Women: M = 11.39, SD = 6.69; t(811) = −0.43, p = .67).