Development and validation of a functional assessment tool for Chinese inpatient rehabilitation: insights from a Delphi study based on the International Classification of Functioning, Disability, and Health (ICF)

Expert selection

The Delphi method is a structured research approach employed to solicit expert opinions and establish consensus (Cai, 2022; Lee, Kim & Han, 2020). Experts were selected based on the following criteria: (1) involvement in clinical medicine, nursing, or health management; (2) holding a senior professional title or having ≥5 years of relevant experience; (3) prior experience with the ICF. Fifteen experts from 11 hospitals across four cities in Jiangsu Province were invited to participate.

Establishing an item pool

An item pool was established through an investigation of current clinical practices and a review of relevant literature. The research team chose to develop an item pool for a functional assessment tool by referencing the ICF-RS (Prodinger et al., 2016a).

Developing the initial questionnaire

The questionnaire is structured into three distinct parts.

The first part serves as an introduction, comprising a letter to the experts, background information regarding this study, and instructions for completion.

The second part constitutes the main body of the questionnaire, which consists of an evaluation form designed to assess the items within the evaluation tool using a 5-point rating scale to gauge the importance of each item in the item pool (1 = unimportant to 5 = very important) (Terwee et al., 2018). The process involves three rounds of meetings in-person, with the first round allowing experts to propose additional items if they identify any suitable items missing from the initial questionnaire.

The third part includes a form for collecting expert information and a survey at the end of the questionnaire. This survey allows experts to explain their basis for judging the importance of each ICF item and their familiarity with the project. This information is essential for calculating and assessing the expert authority coefficient.

The questionnaire will be distributed on-site during the expert consensus meeting and collected immediately upon completion.

Delphi expert consensus process

In this study, the Delphi method was utilized to identify items incorporated in the functional assessment tool based on the ICF-RS through three rounds of on-site meetings.

The criteria for selecting ICF items are as follows: (1) Remove items irrelevant to hospitalization and medical goals; (2) eliminate items that cannot be quantitatively assessed in a clinical setting; (3) retain representative items from those with similar functional dimensions; (4) exclude items for which no functional changes are expected during hospitalization.

Evaluation indicators

The evaluation indicators include the expert positive coefficient, expert authority coefficient, mean importance scores for each ICF item, and the coordination coefficient of expert opinions (Cai, 2022).

The expert positive coefficient quantifies the level of participation and enthusiasm exhibited by the experts during the meeting, as indicated by the questionnaire response rate (Lee, Kim & Han, 2020).

The expert authority coefficient, reflecting the basis for experts’ judgments concerning the content of the questionnaire and their familiarity with it (Chen et al., 2020). A coefficient of ≥0.70 signifies a satisfactory level of authority.

The mean importance score refers to the average rating of the importance of each item based on all returned questionnaires. In this consultation survey, a 1–5 scoring system is employed to evaluate the importance of each item in the functional assessment tool, with higher scores indicating greater importance (He et al., 2022). Generally, a mean importance score of ≥3.5 suggests that the item is acceptable and may be retained in the functional assessment tool.

The coordination coefficient of expert opinions is represented by the coefficient of variation (CV) for each indicator. A smaller CV indicates higher consistency in expert evaluations regarding the importance of the item, suggesting that experts’ judgments about the importance of that item are more unified (Zeng et al., 2023). It is generally accepted that a CV ≤ 0.3 denotes an acceptable level of coordination for that indicator.

Participants

Inclusion criteria: (1) Age ≥ 18 years; (2) patients receiving inpatient treatment in a rehabilitation department or rehabilitation hospital; (3) a clear disease diagnosis upon admission; (4) obtaining verbal informed consent from the patient.

Exclusion criteria: (1) Patients in critical condition or with unstable vital signs; (2) patients who are unable to undergo ICF assessment due to personal or other reasons.

This study received review and approval from the Ethics Committee of the First Affiliated Hospital of Nanjing Medical University (2020-SR-148) and has been registered with the China Clinical Trial Registration Center (No. ChiCTR2000034636).

Following the development of the functional assessment tool, we selected a total of 2,574 patients who met the aforementioned criteria from 12 rehabilitation medical institutions in Nanjing, Wuxi, and Suzhou between August 2021 and March 2023.

Application of the assessment tool and data collection

The Numerical Rating Scale (NRS), which operates on a scale of 0 to 10, serves as the assessment method, where 0 denotes no problem and 10 signifies a complete problem. To enhance objectivity and consistency in assessments derived from the evaluation tool during practical applications and to streamline evaluators’ operations, this study builds upon the Chinese grading assessment method of the ICF-RS (Gao et al., 2018). It further refines and develops specific evaluation criteria for each item, accompanied by a detailed evaluation manual available in the supplementary materials.

This study commissioned a relevant company to develop mobile online assessment software to facilitate data collection and unified management.

Prior to the commencement of formal data collection, evaluators from the participating research units are invited to undergo training and assessment in the ICF. Only those who pass the evaluation will be eligible to proceed to the data collection phase.

Assessing patients using the ICF at both admission and discharge in a rehabilitation inpatient setting yields a total score derived from the sum of individual item scores, which represents the overall functional status of the patients.

Statistics and analysis

All data analyses in this study were conducted using SPSS version 23.0.

Reliability refers to the consistency and stability of the assessment results produced by the measurement tool (Ahmed & Ishtiaq, 2021). This study utilized Cronbach’s alpha coefficient to assess the internal consistency of the functional assessment tool, with a Cronbach’s alpha value exceeding 0.75 generally regarded as indicative of good internal consistency (He et al., 2021).

Validity pertains to the ability of the measurement tool to accurately reflect the traits or concepts it aims to measure (Li et al., 2022). This study employed principal component analysis (PCA) with varimax rotation to assess the structural validity of the functional assessment tool. Prior to PCA, a Kaiser–Meyer–Olkin (KMO) test and Bartlett’s test of sphericity were executed on the scores of each item. A KMO value greater than 0.9 and a P value from Bartlett’s test of less than 0.001 suggest that PCA can be conducted. The number of components (factors) to retain was determined based on eigenvalues >1.0. To enhance the interpretability of the component structure and achieve a simple structure, the extracted components were subjected to orthogonal rotation using the varimax method. Items were considered to load significantly on a component if their rotated factor loading was ≥—0.40—. Items exhibiting cross-loadings (loading ≥—0.40— on more than one component) were assigned to the component that aligned most closely with their conceptual meaning and clinical relevance within the ICF framework and inpatient rehabilitation context, as determined by the research team.

Basic information about the experts

In this study, three iterative consensus rounds were conducted with 15 experts, all of whom participated in every session. Participants possessed a minimum of five years of clinical experience or senior professional titles, ensuring field proficiency. The experts were recruited from 11 hospitals across four cities in Jiangsu Province. Further details are provided in Table 1.

The first round of the Delphi process

Figure 1 summarizes the Delphi process for developing the functional assessment scale.

In the first round of the survey, the expert positive coefficient was 100%. A total of 10 ICF items were found to have a mean importance score <3.0 (Table S1). These items were subsequently removed and excluded from further rounds (b640 Sexual functions, d240 Handling stress and other psychological demands, d415 Maintaining a body position, d420 Transferring oneself, d455 Moving around, d470 Using transportation, d660 Assisting others, d770 Intimate relationships, d850 Remunerative employment, d920 Recreation and leisure). A total of 5 ICF items had a mean importance score ≥3.0 and <3.5, and these items were selected for detailed discussion in the next round (b710 Mobility of joint functions, d465 Moving around using equipment, d520 Caring for body parts, d570 Looking after one’s health, d640 Doing housework). Among the items with a mean score ≥3.5, one item exhibited a CV > 0.3, indicating a lack of consensus among the experts. This item was also marked for detailed discussion in the next round. No new items suitable for the assessment tool, not included in the initial questionnaire, were identified during this round.

The second round of the Delphi process

The second round of the questionnaire included items from the first round with a mean importance score of ≥3.0. Items scoring <3.0 in the first round were excluded from subsequent surveys. Items scoring ≥3.0 and <3.5 or with a CV >0.3 in the first round were the focus of discussion in this round.

In the second round, the expert positive coefficient remained 100%. Two ICF items had a mean importance score <3.0 and were directly removed (Table S2, d570 Looking after one’s health, d640 Doing housework). Three items scored ≥3.0 and <3.5 and remained under discussion for the next round due to their CV >0.3, which indicated insufficient consensus (b710 Mobility of joint functions, d465 Moving around using equipment, d520 Caring for body parts). All other items had a mean score of ≥3.5 and a CV ≤0.3.

The third round of the Delphi process

The third round included items from the second round with a mean importance score of ≥3.0. Items scoring <3.0 in the second round were excluded. Items with a score ≥ 3.0 and <3.5 or a CV >0.3 were discussed in detail.

In the third round, the expert positive coefficient was 100%. This round aimed to finalize the items for inclusion in the functional assessment tool. One item had a mean importance score <3.5 and did not meet the inclusion criteria, leading to its exclusion (Table S3, d520 Caring for body parts). Two items achieved mean importance scores of ≥3.5 but had CV >0.3, reflecting insufficient consensus (b710 Mobility of joint functions, d465 Moving around using equipment). Nevertheless, considering the need to cover as many functional dimensions as possible, these two items were retained. All other items scored ≥3.5 with a CV ≤0.3. After three rounds, 17 ICF items met the inclusion criteria and achieved consensus among the experts. These items are presented in Table 2. Finally, the functional assessment tool is named ICF-RS-17.

Expert authority coefficient

The experts’ average familiarity with the item content during this consensus meeting was 0.74, and their average judgment basis was 0.88. The final calculated authority coefficient was 0.81, indicating a strong level of authority in the results.

Reliability analysis

Using the ICF-RS-17, data were collected from rehabilitation inpatients.

Prior to calculating Cronbach’s α coefficient, the item-total correlations and inter-item correlations were examined to ensure they were positive and of moderate magnitude, satisfying a key assumption for internal consistency reliability.

Statistical analysis showed a Cronbach’s α coefficient of 0.950 for admission assessments and 0.961 for discharge assessments, both exceeding 0.9, indicating high reliability. Split-half reliability coefficients were 0.899 (admission) and 0.922 (discharge), demonstrating strong internal consistency. These findings confirm that the measurement results of this tool are reliable and stable. The classification of ICF items is presented in Table 3.

Validity analysis

KMO testing and Bartlett’s test of sphericity were conducted. For the admission assessment, the KMO value was 0.955, with a Bartlett’s test statistic of 40,745.826 (P < 0.001). For the discharge assessment, the KMO value was 0.958, with a Bartlett’s test statistic of 47,309.846 (P < 0.001). These results supported exploratory factor analysis for structural validity assessment.

In addition to conducting the KMO test for sampling adequacy and Bartlett’s test of sphericity, we evaluated several assumptions relevant to exploratory factor analysis. Our sample size (N = 2,574) significantly surpasses the commonly recommended thresholds. Each item was measured using a 0–10 NRS and treated as a continuous variable. Visual inspections of scatterplot matrices suggested generally linear relationships among the items. Moreover, the absence of extreme multicollinearity was confirmed, as indicated by the results of our multicollinearity assessment.

Principal component analysis with maximum variance revealed two factors with eigenvalues >1 for both assessments. Cumulative variance explained was 68.861% (admission) and 73.245% (discharge). Factor loadings of items ≥0.4 were analyzed. The rotated factor loadings (using varimax rotation) for each item are presented in Table 4.

Based on the pattern of high loadings (≥—0.40—), the components were interpreted and labelled as follows:

Component 1: Self-Care and Activities - This component comprised items primarily related to basic mobility, self-care tasks, and daily activities.

Component 2: Emotion and Mental Health - This component encompassed items primarily associated with emotional states, cognitive functions, and interpersonal interactions.

Two items, b620 Urination functions and b130 Energy and drive functions, exhibited significant loadings (≥—0.40—) on both components (Table 4). After careful consideration of their conceptual content within the ICF framework and their relevance to inpatient rehabilitation goals:

b620 Urination functions was assigned to Component 1 (Self-Care and Activities) as it pertains more directly to a fundamental bodily function impacting daily activities and self-care independence.

b130 Energy and drive functions was assigned to Component 2 (Emotion and Mental Health) as it reflects motivational states and energy levels, which are conceptually closer to emotional and mental well-being, and can significantly influence engagement in interpersonal interactions.