Development and content validity of the evaluation of multidimensional functioning and risks in aging scale

Search and synthesis of literature

A literature review was conducted in the PubMed and Google Scholar databases (in English and Spanish) with the following objectives: (1) identify previous tools for assessing multidimensional frailty; (2) identify another set of tools that assesses physical function, cognitive function, emotional status, and social status; (3) identify the measurement procedure for these variables; (4) analyze which of these assessments present a cross-sectional or longitudinal association with the risk of health-related adverse events (Fierro-Marrero et al., 2025).

Preliminary draft of evaluation of multidimensional functioning and risks in aging scale

Based on the information previously gathered, the selection of items for the first preliminary version of the EMFRA scale (EMFRA-P1) was conducted. This selection process was performed considering criteria of relevance and validity to ensure that the scale adequately reflected the constructs of interest in a functional evaluation.

The preliminary scale consisted of four assessment domains: physical function, cognitive function, emotional status, and social situation. Each of these domains included six items that enabled a multidimensional evaluation of the individual’s condition. The selection of these domains responded to the need to address the construct in an integrated manner, recognizing the interrelationship between various aspects of health and participant well-being.

We then defined the most appropriate assessment procedure for the preliminary scale. Three main modalities were considered: functional assessment through objective testing, self-administration by the participant, and hetero-administration by the evaluator. The choice of the type of administration was made based on the characteristics of the items and the ease of implementation in various evaluation contexts. Additionally, three response categories were established, which were based on the previously synthesized information, allowing for a simple and standardized measurement of participant responses.

This preliminary development was an important first step toward the construction of a robust functional assessment instrument, which enabled the identification of strengths and areas of need across various dimensions of human functioning, thereby contributing to a better understanding of the factors influencing functioning in the individuals evaluated.

Content validation by an expert panel

An expert panel of healthcare providers was formed, including 15 experts (Lynn, 1986). All experts were Spanish healthcare professionals, specialized in any field of knowledge. The inclusion criteria were deliberately broad to ensure a multidisciplinary perspective, reflecting the diverse professional backgrounds expected to use the scale (Davis, 1992; De Vet et al., 2011). Importantly, all panel members were independent from the development team, which enhances the objectivity of the validation process as recommended by COSMIN guidelines (De Vet et al., 2011). An online validation panel was conducted through the Cognitoforms platform for content validation of the EMFRA-P1.

The following information was extracted: (1) sociodemographic data, including age, sex, nationality, native language, country, territory or state of residence, academic level, profession, and current job; (2) years of clinical and research experience with older adults, other frail populations, managing assessment tools, and frailty assessment tools, due to their historical relevance in geriatrics; (3) the number of ongoing research projects related to frail older adults they were currently working on; and (4) the number of validation processes in which they had previously participated.

An initial presentation was provided, outlining the justification, objectives, and scope of the study, as well as informing participants about the study procedures through the informed consent process. The informed consent ensured that participants were fully aware of their role, the purpose of the research, and the voluntary nature of their involvement, following ethical guidelines for research with human subjects (Gupta, 2013).

The validation panel had the objective of systematically assessing several aspects of the preliminary version of the EMFRA-P1. Specifically, the panel focused on evaluating the following components:

1. Comprehension

1. Item/question understanding: Evaluators were asked to determine whether each item or question was clearly phrased, easily understandable, and unambiguous. This step was crucial for ensuring that all respondents, regardless of their background, could consistently interpret the items in the intended manner. Effective comprehension minimizes measurement error and improves the reliability of self-reporting (Beatty & Willis, 2007).

2. Score/response option understanding: This evaluation addressed whether participants could clearly understand the available response options for each question. It was necessary to determine whether the scoring was transparent and appropriately mapped to the construct being measured.

2. Measurement procedure suitability

1. The appropriateness of the measurement procedures was assessed by considering whether the proposed approach for each item (e.g., self-administration, hetero-administration, or functional testing) was feasible, valid, and suited to capturing the construct of interest. The evaluation procedure included a discussion on practicality in terms of the time required for administration, participant burden, and the potential for each procedure to yield accurate and reliable measurements of the domains being investigated.

3. Coherence

1. Item/question coherence regarding multidimensional functioning: Each item was scrutinized to determine whether it adequately captured elements of multidimensional functioning. Physical, cognitive, emotional, and social domains, each contribute to an individual’s overall resilience or vulnerability (Junius-Walker et al., 2018). The panel evaluated whether the items effectively represented these components and whether any domain required further elaboration.

2. Score/response option coherence: The evaluators also assessed whether the scoring or response options were logically aligned with the respective item. For example, a question assessing physical function should include response options that meaningfully capture various levels of ability, or an item assessing task difficulty should be answered through difficulty category levels.

4. Relevance

1. Relevance of items for assessing multidimensional functioning: Relevance was considered in terms of how well each item contributed to the overall goal of assessing functioning across physical, cognitive, emotional and social dimensions. Items needed to reflect critical indicators of functioning, ensuring they collectively offered a comprehensive assessment of the construct. This step was intended to maintain the content validity of the scale, ensuring the instrument measures what it purports to measure (Haynes, Richard & Kubany, 1995).

To systematically capture the feedback from the validation panel, a 5-category Likert scale of agreement was used. The categories were as follows:

• “Totalmente en desacuerdo” (strongly disagree)

• “En desacuerdo” (disagree)

• “Ni de acuerdo ni en desacuerdo” (neither agree nor disagree)

• “De acuerdo” (agree)

• “Totalmente de acuerdo” (strongly agree)

This scale provided a nuanced view of the degree to which panel members concurred with the clarity, coherence, and relevance of each item, response option, and measurement procedure.

In addition to the Likert scale, the panel members were asked to respond to open-ended questions designed to gather suggestions, comments, and recommendations for modifications. These open-ended questions were critical for capturing insights beyond the structured assessment, allowing experts to provide detailed feedback on potential improvements. This approach aligns with the iterative nature of instrument development, in which continuous refinement based on expert input leads to greater validity and reliability of the final tool (DeVellis, 2017).

The feedback from the panel provided a comprehensive basis for refining the EMFRA scale, ensuring that the final version would be both scientifically robust and practically applicable for assessing functioning in diverse populations. The iterative process of validation and feedback is an essential component of psychometric scale development, emphasizing both empirical rigor and responsiveness to stakeholder perspectives.

Analysis and modification procedures after the expert panel

The level of agreement across authors for each of the previous points was explored with Aikens’ V (23): ${\displaystyle Aike{n}^{\prime }sV=\frac{S}{\left[n\left(c-1\right)\right]}.}$ The number of ordinal categories is denoted “c”, the number of evaluators “n”. “S” refers to the sum of the scores of the total number of evaluators. To calculate “S”, the score for the category with the level of agreement selected “ci” is subtracted from the value of the category with the lowest level of agreement, “c_low”. ${\displaystyle S=\sum _n\left(c_i-c_{low}\right).}$ A Likert scale for Level of Agreement was categorized with the following scores:

• “Totalmente en desacuerdo” (strongly disagree) = 1 point (c_low)

• “En desacuerdo” (disagree) = 2 points

• “Ni de acuerdo ni en desacuerdo” (neither agree nor disagree) = 3 points

• “De acuerdo” (agree) = 4 points

• “Totalmente de acuerdo” (strongly agree) = 5 points

A low level of agreement across authors was considered when V < 0.7. Items were excluded if at least one question presented a low level of agreement across authors. Only items with all questions presenting a V ≥ 0.7 were maintained.

Maintained items were modified based on the suitability of the suggestions retrieved by the authors.

The list of suggestions made by every author was reported, indicating which suggestions were considered suitable for conducting modifications. A second preliminary version of the scale was named EMFRA-P2, highlighting the modifications conducted, which is presented Supplementary Material. EMFRA-P2 is laid out for administration to clinicians and older adults in cognitive interviews.

Content validation by clinicians and older adults

The EMFRA-P2 was administered and validated by clinicians and healthy older adults, employing cognitive interviews (Estefania & Zalazar-Jaime, 2018). This process aimed to assess the comprehension and coherence of items for both clinicians and older adults, for their respective implication in administration and evaluation.

A total of 10 clinicians, and 10 healthy older adults were interviewed using a non-probabilistic sampling. This sample size aligns with established recommendations for cognitive interviewing for identifying major issues of interpretation (Beatty & Willis, 2007).

Interview methodology for clinicians. No specific selection criteria were applied to the participating clinicians, except that they were required to have a native or C2 level proficiency in Spanish from Spain. This criterion ensured that their comprehension and evaluation of the scale would not be hindered by language barriers, thereby allowing an accurate exploration of the scale’s comprehensibility and coherence regardless of their level of clinical expertise.

The following data were collected from the clinicians:

1. Sociodemographic information: This included age, sex, nationality, native language, country of residence, autonomous community of residence, academic level, profession, and current employment. Collecting these data allowed for the characterization of the clinician sample and an understanding of the potential influence of sociodemographic factors on their assessment of the scale.

2. Clinical experience: Information regarding their clinical experience was gathered, including years of clinical practice, years of experience working with older adults, years of experience with other populations identified as frail, and years of experience using assessment tests and tools. These data were crucial for understanding the clinicians’ familiarity and comfort with frailty-related concepts and tools, which could influence their ability to effectively evaluate the scale.

Before commencing the interview, the clinicians were provided with a detailed explanation of the administration procedure for the scale. This included specifying which items were functional tests, which required hetero-administration, and outlining the procedure for calculating the total score. By ensuring the clinicians fully understood the process prior to assessment, the validity of their feedback on the scale’s usability was optimized.

The interview then proceeded with the clinicians reading each item along with its associated scoring or response options. During this process, they were prompted to provide detailed feedback on any aspects that could be improved or modified to enhance the clarity, coherence, and applicability of the scale items. Their insights were particularly valuable, given their frontline experience in assessing frailty in clinical settings.

After reviewing all the items, the clinicians were asked to provide a general evaluation of the scale based on the following aspects:

• Perceived difficulty in administering the scale: The clinicians were asked to reflect on the ease or difficulty associated with administering the scale to the intended population.

• Perceived difficulty in calculating the total score: The complexity involved in scoring the scale was also assessed.

The clinicians provided their responses using a 5-point Likert scale of perceived difficulty:

• “Muy difícil” (very difficult)

• “Bastante difícil” (quite difficult)

• “Algo difícil” (somewhat difficult)

• “Poco difícil” (slightly difficult)

• “Nada difícil” (not difficult at all)

Interview methodology for older adults. The older adults were included based on the following criteria: (1) native or C2 level of Spanish from Spain; (2) age ≥65 years; (3) being cognitively healthy, considered when presenting ≥26 points on the Montreal Cognitive Assessment (MoCA) (Nasreddine et al., 2005; Sun et al., 2023). Subjects were recruited from La Salle Clinica Universitaria, Aravaca, Madrid, Spain.

Exclusion criteria included (1) any health condition posing risks when conducting physical tests or physical exercise (e.g., infection, heart failure, pulmonary hypertension, dilated cardiomyopathy); (2) inability to stand without the assistance; (3) inability to walk independently even with assistive walking devices; (4) inability to transfer from sitting to standing even with the use of assistive walking devices; (5) inability to read or understand the investigators’ commands; or (6) inability to observe with clarity.

The healthy older adults were asked for sociodemographic data, including age, sex, height, weight, body mass index, completed educational level, years studied, marital status, and employment status.

At the start of the interview, the older adults were informed that they would be administered the scale. The administrator of the scale (a physiotherapist) was in charge of conducting the interview. The older adults were required to provide commentaries for modification. Additionally, the interviewer would annotate whether participants were able or not to respond to every item. After the older adults had responded to the question, they were asked to employ the previous 5-category Likert scale of difficulty to score two questions:

• Level of difficulty for comprehending the task/question

• Level of difficulty for comprehending response options.

Analysis and modification procedures after the interviews

The interviews conducted with the clinicians and older adults were audio recorded to ensure an accurate and comprehensive capture of all feedback provided (Beatty & Willis, 2007). In addition to quantitative data on perceived difficulty, a qualitative content analysis was carried out to evaluate participants’ comments and suggestions in depth.

Following each interview, one investigator carefully re-listened to the audio recordings. All verbal comments related to the comprehension, clarity, coherence, and applicability of each item were transcribed verbatim. These comments were systematically categorized in a table, linking each piece of feedback to the specific item, scoring option or instruction to which it referred.

The analysis followed a manual, item-by-item coding strategy in which: (1) each comment was classified according to the type of issue it addressed (e.g., substitute a word for another, provide a certain type of specification, include examples, the question is long, etc.); (2) Comments were further assessed for relevance and frequency.

Although no qualitative software was used due to manageable volume of data, this structured procedure allowed for transparent tracking of participant input and supported decisions about item refinement.

In parallel, the quantitative indicators of perceived difficulty were analyzed using frequency distributions. Specifically, the frequency of responses indicating higher difficulty levels (i.e., “very difficult,” “quite difficult,” and “somewhat difficult”) was examined for the following key aspects:

• Level of difficulty in calculating the total score (clinicians): High levels of difficulty in calculating the total score would suggest the need for simplification or clarification of the scoring procedure to ensure practical implementation.

• Level of difficulty in comprehending the task/question (older adults): Difficulty in understanding specific tasks or questions would necessitate modifications to ensure that the scale items are accessible and understandable for older adults, promoting accurate and meaningful responses.

• Level of difficulty in understanding response options (older adults): Similar attention was given to difficulties reported in understanding response options, with the goal of refining these options to make them more intuitive and aligned with the cognitive abilities of older adults.

Based on this comprehensive analysis, a list of proposed modifications was generated. This list included detailed descriptions of the comments received, and it highlighted which suggestions were deemed appropriate for incorporation into the revised version of the scale. The modifications were carefully deliberated to maintain the scale’s psychometric integrity while enhancing its practical utility.

Following these refinements, the third and final version of the scale, named EMFRA-P3, was developed. This version reflects the iterative improvements made to enhance both content validity and usability. To facilitate transparency, the changes made to the scale are highlighted in the Supplementary Material, providing a clear account of the modifications undertaken.