A measurement framework to assess software maturity models

Software process maturity models

Several maturity models have been created following the structure and principles of CMMI. Examples of these models include:

• Software Engineering Maturity Model (SEMM) (Garzás et al., 2013): The AENOR workgroup has proposed the Software Engineering Maturity Model. SEMM is a process maturity model based on the ISO/IEC 12207 standard (ISO, 2008). The main goal of this model is to assess the maturity of software processes in small enterprises by examining capacity levels corresponding to different maturity levels.

• IT Service Management Maturity Model (Picard, Renault & Barafort, 2015): Picard, Renault & Barafort (2015) introduced the IT Service Management maturity model. This model is intended to evaluate IT infrastructure processes and aligns with ISO/IEC 20000-1 standards (ISO, 2011). Organizations can adopt this model to evaluate and enhance their IT service management capabilities, leading to improved service delivery.

• Test Maturity Model Integration (TMMI): The TMMI model, introduced by the TMMI Foundation (van Veenendaal & Wells, 2012), aims to enhance testing processes within organizations (Garousi & van Veenendaal, 2022). The authors suggest that TMMI complements CMMI and focuses on improving the maturity of the testing process, which they argue is not adequately addressed by CMMI.

• Software Process Improvement and Capability Determination (SPICE) (Dorling, 1993): ISO/IEC has proposed this framework, which draws on ISO/IEC 12207 (ISO, 2008). SPICE enhances processes within software organizations and assesses their capacity to deliver high-quality software products.

Software product maturity models

Product maturity models aim to define and measure the maturity of software products. Below are some of the prominent ones:

• Software Product Quality Maturity Model (SPQMM): The Software Product Quality Maturity Model was proposed by Al-Qutaish & Abran (2011) to assess software product quality. Initially, it calculates quality levels based on characteristics, sub-characteristics, and measurements derived from ISO 9126 (ISO/IEC, 1991). These values are then combined into a single value, converted into a six sigma value, and the software’s integrity level is determined using ISO 15026. Finally, the overall maturity level is identified. Assessors are required to use ISO 9126 quality attributes and metrics, making SPQMM dependent on this specific standard.

• SCOPE: Developed by the EuroScope consortium (Jakobsen, O’Duffy & Punter, 1999), the SCOPE product maturity model (SMM) evaluates software product maturity across five levels. Levels 2, 3, and 4 incorporate standards such as ISO 12119, ISO/IEC 9126, and ISO 14598. SMM evaluates quality by matching the specifications with the specified requirements. Additionally, SMM requires documentation of the process to ensure that the final product meets specifications. However, SMM does not focus on the final product’s quality (i.e., the code).

• Open-Source Maturity Model (OSMM): This model was developed by Golden (2005) to assist organizations in effectively implementing open-source software through a three-phase process. The process involves assessing maturity elements by defining requirements, locating resources, evaluating element maturity, and assigning scores. The second phase involves applying weighting factors, and the third phase involves calculating the overall product maturity score. Designed to be lightweight, OSMM can evaluate an open-source product’s maturity within 2 weeks.

• Technical capability model (TCMMI): TCMMI (Abdellatif et al., 2019) evaluates the maturity of a software product as an indicator of its quality. The framework is divided into a reference model and an assessment method. The reference model offers a structure for collecting product quality indicators as evidence of the product’s capability and maturity. The assessment method follows standard procedures to evaluate the product’s maturity by measuring its conformance to the quality attributes defined and agreed upon by the product’s stakeholders.

Design principles for developing maturity models

Examining the various widely used maturity models reveals that the process of model design and the specific steps followed in the design are not explicitly detailed (Becker, Knackstedt & Pöppelbuß, 2009). This raises an issue as the research steps cannot be followed in terms of repeatability, verifiability, and completeness (Frick, Küttner & Schubert, 2013). Searching the literature, we will explore significant contributions in this area, looking at previous attempts to propose methodologies and specific guidelines for creating maturity models. Various studies that revolved around creating new maturity models have attempted to develop guidelines to standardize the development of maturity models. These efforts involved standardizing vocabulary use and adhering to certain validated procedures. Bruin et al. (2005) developed one of the most important works in this field by proposing a universally applicable framework for creating such models. The authors established six essential phases for model development: scope, design, populate, test, deploy, and maintain. Their work clarifies that following a systematic and phased process is crucial as it ensures the developed model is robust and can be applied in real-world environments. The authors state that creating maturity models is an ongoing process requiring maintenance, regular updates, and adjustments to align with the rapidly changing practices in software development. Additionally, the authors gathered different perspectives and insights from domain experts in an attempt to enrich their framework with a broad range of knowledge and experiences. Becker, Knackstedt & Pöppelbuß (2009) developed a procedural model for creating IT management maturity models, following design science guidelines defined by Hevner et al. (2004). The authors proposed the following steps: comparison with existing models, iterative procedure, evaluation, design as a search process, design as an artifact, research contributions, targeted presentation of results, and scientific documentation. Otto, Bley & Harst (2020) proposed a detailed process model to guide users and developers in creating prescriptive and well-validated maturity models following eight steps. Additionally, Solli-Sæther & Gottschalk (2010) proposed a detailed process for developing stage models with methodological considerations. Their work presented a framework for developing stage models by theorizing essential topics related to growth stages. Table 1 illustrates the different design principles for developing maturity models.

Existing assessment frameworks and research gap

Only one assessment method that is related to maturity models was found during our review of the existing maturity models. A framework for evaluating digital health maturity models was presented by Woods et al. (2022) and offers a way to compare the various models that are currently available. The authors acknowledge the importance of evaluating digital health maturity models since they can eventually help achieve better healthcare outcomes. The framework has a total of four distinct categories: actionability, completeness, feasibility, and healthcare context assessments. The healthcare context assessment category assesses the suitability, practicality, and relevance of the model to health settings. The feasibility assessment category assesses the vendor’s commitment to improvements, access to data, the requirements to implement the model, and the resources required. The completeness assessment category assesses how the model expresses maturity across seven dimensions and 27 indicators. The actionability assessment category assesses whether the model’s recommendations are actionable, feasible, and consistent with the organization’s goals. Our goal of developing a framework to evaluate software maturity models can be achieved by following the structured approach detailed in the work of Woods et al. (2022). Their focus on context, feasibility, completeness, and actionability provides us with an evaluation approach that can be adapted for our framework (Woods et al., 2022).

Although extensive research exists around maturity model creation and the widely available established guidelines, a notable gap can be observed by examining the current literature. Missing from this body of work is a unified evaluation approach. Current literature fails to present an integrative assessment methodology that combines multiple dimensions of software maturity model development into a single assessment framework. To the best of our knowledge, no framework in the literature has focused on evaluating the quality of software maturity models. The absence of a comprehensive assessment framework clearly supports the opportunity for developing an integrative assessment and measurement framework that provides a holistic view of the software maturity model’s overall quality. Assessing software maturity models is critical as it eventually leads organizations to utilize validated models to improve and evolve their software development practices. To bridge this identified gap, we aim to develop an assessment framework for software maturity models that can be both adaptable and comprehensive. Our assessment framework will incorporate different criteria and provide a solid approach for assessing and improving software maturity models. Filling this gap will contribute to advancing the development of maturity models in the field. This framework will help navigate the complexity of evaluating and validating software maturity models, improving the software and process quality.

Research methodology

Our research methodology consisted of five phases, as shown in Fig. 1:

• (1)

  Literature review: Initially, we conducted a review of existing literature to understand the characteristics of software maturity models and what indicates a successful model. To do this, we reviewed several articles and industry reports to acquire insights on the strengths and weaknesses of current software maturity models. For instance, the work conducted by Picard, Renault & Barafort (2015) and Otto, Bley & Harst (2020) revealed some important factors that can be used in evaluating software maturity models.

• (2)

  Framework design and development: Based on the literature review, we created an assessment framework that involved the elicited evaluation criteria. The framework consists of the key aspects that we found important in developing high-quality maturity models. For instance, we considered aspects such as scalability, relevance, empirical evidence, and practical usability.

• (3)

  ISO/IEC TR 15504-3 integration: We reviewed the ISO/IEC TR 15504-3 assessment process guidelines to incorporate relevant features into our measurement framework to aid the assessment process.

• (4)

  Validation: After developing our criteria, we conducted a pilot study to evaluate several available software maturity models to assess how effectively our criteria capture maturity model quality. This pilot was carried out by two domain experts who reviewed the criteria in depth and suggested several refinements, including merging overlapping items and removing less relevant ones. Based on their feedback, we adjusted the criteria to reflect the essential aspects of quality more accurately. Once all researchers reached a consensus on the completeness and clarity of the revised criteria, we gathered structured feedback from academics and practitioners. The feedback concentrated on the participants’ overall satisfaction with the framework, its overall structure, its assessment capabilities, and its ease of use. We conducted four detailed case studies; participants were selected through purposive sampling, inviting academics who had developed maturity models in different domains. Each participant assessed their own developed model using our framework under researcher guidance to ensure objectivity. These case studies provided comprehensive empirical validation of our framework’s effectiveness across diverse contexts, helped us identify areas for improvement, and provided us with insightful information about our developed framework.

• (5)

  Documentation and guidelines: We present our developed assessment framework and provide instructions and supporting materials to assist users in implementing it.

The developed measurement framework

We utilized a focused literature search strategy based on the most reliable, established, and methodologically sound works in maturity model development and evaluation. Instead of reviewing all existing maturity model publications comprehensively, we aimed to strategically select foundational and influential sources that represent state-of-the-art maturity models’ assessment and development principles. The literature selection criteria were based on identifying the works that significantly contributed to the maturity model theory, methodology, and assessment practices. Sources were further assessed to select those that demonstrated impact, methodological rigor, and comprehensive coverage of assessment principles. We carefully considered frameworks and standards such as ISO/IEC (SPICE) and Capability Maturity Model Integration (CMMI) version 2.0 in this process. These well-known models were used to understand the key aspects of maturity and the stages of maturity levels. The identified evaluation criteria capture aspects of the model’s development and design. These criteria address design elements, such as clarity, standardization, documentation, and development aspects, including accuracy, effectiveness, validation, practical implementation, and ongoing support (dos Santos-Neto & Costa, 2019; Maier, Moultrie & Clarkson, 2012). Furthermore, insights gained from academic articles such as the work done by Bruin et al. (2005) and Wendler (2012) were instrumental in extracting several criteria for developing effective maturity models. Their emphasis on developing maturity models based on sequential phases of development (e.g., scope, design, evaluation) influenced the development of criteria such as criterion 1.2.

We aimed to capture comprehensive criteria across scalability, empirical evidence, and usability in practice. These themes consistently emerged in the literature as essential to high-quality maturity models. We drew upon the work of Frick, Küttner & Schubert (2013) for the need for tool support and Bruin et al. (2005) for practical implementation in real-world scenarios. Our criteria also reflected this since it indicates that support and real-world application are important in developing a robust maturity model.

The framework development was guided by a comprehensive literature review complemented by insights derived from the authors’ prior work in developing and implementing multiple software maturity models (Niazi et al., 2020; Alam et al., 2024). Combining literature review and domain-specific expertise enabled a comprehensive approach to criteria formulation. Figure 2 illustrates our framework development methodology, showing the clear progression from three primary input sources (literature review, existing maturity models, and authors’ prior knowledge) through systematic criteria identification and categorization, ultimately resulting in our comprehensive assessment framework. The assessment criteria were grouped into four categories: model basic information, model structure, model assessment, and model support. Each of these categories is important for developing and applying software maturity models. The categories were derived through thematic analysis of our literature review findings. This categorization emerged from identifying recurring themes and critical success factors across the examined sources. This four-category structure captures several aspects of maturity model development, from conceptual foundation through practical application, ensuring comprehensive quality assessment across all critical dimensions. Furthermore, an automated assessment tool is available at https://zenodo.org/records/15834005, allowing users to evaluate models more efficiently. Each criterion is assessed using a 5-point scale:

0–No

1–Somewhat/Maybe

2–Yes

U–Unknown

NA–Not Applicable

Scores are summed within each category and converted to percentages. We applied equal weighting to all evaluation criteria to ensure fairness, simplicity, and transparency. Since there is no established evidence or consensus on which criteria are more important, treating each equally avoids introducing subjective bias. This approach also makes the evaluation process easier to understand and replicate.

Table 2 identifies the specific software development contexts in which the maturity model can be applied. Each of the four categories is detailed below:

• The model’s basic information: This category covers the basic aspects that define the purpose of a software maturity model. It includes the theories or frameworks on which the model is built and the application domain. Additionally, this category emphasizes how the model differs from others and offers unique value. Table 3 contains the criteria for this category.

• Model structure: Table 4 outlines the key criteria for the model’s structure, focusing on its architecture and design. This includes the various maturity levels and elements illustrating different maturity aspects. An important aspect that is addressed in this category is the necessity for clear and straightforward definitions, as these help organizations accurately assess their current maturity level. Additionally, it addresses criteria that emphasize the simplicity and user friendliness of the software maturity model being evaluated.

• Model assessment: The assessment category’s criteria focus on the methods utilized by organizations to determine their maturity levels. This involves evaluating the methods used for assessments, ensuring the reliability of the tools, and validating of the assessment instruments. These criteria help guarantee that evaluations are objective and evidence-based, enabling organizations to measure their progress accurately. Also, the availability of various assessment methods is considered, for example, self-assessments or third-party assessments, which may be warranted for a variety of needs and situations. The criteria for this category are shown in Table 5.

• Model support: This category focuses on the availability of support mechanisms essential for the effective implementation of the software maturity model. It includes criteria such as training, documentation, and guidance. These resources help organizations apply the model to their specific needs and help them understand the results clearly. Furthermore, it highlights the importance of keeping the model relevant and regularly updated to maintain its applicability over time. Table 6 includes the criteria for this category.

Results summary

To ensure we can appropriately and objectively evaluate different maturity models, we should note the difference between the total and the grand total. the total represents the raw sum of valid scores across all applicable criteria, excluding items marked as not applicable (NA) or unknown (U). In contrast, the grand total is a normalized score calculated using the formula:

$$\left({\displaystyle \frac{\mathbf{T}\mathbf{O}\mathbf{T}\mathbf{A}\mathbf{L}}{\mathit{T}\mathit{o}\mathit{t}\mathit{a}\mathit{l}\mathit{n}\mathit{u}\mathit{m}\mathit{b}\mathit{e}\mathit{r}\mathit{o}\mathit{f}\mathit{a}\mathit{p}\mathit{p}\mathit{l}\mathit{i}\mathit{c}\mathit{a}\mathit{b}\mathit{l}\mathit{e}\mathit{c}\mathit{r}\mathit{i}\mathit{t}\mathit{e}\mathit{r}\mathit{i}\mathit{a}\times 2}}\right)\times 100$$

Therefore, the model is not penalized for criteria that involve something outside its intended scope or are insufficiently defined in the source material. By excluding NA and U scores from the denominator, the grand total reflects only the evaluated dimensions, thus preserving the integrity and comparability of the assessment outcomes. Table 7 illustrates the assessment results per category.

Score interpretation

0–25: Basic—The software maturity model is in the early stages, requiring significant development.

26–50: Emerging—The software maturity model shows foundational strengths but needs further refinement for broader applicability and impact.

51–75: Mature—The evaluated criteria demonstrate a functional approach, suggesting a reasonably developed software maturity model.

76–100: Advanced—The model meets standard maturity expectations, demonstrating broad applicability and significant impact.

Literature analysis revealed common practices in maturity assessment, where a four-level categorizations are widely used, similar to CMMI’s initial, managed, defined, and optimizing levels. Expert consultations with three experts confirmed that these ranges correspond with practical expectations for quality assessment of maturity models. The ranges were validated through pilot testing with two maturity models to ensure that the categorizations effectively differentiate between models of varying quality levels.

Defining the assessment input

Our approach begins by defining the assessment input, which includes the purpose, scope, and any constraints that may affect execution. We clearly specify the aspects of the models to be assessed, including their impact on enhancing software development processes and their relevance for various sizes and types of organizations. The assessment input will be collected during the initial phase, following these steps:

• •

  Step 1: The developer of the maturity model prepares the maturity model to be assessed.

• •

  Step 2: The developer of the maturity model appoints themselves or another qualified individual to be responsible for the assessment process of the maturity model.

• •

  Step 3: In consultation with domain experts, the developer selects qualified assessors who will participate in the assessment.

• •

  Step 4: The developer or the assessor defines the scope and objectives of the assessment:

  • Scope: To evaluate the effectiveness and comprehensiveness of the maturity model.

  • Objectives: Identify strengths, weaknesses, and areas for improvement in the maturity model.

• •

  Step 5: The developer or the assessor lists the various artifacts of the developed software maturity model to be evaluated.

• •

  Step 6: The developer ensures that the assessor has access to all required documents needed for the assessment.

• •

  Step 7: The developer or the assessor defines a clear schedule for the assessment process.

• •

  Step 8: The developer ensures the assessor is available during the assessment period.

Any changes to the assessment input should be approved and documented by the maturity model developer and assessors.

Structured assessment process

As described above, the framework has four main categories, each evaluated against a set of predefined criteria. These criteria reflect the software maturity models’ efficiency, relevance, and thoroughness. Once the assessment input from the previous phase has been finalized, the assessment process moves to the next phase with the following steps:

• Step 1: The developer confirms that all necessary documents are available and accessible for the assessment.

• Step 2: The developer or the assessor validates the consistency and sufficiency of the collected data before proceeding with the assessment.

• Step 3: The developer or the assessor assigns a score to each criterion in the defined categories.

• Step 4: The developer or the assessor calculates each category’s scores by summing each criterion’s score.

• Step 5: Convert the total score to a percentage (out of 100).

• Step 6: Based on the overall score, determine the quality of the maturity model.

Assessment output

After completing each phase, it’s important to document and report the findings obtained during that phase in detail as part of the output process. The developer of the maturity model is responsible for recording the assessment results in a comprehensive document. The resulting document provides detailed explanations for each decision made and the scores given, thus ensuring all supporting evidence is documented.

Expert feedback analysis and discussion

The integration of expert feedback served as a critical mechanism for ensuring our measurement framework’s relevance and effectiveness. We analyzed the survey results from eleven domain experts, with diverse experience levels in software process assessment, revealing several critical findings.

The evaluation demonstrated strong positive outcomes. A total of 81.8% of experts agreed, and 18.2% strongly agreed that the framework is logically structured and easily understandable, indicating excellent foundational design acceptance. Furthermore, 90.9% of experts affirmed that the criteria are self-explanatory, with only one expert expressing concerns. This demonstrates the framework’s clarity and accessibility for users with varying expertise levels. Expert validation of the framework’s evaluation capabilities showed positive results, with 72.7% agreeing and 27.3% strongly agreeing that it provides precise and reliable indicators for identifying areas of weakness and strength in software maturity models. This confirms the framework’s ability to provide meaningful and accurate evaluations. The agreement on the framework’s time efficiency is particularly noteworthy, with all eleven experts agreeing that the assessment process is time-efficient and resource-conservative. This addresses a critical practical concern for framework adoption in industry settings.

The framework’s comprehensive coverage received strong validation, with 81.8% agreeing and 18.2% strongly agreeing that it is thorough and considers all critical elements vital for assessing software maturity models. Additionally, 90.9% of experts confirmed the framework’s applicability across diverse software development contexts and organizational sizes, demonstrating its flexibility and broad utility.

The recommendations criterion produced similarly strong results, with 81.8% agreeing and 18.2% strongly agreeing to recommend the framework to colleagues and industry peers as a useful evaluation tool. We had balanced results when asking about the need for training before utilizing the framework, with 27.3% disagreeing, 27.3% being neutral, and 45.5% agreeing that some training is required. The results indicate that additional training may enhance users’ confidence, ultimately improving our assessment framework’s effectiveness. We also plan to supplement our framework with further guidance materials to increase the practicality of the assessment process. Additionally, to further facilitate users’ understanding of the criteria, several criteria have been reworded for clarity and comprehensibility. Table 9 illustrates evaluation scores and expert responses for each criterion.

Case study results

Threats to validity

Although our assessment framework was developed using a rigorous methodology, it is important to acknowledge several threats to its validity:

• Construct validity: The criteria used in the framework are derived mainly from the literature. However, there may be discrepancies between these criteria and other factors influencing the quality of software maturity models. Although we tried to ensure that our criteria comprehensively covered all relevant aspects of software maturity models, we may have overlooked research articles containing additional important criteria. Another construct validity concern is whether the framework truly measures maturity model quality. Even though we did not conduct statistical or qualitative testing for construct validity, the design of our approach supports the framework’s validity. The criteria were derived from a literature review, ensuring alignment with established quality indicators for maturity models. They were further validated by domain experts to confirm their relevance and coverage of key quality dimensions. Additionally, case studies were utilized to measure the framework’s practical utility in assessing different models and identifying meaningful areas for improvement, while aligning with ISO/IEC 15504-3 as a theoretical framework.

• Internal validity: The validation of our framework involved structured expert evaluations from industry practitioners and academics. Although their insights are valuable, potential bias may have influenced their responses due to personal experiences and perspectives. Moreover, in the case studies, assessors were required to evaluate their own developed maturity models. This might introduce the risk of bias, as they may unintentionally evaluate their maturity models positively. To mitigate this threat, we provided assessment guidelines from ISO/IEC 15504-3 standards to ensure reliable and comparable assessment results.

• Reliability: The consistency of the assessment using our framework could be affected by the subjective interpretation of the criteria by different assessors. Although some subjectivity is inherent in qualitative assessments, training and clear guidelines will be provided to mitigate this threat.