A Generalized approach to the operationalization of Software Quality Models

Research question

Can we develop an extensible, flexible, and independent framework for operationalizing software quality across various domains, while enabling modelers to independently select their tools?

We describe models created with our approach as Hierarchical Software Quality Assurance (HSQA) models. HSQA models are improvements to existing SQMs. HSQA models build on years of theory involved with hierarchical SQA. In addition, they incorporate multiple steps to reduce threats to validity and enhance the calibration and evaluation of software artifacts.

Objectives and Contributions

Recognizing the limitations of current methodologies and the imperative to advance existing practices, this work’s principal objective is to create a framework empowering software quality engineers to generate, validate, and operationalize quality models. The aim is to enhance the efficiency of efforts, facilitate experimentation, and foster collaborative opportunities. The contributions of this work encompass:

1. Platform for Investigative software Quality Understanding and Evaluation (PIQUE), an HSQA framework: Engineered to expedite the creation of experimental quality models that are easy to operationalize and emphasize improvements in aggregation techniques, Machine Learning (ML) weight selection, tool selection, and the overall ease of model generation.

2. Developer visualization tool: A tool tailored for developers, enhancing their ability to comprehend and engage with the intricacies of quality models.

3. Deployed HSQA models: Exemplars of operationalized HSQA models addressing diverse quality concerns, serving as practical demonstrations of the framework’s capabilities.

We address these objectives as follows. In ‘A Brief History of Software Quality Modelling’ we provide a longitudinal description of related work beginning with early techniques in quality assurance modeling followed by hierarchical approaches alongside complementary tools. In ‘A Generalized Approach’, we explore the PIQUE framework and describe its internal structure, metadata model, the software components that make the core framework, the tool selection process, and the generation, execution, and visualization of PIQUE models. ‘Internal Framework Mechanics’ focuses on the internal mechanisms that PIQUE uses to calculate a quality score, specifically, normalization, score mapping, aggregation, and ML weight selection. In ‘Use Cases’ we exemplify the PIQUE framework through two use cases. Finally, ‘Threats to Validity and Conclusions and Summary’ describe the threats to validity and conclusion respectively.

Modern hierarchical models

The 1990s mark a transition for software quality models. This decade sees a shift to vendor-neutral standards. In 1991, the ISO/IEC 9126 (ISO-IEC, 1991) surfaced as the first theoretical hierarchical model with a limited number of high-level characteristics. This model, however, lacked important quality aspects and evolved into the ISO/IEC 25010 SQuaRE model (ISO, 2011), which kept the hierarchical approach but expanded upon its predecessor by adding the security and compatibility characteristics.

Despite improvements, these models only focus on the top half of the hierarchy (i.e., external quality characteristics), leaving out necessary details to make them useful. Van Zeist & Hendriks (1996) and Samoladas et al. (2008) both presented extensions to the ISO/IEC 9126. Van Zeist & Hendriks (1996) collaborated with six established companies to produce the QUINT (Quality in Information Technology) framework. The first version resulted in a handbook (Punter, van Solingen & Trienekens, 1992) describing a software quality model and guidelines on how to use it. The second version of QUINT extended the framework by referencing the Extended ISO 9126 model. Samoladas et al. (2008) quality model focused on Open Source Systems (OSS) and extended some characteristics based on the ISO/IEC 9126. Although other quality models had tackled OSS systems, they required significant effort to set up. In this model, the authors required limited user interaction once the profile of the quality model assessment was established.

One of the earlier operational hierarchical models focused on object-oriented systems. The Quality Model for Object Oriented Design (QMOOD) software quality model was proposed by Bansiya & Davis (2002) and was based on Dromey (1995) which presented a quality model that used a hierarchy based on structural concepts of a system (class, function, object) instead of quality concepts such as maintainability, reliability, and usability. QMOOD is based on six quality characteristics that tie the quality of code to the quality of the design. The model is still used today and is effective in several open-source and e-commerce studies. However, operational constraints hinder the metrics’ effectiveness across software updates and releases. The advent of Agile, DevOps, and Continuous Integration/Continuous Development (CI/CD) characterize new development approaches marked by frequent software changes. The instability of these environments with integrated stakeholder feedback, significantly diminishes the efficacy of the QMOOD metric suite.

In 2003, Franch & Carvallo (2003) presented a variety of metrics in an attempt to use quality models to assess the quality of software package selection. Most methodologies proposed for choosing software packages compared user requirements with the packages’ capabilities rather than focusing on quality requirements. The authors attributed this to the lack of package descriptions and their corresponding quality requirements in specific domains. They proposed a six-step method to operationalize the ISO/IEC 9126 model.

The SIG maintainability model (Heitlager, Kuipers & Visser, 2007) was introduced as a distinct approach that only presents a maintainability model. This model assesses sub-characteristics related to maintainability by contrasting them with five source code properties: volume, complexity per unit, duplication, unit size, and unit testing. This approach coincides with efforts to focus on parsimonious models of characteristics that can be operationalized and that are well understood, or that resonate with software developers. The focus on maintainability coincides with the efforts of the technical debt community (e.g., Izurieta et al. (2018); Izurieta & Prouty (2019)) where there is now consensus to address maintainability issues earlier rather than later. “Technical Debt presents an actual or contingent liability whose impact is limited to internal system qualities, primarily maintainability and evolvability” (Avgeriou et al., 2016).

Introduced in 2009, the SQUALE model and framework (Mordal-Manet et al., 2009) is founded on the ISO/IEC 9126 quality model, emphasizing practical applicability in industrial settings. SQUALE enhances ISO/IEC 9126 by incorporating a more detailed intermediate layer through a concept known as practices. While it furnishes additional metrics for in-depth assessment, the framework includes valuable tool support and visualization features. However, the model lacks modularization and doesn’t establish a robust link between low-quality scores and specific low-level code issues.

The SQALE model by Letouzey & Coq (2010), introduced in 2010, adopts a distinct approach to quality modeling by associating quality with technical debt values determined by remediation costs. It organizes the model into hierarchical layers, ranking them using calculated values of testability, reliability, changeability, efficiency, maintainability, and reusability. Notably, SonarQube (https://www.sonarsource.com), a widely used framework and continuous integration quality monitoring service, employs SQALE as its foundational quality model for assessments.

The PIQUE framework presented in this study is preceded and influenced by multiple publications on quality models, namely, Quamoco by Deissenboeck et al. (2011), Wagner et al. (2012), Wagner et al. (2015), Izurieta, Griffith & Huvaere (2017), and the Quality Assessment Tool CHain (QATCH) by Siavvas, Chatzidimitriou & Symeonidis (2017). These models bridge the gap between the low-level layers representing directly measurable metrics, and the higher-level layers representing more abstract theoretical quality aspects. The authors of Quamoco explicitly stated, “Our aim was to develop and validate operationalized quality models for software together with a quality assessment method and tool support to provide the missing connections between generic descriptions of software quality characteristics and specific software analysis and measurement approaches.”

Kitchenham et al. (1997) recognized the need for a meta-model in their early SQUID approach to describe the complexity of potential instances of models. Quamoco also includes a meta-model that is generic and extendable, modularized, and integrated with benchmarking data. Developed in 2017, QATCH is a quality modeling project that, unlike the Quamoco project, doesn’t adhere to a meta-model description and lacks extensive validation efforts. However, QATCH stands out by concentrating on an automated approach to generate quality models that are attuned to and responsive to the subjectivity of stakeholders. In contrast to the Quamoco model, the number of layers in QATCH is restricted to three. The authors argue that the benefits of comprehensibility, explainability, and ease of extension their model brings outweigh the loss of granularity brought by large, complex models. The QATCH model also introduces the Analytical Hierarchy Process (AHP) (Saaty, 2008) that uses pairwise comparisons between characteristics to derive a numeric ordering of importance to high-level aspects of quality. AHP can be used as a rudimentary way to elicit the model’s higher-level weights. The obvious drawback is that as the number of characteristics increases, the pairwise comparisons square in size.

A current effort in quality modeling is the contribution made by Bass, Clements & Kazman (2021) from the Software Engineering Institute (SEI). They attempted to provide an alternative to the ISO/IEC 25010 standard by significantly reducing the high-level characteristics and sub-characteristics from approximately 40 to 10. Reservations remain regarding the content validity of this model.

Measurement and Tools

Numerous tools can inform HSQA models by providing critical data that help assess many aspects of software quality. These tools can be broadly categorized into several types based on their functions and the aspects of quality they address, however, PIQUE focuses on Static Analysis Tools (SATs) that capture metrics of source code or binaries. Other categories such as dynamic analysis, code coverage, and continuous integration tools are not specifically addressed. The comparison and contrast of these tools is beyond the scope of this paper; however, this section is meant to help communicate the expanse of available tools, in particular, the tools that the authors have been exposed to. These tools are available for conducting static analysis of source code, binaries, Software Bill of Materials (SBOM) components, cloud services, and microservices packages. Examples include tools for assessing coding styles (e.g., Checkstyle (https://checkstyle.sourceforge.io/)) and identifying source code warnings and bugs (e.g., Lint (https://man.freebsd.org/cgi/man.cgi?query =lint&manpath =FreeBSD+11.2-RELEASE), PC-Lint (https://pclintplus.com/), FindBugs (https://findbugs.sourceforge.net/), PMD (https://pmd.github.io/), Roslynator (https://github.com/dotnet/roslynator), Security Code Scan (https://security-code-scan.github.io/), SonarQube, and Insider (https://github.com/insidersec/insider). Further, specialized tools focus on binary analysis, with the aim to detect potential security threats, weaknesses, and vulnerabilities in a compiled program. Examples include CVE Binary Tool (https://github.com/intel/cve-bin-tool) (“cve-bin-tool” developed by Intel) and cwe-checker (https://github.com/fkie-cad/cwe_checker) (developed by the German research organization Fraunhofer FKIE). Tools to help assess SBOM analysis include Grype (https://github.com/anchore/grype) and Trivy (https://github.com/aquasecurity/trivy).

Model Structure

PIQUE (https://github.com/MSUSEL/msusel-pique) features the breakdown of quality-related characteristics into a hierarchical tree-based data structure. In Fig. 1 we depict a generalized view of the structure of PIQUE, and in Fig. 2 we depict the general approach to generating and operationalizing a quality assurance model based on the PIQUE technology. Because PIQUE is tool agnostic, we can leverage multiple tools and interface them with PIQUE, thus allowing practitioners a choice in using their preferred tools. PIQUE models are hierarchical and provide layered and holistic overviews of assets. The technology leverages existing static analysis methods as inputs and scores the quality and security of software artifacts. Scores are provided at multiple levels in the hierarchy—with audiences that range from developers to project managers to C-suite executives. Troubleshooting at lower levels allows developers to understand scoring techniques and calibrate tool inputs to the model, whereas the higher levels of the PIQUE model provide summaries (albeit, with a loss of accuracy) that target management and security operations.

PIQUE is a collection of library functions and runner entry points designed to support experimental software quality analysis from a programming language and component-agnostic perspective. To remain agnostic, the PIQUE framework is structured as a library that provides the abstractions, interfaces, and algorithms necessary for quality assessment but leaves the task of defining specific analysis operations to dependent projects. To improve adoption, this framework provides default classes for each quality assessment component, and thus allows the platform to be used “out of the box.” For experienced practitioners with quality assessment approaches, the platform allows each component to be overridden with experimental approaches.

PIQUE metamodel

The design of the PIQUE framework metamodel is shown in Fig. 3 and comes from reviewing the strengths and weaknesses of previous modeling approaches, and is heavily influenced by the QATCH and Quamoco hierarchical models. The former was designed with simplicity in mind by keeping the number of layers constrained to three (i.e., characteristics, properties, and measures layers), while the latter allowed for arbitrarily deep hierarchies consisting of characteristics, aspects, factors, and measures, where factors could also be subdivided into more sub-factors. In PIQUE we balanced these approaches and subdivided the hierarchy (from top to bottom) with aspects, factors, measures, diagnostics, and findings, where the terms are defined in Table 1.

These terms provide descriptions of concepts that serve as an initial starting point for hierarchical model design, yet are flexible enough to allow for interchangeability based on the scope of the system under analysis. In our observations, we’ve discovered that practical scenarios, such as specific stakeholder demands and situations necessitating systems analysis, often determine whether the node embodies a more abstract or concrete meaning.

Tool selection

Tool selection requires careful consideration of the underlying model. PIQUE features the breakdown of quality-related characteristics into a hierarchical tree. The characteristics and sub-characteristics comprising the PIQUE model are selections made by domain experts. For example, a practitioner wishing to evaluate the quality of source code in a given system will likely select an ISO model (e.g., ISO 25K) as the top-level characteristics, whereas a practitioner focusing on security may select Microsoft’s STRIDE model as the top-level characteristics. Further, quality is subjective, and the construction of a hierarchical model can be tailored to adjust to new notions and interpretations of quality that potentially combine underlying models, reference documents, or characteristics from diverse sources. For example, the quality of an SBOM artifact is of high importance to help assess the security of the supply chain and could combine notions of adherence to SBOM standards as well as the quality of the software represented and its dependencies.

The top-level characteristics help modelers with the selection of complementary tools. The tools selected should ideally meet the representation condition (Fenton & Pfleeger, 1996) of the measured theoretical characteristics (and sub-characteristics). Poor representation increases the construct validity of these models. As shown in Fig. 2, tools are connected to the model during PIQUE model instantiation. Importantly, we emphasize the ability of an organization to leverage existing tool investments in their organizations. By leveraging and combining tools we help with the generation of a holistic review of quality in context. Our experience selecting tools during the development of various PIQUE models is shown in Table 2. We also provide information describing the underlying model (i.e., characteristics) used to compare operational results against. Measurements obtained from tools are diverse (i.e., different ranges, scales, etc.) and the processes of aggregation and normalization are explained in detail in ‘Internal Framework Mechanics’.

Table 2:

Metadata associated with various models was used to build PIQUE models.

The first column shows the metadata. The underlying model describes the reference(s) used to select top-level characteristics in a PIQUE model. The High-Level Characteristics are the selected Quality Aspects in the PIQUE model, and the Tools represent the best available tool selected as a proxy to measure the characteristics.

	Software quality	Software security	Binaries	Programmable logic controllers	Cloud	SBOM security
Underlying Model	ISO 25010	STRIDE	STRIDE CWE-699	Top 20 Secure PLC Programming Practices (https://github.com/VirusTotal/yara) Valentine Taxonomy (Valentine, 2013) PLCOpen Guidelines on Software Quality Metrics (https://github.com/VirusTotal/yara)	ISO 25010 CWE-1000	ISO 25010 security section STRIDE CWE-699
High-Level Characteristics	Functionality, Performance, Compatibility, Usability, Reliability, Security, Maintainability, Portability	Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, and Escalation of privileges	Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, and Escalation of privileges	Reusability, Testability, Efficiency, Maintainability, Reliability, Rule-based Issues	Functionality, Performance, Compatibility, Usability, Reliability, Security, Maintainability, Portability	Confidentiality, Integrity, Non-repudiation, Authenticity, Accountability, Availability, Authorization
Tools	Roslynator	Security Code Scan Insider	cwe_checker CVE-bin-tool Yara (https://github.com/VirusTotal/yara) with rule repository ‘yara-rules’ (https://github.com/VirusTotal/yara)	CODESYS (https://github.com/VirusTotal/yara)	Grype Trivy	Grype Trivy

DOI: 10.7717/peerjcs.2357/table-2

Model Generation

Generating a PIQUE model involves several steps including model design, model implementation, benchmarking, and finally model assessment. The greater process is analogous to an iterative lifecycle from a Software Development Life Cycle (SDLC), with opportunities to cycle on previous steps based on stakeholder feedback. We also enhance the process with a visualization tool described in the last subsection. Visualization tools are lacking and are needed by modelers.

Visualization

Although many point tools do offer graphical user interfaces, our goal is to improve visualization in frameworks to capture holistic scores (i.e., a score based on the aggregation of potentially many tool scores). One visualization tool we found that also aims to address the needs of managers as well as model developers is exemplified in work by Martinez-Fernandez et al. (2019). The authors provide a strategic and a raw data dashboard to visualize information. We have also developed dashboard technology that aims to address the concerns of C-suite level managers by focusing on the high-level characteristics of a PIQUE model, as well as developer/modeler technology that allows a practitioner to focus on the details associated with the nuances of scores calculated at the lowest levels of the PIQUE models. The PIQUE dashboard is informed with JSON files that are output from the PIQUE framework.

At the higher layers of the model, the scores represent the outlook of a system that is of interest to program and project managers. This is shown in Fig. 4A. The lower you move in the layers of the model, the closer you get to the sub-characteristics of interest to developers and modelers. This is shown in Fig. 4B.

Normalization

Normalizing values associated with metrics and finding counts allows modelers to compare and combine scores from disparate sources. In PIQUE, the DefaultNormalizer software component is provided with the framework and is the default algorithm used; it divides the value of a node by the LOC in the target project. However, the diversity of the software artifacts makes normalization a challenge.

When normalizing projects across multiple languages, it is more likely that modelers will use a metric suite like Halstead’s complexity metrics described by Remoortere (1979). Modelers assessing binary file quality may first produce a high-level representation of the program that includes reconstructed control flow elements suitable for human consumption or machine-based inspection. Metrics from the control flow can then be used to normalize measures. Finally, if modelers are comparing SBOM files, they may focus on normalizing based on the number of third-party dependencies associated with each SBOM.

Score Mapping

PIQUE uses utility functions to map values or scores between layers to an appropriate range. The utility function provides normalization at the most atomic level in a PIQUE model. PIQUE has been designed so that every node in PIQUE has a unique utility function, one of which can be the identity function (https://en.wikipedia.org/wiki/Identity_function). In general, each utility function measures some combination of findings for a specific software project and outputs a value based on how it compares to other software projects in the benchmark repository. Our original utility-function algorithms are based on the QATCH operational framework from Siavvas, Chatzidimitriou & Symeonidis (2017). Currently, multiple utility functions are available in PIQUE, but the default is a variation of the Linear Interpolation utility function. This utility function ingests the minimum and maximum values found within the benchmark repository for each measure and outputs a score based on interpolating the value for the project under analysis.

In our early research, we extended the work of Siavvas, Chatzidimitriou & Symeonidis (2017) by offering a NaiveBenchmarker and a Gaussian-function-based Benchmarker, named BinaryBenchmarker. The NaiveBenchmarker calculates the lowest and highest Diagnostic value, wherein each Diagnostic value is generally a sum of Findings. This function generates values that are used as thresholds and later ingested in the utility function. In contrast, the BinaryBenchmarker calculates the mean plus or minus the standard deviation of each Measure to produce the threshold values. Subsequent versions of PIQUE introduce two additional utility functions: the GAMUtilityFunction, and the GaussianUtilityFunction, however, these functions are deprecated. PIQUE’s design is flexible, allowing for any Benchmarker to be used for calculating thresholds and providing the framework to define custom Benchmarking functions based on the nature of the data.

In our current framework, we use a scaling approach. This scaling function compares each artifact against the benchmarks using density-based scoring. This method is termed the ProbabilityDensityFunctionUtilityFunction, herein referred to as a PDFUtilityFunction. This function is depicted in Fig. 5 (Reinhold et al., 2024). The PDFUtilityFunction provides an objective method for mapping scores across a wide range of distributions. Scores are calculated as follows. We first create a benchmark repository by assembling a collection of software artifacts and assessing each with one or more static analysis tools (Steps 1–3 in Fig. 5). Currently, the results from all static analysis tools for all software artifacts are stored in system memory, however, we are working toward storing these results in an aggregation file. This aggregation file will be generally preserved as a flat file wherein each row represents a software artifact in the benchmark repository and each column corresponds to a finding measured by a static analysis tool. The values in the cells will contain the results from the static analysis tool(s). We determine the density of scores for each finding in the benchmark by calculating a probability density function based on the distribution of the values in each column.

When an end user wishes to evaluate a new software artifact, they run it through the same SATs that the collection was evaluated with. Each finding is then scored by comparing it against the score against the density of scores in the benchmark repository. In the example in Fig. 5, the end user’s software artifact of interest is found to have 40 instances of the finding named “CVE-Unknown-Other-Diagnostic”. The density-based scoring indicates that this artifact has fewer instances of this vulnerability than 66% of the artifacts in the collection.

The mathematical underpinnings of the PDFUtilityFunction are as follows. Each finding reported by SATs is treated independently when calculating the quality score. We use the “trapezoidal rule” by Yeh (2002) to approximate the definite integral representing the probability density function in Step 6 of Fig. 5 (Eq. (1)). The definite integral of a function represents the area under the curve between two specific points. (1)${\displaystyle A={\int }_a^{b}f\left(x\right)dx\simeq \frac{b-a}{2n}\left[f\left(a\right)+2f\left(x_1\right)+...+2f\left(x_{n-1}\right)+f\left(b\right)\right]}$

We obtain f(x) using kernel density estimator (KDE), a non-parametric method to estimate probability density function (PDF) from the counts of findings across all files in the collection (Steps 4–5 in Fig. 5). The PDF describes the probability of the count of the finding in the artifact of interest falling within the interval denoted by a and b; a is zero (as the minimum count of a finding in any artifact is zero) and b is the upper bound (established by the software artifact in the collection having the highest count of a finding). By fundamental property, the total area under the curve of a PDF must be one.

Suppose an end user has a software artifact that they are evaluating and the count of a particular finding in that artifact is represented by d (we call this d because the count of a finding is “diagnostic”; d = 40 in Fig. 5). We compute the area under the curve from zero to d; thus, A_d indicates the proportion of samples in the collection having fewer counts of a finding than d.

The scaling method employed in PDFUtilityFunction is appealing because it is unbiased, repeatable, and effective at normalizing results from diverse sources (here, multiple SATs) to enable aggregation. However, it has some limitations. The external validity and accuracy of inference derived from this solution is heavily dependent on the benchmark repository—i.e., the representativeness of the collection of artifacts against which a new artifact is being compared. In addition, the accuracy of the outputs of the PDFUtilityFunction is constrained by the accuracy of the SATs.

Aggregation

A third component that can significantly affect quality scores is the variability associated with external sources to the PIQUE quality model. This variability can come from different tools and different versions of the same tools. Further, results produced by tools may come in different scales and different ranges.

The methods described in ‘Normalization and Score Mapping’ alleviate many of these challenges, however, as scores propagate up the hierarchy in a PIQUE model, a modeler is given an opportunity to weight scores from child nodes.

As discussed in ‘Modern hierarchical models’, the QATCH framework introduced the Analytical Hierarchy Process (AHP) by Saaty (2008) that uses pairwise comparisons between characteristics at the higher levels of a hierarchical model. We have found that using the AHP algorithm to weigh higher levels of a hierarchical model is a tedious process that stakeholders struggle to accomplish. Consequently, we are adopting the process of replacing the manual weighting with semi-automated approaches. Our current research led us to applying ML to algorithmically assign weights. This approach is showing great promise for learning the profiles of weight combinations based on documentation originating from benchmarks of tool findings and relevant standards. In ‘Machine Learning’ we provide an in-depth discussion of this approach. The result is that aggregated scores more accurately represent the quality of the target system.

Machine learning

Experimental approach

We are actively experimenting and applying ML methods to programmatically learn the edge weights between the measure, product factor, and quality aspect nodes in any PIQUE model. The goal of this ML approach is to assign appropriate weights that enable PIQUE to produce meaningful scores for each node, thereby enhancing the TQI and improving decision-making. Other approaches that experiment with improvements to decision making include Bayesian techniques such as work by Manzano et al. (2018).

In our initial approach, the weights are assigned by manually analyzing the definition of each measure or product factor and creating a binary relationship with each quality aspect. The result of this approach is an indication of whether a given factor or measure impacts a quality aspect. Yet, the definition does not inform how much a measure affects a quality aspect relative to other measures and factors. Hence, the initial approach of assigning equal weights assumes that each measure/factor is independent, has an equal frequency of occurrence, and has an equal impact on software quality.

In the ML approach, we supplement the initial model with a benchmark dataset to establish an informed relationship between measure/factor and quality aspects. We look at the data distribution of the benchmark dataset and update the weights from the initial model to fit the distribution. This distribution provides an estimate of how the measures are implemented in practice. By fitting the initial model to the distribution, the final scores are relative to the benchmark repository, and better represent the project under analysis. Hence the scores have more meaning and can be compared to known software in the benchmark dataset. The ML approach can capture the dependencies between the measure/factors and identify trends in practical implementations. The modular design of PIQUE facilitates the generalization of this ML approach to all other PIQUE models, with ease.

As an example, consider the two software weakness measures CWE-117 and CWE-1024 in the PIQUE-bin model. Figure 6 shows the overview of the initial and ML approaches workflow. According to the definition, taken from the database of Mitre software weaknesses, these weaknesses affect the software security characteristic “Integrity”. The definition only tells us that higher counts of CWE-117 (https://cwe.mitre.org/data/definitions/117.html) and CWE-1024 (https://cwe.mitre.org/data/definitions/1024.html) should be correlated with the “Integrity” aspect. Hence, the initial approach is to assign equal weights for the two measures and linearly increase the score with the number of findings. This model manifests as a 45-degree line in the orthogonal space of CWE-117 and CWE-1024. When a new project is under analysis, the findings of the SAT on the project are projected onto the line to calculate the score by normalizing. In contrast, the ML approach adjusts the initial model line using regression based on the findings of CWE-117 and CWE-1024 in the benchmark data (https://github.com/MSUSEL/benchmarks). This allows the “Integrity” characteristic to change based on the distribution of findings in the benchmark data, resulting in a more meaningful quality model.

Although we only considered two measures in the example above, the number of measures affecting a quality aspect is determined by the PIQUE model definition. Thus, the quality aspect “line” will be in a higher dimensional space consisting of all the measures contributing to a quality aspect. Further, since quality aspects are independent, separate models will be learned for each quality aspect (e.g., “Integrity”, “Authenticity”, and “Availability”). Users can use different linear and non-linear regression ML models depending on the PIQUE model and the benchmark dataset. Due to the need for explainable ML models, classical low-order ML models are preferred over Deep Neural Networks. The ML models are intentionally trained to be over-fitted to the benchmark data, allowing the calculated score to be compared to an existing program in the benchmark. Consequently, the ML approach is sensitive to the benchmark data.

Analysis of C# source code security

While many analysis tools exist that can be used to identify security vulnerabilities, the use of a quality model like the PIQUE model is beneficial in aggregating outputs from multiple analysis tools thus providing better coverage of security vulnerabilities (as compared to the use of a single tool). The aggregation of findings from multiple tools provides a broader security quality context accessible at multiple layers of the model. A partial view of the model is depicted in Fig. 7.

The requirements for the PIQUE security model were gathered by working with stakeholders directly. Their approval was obtained on design choices such as the static analysis tools used, the Security Aspects layer of the model, and the linkage between the Security Aspect layer and Product Factor layers. The model was created using a top-down approach, where the root note is the Total Security Index (TSI). This node decomposes into security aspect nodes, which are nodes taken from both the ISO/IEC 25010 standard ISO (2011) (security sub-characteristics) and the Microsoft STRIDE (https://learn.microsoft.com/en-us/azure/security/develop/threat-modeling-tool-threats) model.

The security model links findings from tools Security Code Scan and Insider. Although we also used the Roslynator tool during our exploratory phase, we decided to remove it because Roslynator diagnostics focused on code quality, while Security Code Scan and Insider focus on security-related findings. The Roslynator tool was only used during normalization to count the physical lines of code in the specified project.

We analyzed the performance of the security model by comparing its performance against 72 benchmark projects (26 open-source and 46 closed-source), with sizes ranging from 11 to 286,151 lines of code. We found significant evidence of a positive correlation between the size (lines of code) of a C# open-source project under analysis and its TSI. No similar conclusions could be drawn for closed-source projects.

We further validated the security model by investigating the effectiveness of the selected static analysis tools used within the model. We measured the ability of the selected tools to detect security vulnerabilities from the CWE Top 25 Most Dangerous Software Weaknesses by comparing all 59 Diagnostic nodes in our model to the CWE Top 25 list. Our security model has direct coverage for 52% of the CWE Top 25 Most Dangerous Software Weaknesses (2021 list). When we add the related responses to this count, coverage increases to 76% of the Top 25 list. Thus, the static analysis tools selected by the stakeholder were effective when measuring and reporting vulnerabilities from the Top 25 list. An in-depth analysis of the vulnerabilities revealed that 7 diagnostics within the security model had the greatest impact on the TSI. A full description of this study can be found in Harrison (2022).

Analysis of software bill of materials security

An increasingly vulnerable software surface is the software supply chain (SSC). SBOMs enable the analysis of SSC security and is quickly becoming a fundamental cornerstone of SSC security. Assessing SSC security is achieved through scanning dependencies present in SBOMs and applying vulnerability mapping techniques to identify vulnerabilities present in SBOMs. As the SBOM space has evolved numerous SBOM analysis tools have been developed that assist in this process.

While these tools help identify vulnerabilities in SSCs, the use of a quality model like the PIQUE model is beneficial for multiple reasons. First as with other PIQUE models, aggregating outputs from multiple analysis tools provides broader coverage of security vulnerabilities through a holistic score. Additionally, SBOM presents a unique challenge when scanning for vulnerabilities. Due to the complex process of SBOM generation, we found that the process of generating an SBOM does impact the ability of SBOM analysis to find and report vulnerabilities (O’Donoghue, Reinhold & Izurieta, 2024), so using multiple SBOM analysis tools does help overcome this issue. Finally, each SBOM analysis tool is tailored to specific software ecosystems, potentially overlooking vulnerabilities in other ecosystems. Utilizing multiple SBOM analysis tools proves beneficial in tackling this challenge. Therefore, the aggregation of findings from multiple tools provides a broader security quality context that is accessible at multiple layers in the model.

The requirements for the PIQUE SBOM security model were gathered by working with stakeholders directly. Their approval was obtained on design choices such as the static analysis tools used, the Security Aspects layer of the model, and the linkage between the Security Aspect layer and Product Factor layers. The model was created using a top-down approach, where the root note is the Total Security Index (TSI). This node decomposes into Security Aspect nodes, which are nodes taken from both the ISO/IEC 25010 standard ISO (2011) (security sub-characteristics) and the Microsoft STRIDE model. These high-level Security Aspects are mapped to the category CWEs present in Mitre’s CWE-699 Software Development view (https://cwe.mitre.org/data/definitions/699.html). As the CWE-699 view encompasses the software development lifecycle, organizing vulnerabilities in this way enables software providers to understand what aspects of their SSCs are weak in relation to software development. The model is large, and we provide a partial view in Fig. 8 that depicts some exemplary nodes at different levels. The visualization tool described in ‘Visualization’ can be used to view the entire model.

The security model links vulnerability findings from tools Trivy and Grype. We validated the security model by investigating the effectiveness of the selected static analysis tools used within the model. We leveraged mining software repository techniques to collect a large corpus of SBOMs (1,151) from common open-source repositories and Docker images with packages ranging from one to 4,135. Both Trivy and Grype reported a large number of vulnerabilities across the SBOM corpus, however, vulnerability reports were rarely consistent between the tools as reported by O’Donoghue, Reinhold & Izurieta (2024).

We are planning additional validation of the tools by gathering SBOMs containing ‘ground truth’ sets of vulnerabilities and comparing the vulnerability reports from Trivy and Grype against these established ground truths. Initial feedback from our collaborators has revealed several interesting findings, with one of the most interesting being the variations in model scores linked to the selection of SBOM generation tools and formats. These findings have spurred further discussion into the value that SBOMs provide.

RQ

Can we develop an extensible, flexible, and independent framework for operationalizing software quality across various domains, while enabling modelers to independently select their tools?

Capturing the notion of quality is a wicked problem (Rittel & Webber, 1973) because of the inherent variability from multiple sources such as tools, tool versions, and measurement scales. Further, the subjectivity of a quality score is influenced by the mechanisms associated with the weighting of characteristics against each other, the lack of appropriate domain-specific benchmarks that can be used to calibrate scores, and the selection of tools that can act as proxies for theoretical characteristics (e.g., ISO models). However, we posit that progress needs to be made in this space and PIQUE delivers in its ability to provide extensibility, flexibility and independence of operationalizations. Specifically, we deliver the following contributions with the PIQUE framework:

1. We extend the capabilities of prior art by specifying a new meta-model that allows for the operationalization of agnostic quality models that can be tailored to use any underlying standards such as ISO, STRIDE, CWE families.

2. We have developed improvements in aggregation techniques, benchmarking and utility function mapping. We have also improved on manual weighing approaches between layers of a hierarchical model by employing ML techniques that are trained on exemplary data.

3. We allow developers to integrate new tools, and leverage existing tools that can connect to PIQUE models.

4. We provide visualization technology that addresses concerns of high-level management as well as model developers. Visualization aids for the latter, to aid with debugging at individual node levels, are lacking.

5. We exemplify the deployment of PIQUE through two deployed use cases.

6. We provide an extensible metamodel of the PIQUE framework. The metamodel is designed with extensibility in mind.

PIQUE contributes to the academic community because it offers a platform that can be used to experiment with new techniques to assess the quality of systems. There are many potential areas for improvement including but not limited to aggregation techniques, weighing of characteristics, and visualization. The practitioner community also benefits because many organizations are unlikely to devote resources to developing these frameworks. They instead use “out of the box” technology that can be deployed quickly. Practitioners can either deploy out of the box or choose to dedicate additional resources to calibrate their models.

Finally, it is important to emphasize that future work in SQA is paramount given the proliferation of software technology. Everything is connected, from personal technologies to critical infrastructure, which creates a landscape where the ripple effects of a poorly constructed software component can propagate through the supply chain with unknown and untraceable consequences. For this reason, we must address software quality as a first class citizen and evangelize software quality by design. With PIQUE, we aim to continue to elevate the importance of software quality by incorporating newer techniques and technologies as they emerge. Significant improvements in machine learning and accessibility to a larger open source corpora of examples are compelling reasons to continue to evolve the quality assurance landscape.