Empirical evaluation of an architectural technical debt index in the context of the Apache and ONAP ecosystems

Definitions

Definition 1. Architectural rule. Given a source code analysis tool T and the set of its supported analysis rules R^T, the architectural rules AR^T supported by T are defined as the subset of all rules $R_i^T\in R^T,i=\left\{1,\dots ,\left|R^T\right|\right\}$ such that:

• $R_i^T$ is relevant from an architectural perspective, i.e., strongly influences one choice of structures for the architecture (Keeling, 2017);

• $R_i^T$ is able to detect a technical debt item, i.e., “design or implementation constructs that are expedient in the short term but that set up a technical context that can make a future change more costly or impossible” (Avgeriou et al., 2016).

In ATDx, we consider every $A{R}_i^T$ as a function $A{R}_i^T:E\to \left\{0,1\right\}$, where E is the set of architectural elements according to a granularity level (see below). In case that an element e ∈ E violates rule $A{R}_i^T$, then $A{R}_i^T\left(e\right)$ returns 1, and 0 otherwise.

For example, a rule $A{R}_i^T$ checking that method overrides should not change contracts is (i) architectural since it predicates on the high-level structure of a Java-based software project (i.e., its inheritance tree), and (ii) related to technical debt as violating such rule might not lead to immediate repercussions, but could potentially cause unexpected behaviour and cumbersome refactoring as the software project evolves.

Definition 2. Architectural rule granularity level

(Granularity level). Given an architectural rule $A{R}_i^T$, its granularity level $G{r}_i^T$ is defined as the smallest unit of the software project being analysed which may violate $A{R}_i^T$, e.g., a class, a method, or a line of code. As an example, if we consider a rule which deals with cloned classes, its corresponding granularity level is “class”. Such mapping of architectural rules to different granularity levels enables us to evaluate and compare the occurrence of rules violations across different software projects at a refined level of precision, instead of trivially adopting a single metric for the size of software projects for all the rules in AR^T, e.g., source lines of code (SLOC). In addition, it enables us to assess the scope of the technical debt and as needed differentiate from defects.

Definition 3. ATD dimension. Given a set of architectural rules AR^T for an analysis tool T, the set of ATD dimensions ATDD^T contains subsets of architectural rules $A{R}_i^T\subseteq A{R}^T$ with similar focus. One architectural rule $A{R}_i^T$ can belong to one or more ATD dimensions ${\text{ATDD}}_j^T\subseteq {\text{ATDD}}^T$ and the mapping between $A{R}_i^T$ and ${\text{ATDD}}_j^T$ is established by generalizing the semantic focus of $A{R}_i^T$. For example, if an architectural rule $A{R}_i^T$ deals with the conversion of Java classes into Java interfaces, the $A{R}_i^T$ could fall under the general Interface ATD dimension. In ATDx, we use the 3-tuple $〈A{R}_i^T,{\text{ATDD}}_j^T,G{r}_i^T〉$ to represent the mapping of each architectural rule $A{R}_i^T$ to its granularity level $G{r}_i^T$ and ATD dimensions ${\text{ATDD}}_j^T$. It is important to note that, while an $A{R}_i^T$ can be associated to one and only one granularity level $G{r}_i^T$, an $A{R}_i^T$ can be mapped to multiple dimensions ${\text{ATDD}}_j^T$s, and vice versa.

ATDx formalization

ATDx aims to provide a birds-eye view of the ATD present in a software project by analyzing the set of architectural rules AR^T supported by an analysis tool T, and subsequently aggregating the analysis results into different ATD dimensions ATDD^T.

The goal of ATDx is portfolio analysis of projects in respect to their level of ATD. Intuitively, starting from a dataset of AR^T and Gr^T values belonging to a set of software projects S. ATDx performs a statistical analysis on the elements contained in the dataset, in order to classify the severity of the architectural rule violations of the software projects. The level of severity the system-under-analysis (SUA) exhibits for each rule $A{R}_i^T\in A{R}^T$, is then reported as a constituent part of the ATD dimension ${\text{ATDD}}_i^T\in {\text{ATDD}}^T$ mapped to $A{R}_i^T$. Notice that the ATDx analysis results of a specific SUA are relative to the other projects S in the same portfolio, and hence should not be interpreted as absolute values.

ATDx is based on the calculation of the number of architectural rule violations of a software project S (normalized over the size of S) in order to compare the occurrence of rule violations across projects of different sizes. Specifically, for each architectural rule $A{R}_i^T$, we first calculate $A{R}_i^T\left(\text{S}\right)$, defined as the set of architectural elements in S violating $A{R}_i^T$, i.e., (1)${\displaystyle A{R}_i^T\left(\text{S}\right)=\bigcup _{e\in G{r}_i^T\left(\text{S}\right)}a{r}_i^T\left(e\right)}$ where $G{r}_i^T\left(\text{S}\right)$ is the set of all elements e in S according to the granularity $G{r}_i^T$ (e.g., the set of all Java classes in a Java-based project), and $a{r}_i^T\left(e\right)$ is a function returning e if the element e violates the architectural rule $A{R}_i^T$, the empty set otherwise.

Subsequently, we calculate ${\text{NORM}}_i^T\left(\text{S}\right)$, defined as the normalized number of architectural rule violations $\left|A{R}_i^T\left(\text{S}\right)\right|$ over the total number of elements e according to granularity $G{r}_i^T$, i.e., (2)${\displaystyle {\text{NORM}}_i^T\left(\text{S}\right)=\frac{\left|A{R}_i^T\left(\text{S}\right)\right|}{\left|G{r}_i^T\left(\text{S}\right)\right|}}$ where $\left|G{r}_i^T\left(\text{S}\right)\right|$ is the size of S expressed according to granularity level $G{r}_i^T$ (e.g., the total number of Java classes in S), and $\left|A{R}_i^T\left(\text{S}\right)\right|$ is the total number of violations of rule $R_i^T$ (see Eq. (1)).

Once the ${\text{NORM}}_i^T\left(\text{S}\right)$ for rule $A{R}_i^T$ in S is calculated, we statistically establish its severity. In order to do so, we require the set ${\text{NORM}}_i^T$, which contains the values of ${\text{NORM}}_i^T\left(\text{S}\right)$ for each software project belonging to the portfolio, i.e., (3)${\displaystyle {\text{NORM}}_i^T=\left\{{\text{NORM}}_i^T\left({\text{S}}_1\right),\dots ,{\text{NORM}}_i^T\left({\text{S}}_n\right)\right\}}$ where n is the total number of projects belonging to the considered portfolio of software projects.

Given the calculation of ${\text{NORM}}_i^T$, we can establish the severity of the ${\text{NORM}}_i^T\left(\text{S}\right)$ measurement by comparing its value with the other ones contained in ${\text{NORM}}_i^T$. More specifically, given the set of values ${\text{NORM}}_i^T$ and the value of ${\text{NORM}}_i^T\left(\text{S}\right)$, we define the function severity as: (4)${\displaystyle severity:X^m\times \left[0,1\right]\to \left\{0,1,2,3,4,5\right\}}$ where X = [0, 1] and m is the total number of software projects belonging to the portfolio. The severity function returns a discrete value between 0 and 5, indicating the level of severity of ${\text{NORM}}_i^T\left(\text{S}\right)$ w.r.t. the other values in ${\text{NORM}}_i^T$. In order to do so, we adopt a clustering algorithm, namely CkMeans (Wang & Song, 2011), which guarantees optimal, efficient, and reproducible clustering of univariate data(i.e., in our case, ${\text{NORM}}_i^T$ values). Consequently, this step consists of identifying the severity cluster of ${\text{NORM}}_i^T\left(\text{S}\right)$ that contains similar ${\text{NORM}}_i^T$ values of other software projects within the portfolio. The usage of the CkMeans algorithm replaces the outlier-based calculation of ATD on which the original ATDx approach was based (Verdecchia et al., 2020); this decision allows us to gain finer-grained results (i.e., a discrete value between 0 and 5 instead of a boolean value).

An example of AR^T values distribution, and relative severity clustering, is provided in Figs. 1, 2. As we can observe in Fig. 1, the majority of the projects possess ${\text{NORM}}_i^T$ values between 0.0 and 0.1, which are grouped via CkMeans into three distinct clusters, as depicted in Fig. 2. Such clusters correspond to the lowest levels of severity, namely severity 0, 1, and 2 respectively. The other three clusters, possessing centers (i.e., weighted mean of cluster) of respectively 0.12, 0.21, and 0.5, correspond to the higher levels of severity, namely severity levels 3, 4, and 5. From the clustering depicted in Fig. 2 we see that, according to their distribution, most projects are classified as possessing low severity (severity < = 2), while only a smaller number of projects possesses a relatively high severity (severity > = 3).

In order to provide an overview of the ATD dimensions of a software project S, for each ATD dimension ${\text{ATDD}}_j^T\subseteq {\text{ATDD}}^T$ we define the value of ${\text{ATDD}}_j^T\left(\text{S}\right)$ as the average severity of the $A{R}_i^T$ mapped to it, i.e., (5)${\displaystyle {\text{ATDD}}_j^T\left(\text{S}\right)=\frac{\sum _{i=1}^{n}severity\left({\text{NORM}}_i^T,{\text{NORM}}_i^T\left(\text{S}\right)\right)}{j}}$

where j is the total number of rules in AR^T mapped to ATDD${}_j^T$.

Finally, we define an overall value ATDx^T(S), embodying the overall architectural technical debt of S calculated via our approach, as the average value of all the defined ATD dimensions ATDD^T, i.e., (6)${\displaystyle {\text{ATDx}}^T\left(\text{S}\right)=\frac{\sum _{j=1}^k{\text{ATDD}}_j^T}{k}}$

where k is the total number of ATD dimensions ATDD^T considered in the specific implementation of ATDx.

ATDx building steps

In this section, we report the steps for building ATDx. It is important to note that the whole process is generic, i.e., it is not bound to any specific analysis tool or technology and extensible. The described process can be performed by both (i) researchers investigating ATD phenomena and (ii) practitioners analyzing their own software portfolios. In fact, following the steps of the process allows its users to implement the instance of ATDx which best fits their specific technical, organizational, and tool-related context.

Figure 3 presents the building steps for implementing the ATDx approach. Given an analysis tool T (e.g., SonarQube), five steps are required to build an instance of ATDx, namely: (i) the identification of the set of architectural rules belonging to AR^T, (ii) the formulation of the 3-tuples in the form $〈A{R}_i^T,G{r}_i^T,ATD{D}_j^T〉$, (iii) the execution of T on a set of already available software projects to form the dataset of $A{R}_i^T\left(S\right)$ measurements, (iv) the execution of the ATDx analysis on the constructed dataset, and (v) the application of the ATDx approach on the specific SUA.

 
_______________________ 
 Algorithm 1: Computing the ATDDT dimensions and the ATDxT value 
  for a single SUA                                                                           ____ 
  Input: SUA, ART, NORMT, ATDDT 
  Output: ATDDT(SUA), ATDxT(SUA) 
  1  dimensions ← empty dictionary 
  2  atdx ← 0 
  3  for all dimensions j in ATDDT  do 
     4   dimensions[j] ← 0 
  5  end 
 6  for all rules ARTi  in ART  do 
     7   violations ← ARTi(SUA) 
8   normalizedViolations ← NORMT 
           i (SUA) 
9   dimensions[j] ← dimensions[j] + severity(NORMT 
          i , 
   normalizedViolations) 
10  end 
11  for all entries j in dimensions do 
     12   dimensions[j] ← dimensions[j] / getNumRules(j) 
13   atdx ← atdx + dimensions[j] 
14  end 
15  atdx ← atdx / |ATDDT| 
16  return dimensions, adtx    

Differences with respect to the original ATDx approach

As indicated in ‘Introduction’, the ATDx approach presented in this research constitutes an evolution of the original approach first introduced by Verdecchia et al. (2020). While all the building steps of the approach remain unvaried (see Fig. 3), one crucial aspect was redesigned, namely the technique with which the severity of ${\text{NORM}}_i^T$ violations are calculated. This design choice constitutes a change to the logical core of ATDx, overcoming one of the most prominent drawbacks of the original approach, namely the “emphasis on outlier values” (Verdecchia et al., 2020). The severity calculation in the original ATDx approach relied exclusively on the identification of ${\text{NORM}}_i^T$ “outlier” values, i.e., only the ${\text{NORM}}_i^T$ values greater than the upper inner fence² of ${\text{NORM}}_i^T$. By focusing exclusively on “outlier” values, the naive statistical technique used by the original ATDx posed two major disadvantages. On one hand, ATDx would provide only a boolean severity level of granularity, i.e., if ${\text{NORM}}_i^T$ values were “outliers” or not. On the other hand, the approach would focus exclusively on anomalously high ${\text{NORM}}_i^T$ values, disregarding via a lossy statistical analysis all ${\text{NORM}}_i^T$ values that were not identified as “outliers”. In order to overcome this drawback, in the version of ATDx presented in this research the “outlier” identification was substituted with a clustering algorithm, namely CkMeans (Wang & Song, 2011). By providing optimal, efficient, and reproducible clustering of univariate data, CkMeans allows ATDx to (i) provide finer-grained results by considering a range of severity levels, namely 0, 1, 2, 3, 4, 5, where 0 is the lowest severity and 5 the highest, instead of a simple boolean value, and (ii) considering the totality of the input ${\text{NORM}}_i^T$, instead of exclusively the “outlier” values. In addition, by considering the totality of ${\text{NORM}}_i^T$ rather than only outlier values, CkMeans mitigates the “potential empirically unreachable ATDD” that noticeably affected the analysis results of the original ATDx. Finally, the set of predefined number of clusters used by CkMeans allows for ad-hoc customization of the severity calculation, providing the capability to increase or decrease the number of clusters, i.e., the discrete levels of severity of ${\text{NORM}}_i^T$ values according to the specific needs of the users.

Goal and research questions

Intuitively, with our empirical evaluation we aim to understand if the ATDx analysis results faithfully represent the ATD present in real life software projects. In addition, we aim to assess if the ATDx analysis results are actionable, i.e., they motivate practitioners to refactor their ATD. More formally, we formulate the goal of our empirical evaluation by following the template proposed by Basili & Rombach (1988) as follows: Analyze ATDx analysis results For the purpose of evaluating their representativeness and their ability to stimulate action With respect to architectural technical debt From the viewpoint of software practitioners In the context of open-source software projects

It is important to note that ATDx can be applied to both open-source and proprietary software. In this study we focus on open-source software projects due to (i) the availability of rich data about their source code and development process and (ii) the ease of mining of such type of software projects with respect to proprietary ones (Hassan, 2008). Further considerations on the this empirical evaluation design decision are reported in ‘Threats to Validity’.

By taking into account our research goal, we can derive the following two research questions:

• To what extent are the ATDx results representative of the architectural technical debt present in a software project?

By answering this research question, we aim at assessing the representation condition (Fenton & Bieman, 2014) of ATDx, i.e., the extent to which the characteristics of the ATD present in a software intensive-system are preserved by the numerical relations calculated via the ATDx approach. In other words, this research question evaluates to which extent the ATDx analysis results are representative of the ATD in a software project, both by considering individually the results for each software-intensive system, than by comparing results across different systems.

• To what extent do the ATDx results stimulate action of developers to address their architectural technical debt?

By answering this research question, we aim to assess the extent to which the ATDx analysis results stimulate developers to address ATD, i.e., if the results motivate developers to actively manage the ATD detected via ATDx.

Empirical evaluation design

The empirical evaluation is designed according to our research questions and includes all the building steps of the ATDx approach described in ‘The ATDx Approach’. The empirical evaluation is composed of nine main phases:

• Phase 0 – Identification of the analysis tool T to be used in the empirical evaluation.

• Phase 1 – Identification and classification of the set of architecturally-relevant rules (i.e., AR^T); this step corresponds to the ATDx building Steps 1 and 2 in Fig. 3.

• Phase 2 – Identification of one or more software portfolios to be analyzed.

• Phase 3 – Establishment of the AR^T dataset(s) for the selected software portfolio(s); this step corresponds to Step 3 in Fig. 3.

• Phase 4 – Analysis of the dataset(s) via ATDx; this step corresponds to Steps 4 and 5 in Fig. 3.

• Phase 5 – Identification of a curated set of contributors of the selected software portfolio(s).

• Phase 6 – Generation of personalized ATDx reports.

• Phase 7 – Distribution of the ATDx reports.

• Phase 8 – Online survey on the ATDx analysis results.

In the remainder of this section we explain each phase of our empirical evaluation; we present the phases in general terms, so that independent researchers can fully reuse them in future replications of this study. Then, in ‘Empirical Evaluation Execution’ we provide the technical details about how we implemented and executed each phase in the context of (i) Java projects, (ii) the SonarQube analysis tool, and (iii) the Apache and ONAP software ecosystems.

In order to evaluate ATDx, we implement a concrete instance of ATDx by following the building steps presented in ‘The ATDx Approach’. As a first step, in Phase 0 we select a source code analysis tool T, implementing the R^T rules.

Phase 1 aims at identifying a set of AR^T rules on which the ATDx approach will be based. Specifically, the AR^T identification process is conducted by considering: (i) the soundness of the AR^T rules, demonstrated by industrial adoption and scientific evidence, (ii) the industrial relevance of the tool implementing the AR^T rules, and (iii) the feasibility of calculating AR^T values. In addition, during this phase, the identified AR^T rules are manually classified, in order to derive the 3-tuples $〈A{R}_i^T,{\text{ATDD}}_j^T,G{r}_i^T〉$ required by the ATDx approach.

As discussed in ‘The ATDx Approach’, the ATDx calculation relies on a portfolio of software projects. Hence, in order to gather ATDx analysis results, in phase 2 we identify the software portfolio(s) that will be used as experimental subject in our evaluation. The focus on software portfolios, rather than a collection of unrelated software projects, allows to focus on software projects that potentially share a similar context, and overlapping contributors, and hence are closer to the envisioned usage scenario of ATDx. Accordingly, in case of more than one portfolio is identified during this phase, the portfolios will be analyzed via ATDx independently. Driving factors for the identification of software portfolios is the availability of the software projects contained in the portfolio, and the possibility to calculate AR^T values for the portfolio, according to the AR^T rule set identified in the previous evaluation phase.

In phase 3 we automatically compute the values of AR^T and Gr^T for the software projects in the identified portfolio(s). This process is carried out either by gathering the source code of the projects, automatically extracting the Gr^T values, and executing the tool implementing the AR^T rules locally, or by directly mining pre-computed AR^T and Gr^T values made available remotely (e.g., if provided by contributors of the software portfolios, or a cloud service of the tool implementing the AR^T rules).

In phase 4, we execute the ATDx analysis, and calculate the ATDD^T(SUA) and ATDx^T(SUA) for each project of the identified portfolios.

In phase 5 we identify the relevant contributors of the selected software portfolio(s). Such contributors will then be contacted, in a following evaluation phase, in order to gather insights into the obtained ATDx analysis results. Specifically, we are interested in contributors who are familiar with multiple software projects of the portfolio(s), in order to enable them to compare ATDx results across different projects. Hence, we select out of the all contributors of the software portfolio(s), the ones who contributed to at least two projects of the portfolio in the past 12 months.

Once we obtained the ATDx analysis results for each project of the portfolio(s), and established a curated set of contributors to be contacted, in phase 6 we generate a personalized report for each contributor. Such report contains the ATDx summary results of each project of the contributor. The ATDx summary presents radar-charts (see Fig. 4 for an example) and further insights into the obtained results, e.g., architectural elements most affected by ATD (refer to ‘Phase 6: ATDx Report Generation’ for more information regarding the data provided in the ATDx reports).

In phase 7, the reports are shared with the contributors via a customized email, jointly with an invitation to participate to an online survey.

Finally, in phase 8 we collect insights on the ATDx analysis results via an online survey. The survey is designed in order to require a short amount of time to be filled (this helps in terms of both response rate and participants fatigue). Moreover, various factors that influence response rates of developers are considered while designing the survey, such as authority, brevity, social benefit, and timing (Smith et al., 2013). An overview of the questions composing the survey is reported in Table 1.

The survey is designed with a two-step approach. In the first step, a pilot version of the survey is drafted and shared with 5 industrial practitioners within our personal network. In the second step, the questionnaire is reviewed and finalized by taking into account the collected feedback.

Questions Q1–Q3 assess the experience of the participants in terms of their experience (Q1–Q2) and familiarity with the open source projects included in the personalized ATDx report (Q3). To ensure the quality of the data gathered via the survey, survey responses of contributors not familiar with the projects included in their personalized report will be discarded. The subsequent 6 questions are designed to collect the data relevant to answer our RQs. Specifically, Q4–Q7 aim at assessing the core RQ of our study (RQ1), namely if the ATDx results are representative of the ATD present in the software projects. More in detail, with Q4 and Q5 we aim at evaluating if the inter- and intra-relations of the ATD present in software projects are preserved by the numerical relations calculated via ATDx (Fenton & Bieman, 2014). With Q6 instead, we assess if the ATD dimensions (ATDD^T) identified during the building Step 2 of the ATDx approach (see ‘Step 2: Formulation of 3-tuples $〈{\text{AR}}_i^T〉$, ${\text{ATDD}}_j^T$, $〈{\text{Gr}}_i^T〉$’) are a faithful representation of the overall ATD present in the software projects³. As a follow up to the previous question, Q7 investigates if any prominent ATD dimensions are missing in the used instance of ATDx. We opted to include two separate questions, Q6 and Q7, both focusing of ATD dimensions, in order to provide participants with a swift mean to provide input via a closed-ended question, Q6, while enabling them provide further details via the open-ended question, Q7. In order to evaluate if the ATDx results stimulate the active management of ATD (RQ2), we use the final two survey questions (Q8 and Q9). Specifically, with Q8, we directly assess the extent to which contributors are inspired to take action based on the ATDx results. With Q9 we gather further insights on the potential use of the ATDx in development practices. In addition to the questions reported above, the closed-ended questions targeting RQs (Q4, Q5, Q6, Q8) are supported by a complementary question (“Comments?”), allowing participants to add further detail into their closed-ended answer. In addition, the survey closes with a final complementary question (“Do you have any final comments or suggestions?”), designed to gather any additional input the participants may like to provide. To ensure that participants would be able to freely express themselves, an informative note is included in the survey invitation text, to assure them that all collected data would be anonymous.

The complete survey, comprising the aforementioned questions and supporting text clarifying terms and questions, is made available for review and replication in the publicly available supporting material of the paper.

In the following section, we report the details of our empirical evaluation execution, which was conducted by rigorously adhering to the evaluation design presented in this section.

Phase 0: selection of the SonarQube tool

For this empirical evaluation we implement ATDx based on the (https://www.sonarqube.org/) static analysis tool. The rationale behind the adoption of SonarQube to implement the experimental ATDx instance is multifold: (i) SonarQube is widely used in industrial contexts (Janes, Lenarduzzi & Stan, 2017), allowing us to have an ATDx instance potentially with high industrial relevance (which could be used by practitioners independently of our empirical evaluation), (ii) SonarQube was previously utilized in academic literature to identify design issues, hence providing us a sound initial set for the identification of AR^SQ rules, (iii) SonarQube is open-source, hence the source code of each of its $A{R}_i^{SQ}$ rules can be inspected and associated to its granularity level $G{r}_i^{SQ}$ with relatively low effort, and(iv) the pre-computed SonarQube analysis results of several OSS projects are publicly available via the SonarCloud (https://sonarcloud.io/) platform, hence easing the $A{R}_i^{SQ}$SUA measurement retrieval process; those projects are actively maintained by several well-known organizations, such as the Apache Software Foundation (https://sonarcloud.io/organizations/apache/), Microsoft (https://sonarcloud.io/organizations/microsoft/), and the Wikimedia Foundation (https://sonarcloud.io/organizations/wmftest/).

Phase 1: AR^SQ identification and classification

The goal of this phase is to establish the set of architectural rules AR^T from SonarQube. As input to this phase, we use a set R^SQ of Java-based SonarQube rules that were identified as design rules in a previous research (Ernst et al., 2017). Those rules represent a sound starting set of rules of potential architectural relevance, according to our definition of architectural rule presented in ‘Definitions’.

To select the architectural rules among the ones presented by Ernst et al. (2017), we carry out a manual inspection of the definition of each single rule. Such inspection is based on the publicly-available official documentation of SonarQube (https://docs.sonarqube.org/latest/user-guide/rules/). The identification process is carried out by (i) analyzing the content of each rule description and (ii) evaluating it against the two criteria presented in ‘Definitions’. To mitigate potential threats to construct validity, two researchers independently carry out the identification. The identification process results in a 72.2% of agreement between the two researchers, with a substantial inter-rater agreement calculated via Choen’s Kappa (k = 0.62). Then, a third researcher with several years of experience in software engineering takes over by (i) resolving possible conflicts and (ii) reviewing the the final set of architectural rules AR^SQ. From the initial set of 72 SonarQube design rules presented by Ernst et al. (2017), we identify 45 architectural rules. As detailed in ‘Phase 3: AR^SQ Dataset Building’, we further refine the set of identified rules during Phase 3 by removing the architectural rules which are not included in the SonarQube quality profiles (https://docs.sonarqube.org/latest/instance-administration/quality-profiles/) of the selected software portfolios, leading us to a final set of 25 included rules. An overview of the final set of architectural rules used in this study is reported in Table 2.

Once we established the set of architectural rules AR^SQ, we classify them in order to derive their associated granularity level GR^SQ and ATD dimensions ATDD^SQ, i.e., we formulate the ATDx 3-tuples $〈A{R}_i^{SQ}〉$, $G{r}_i^{SQ},〈{\text{ATDD}}_j^{SQ}〉$. This classification process is carried out collaboratively by three researchers, who utilize open and axial coding to label AR^SQ to their respective GR^SQ and ATDD^SQ, and discuss potential divergences until a consensus is reached among all researchers. Columns 1, 3, and 4 in Table 2 give an overview of the final mapping between the rules $A{R}_i^{SQ}$, their granularity $G{r}_i^{SQ}$, and ATD dimensions ${\text{ATDD}}_j^{SQ}$.

Regarding the granularity levels Gr^SQ, we identify the four levels of granularity reported in column 3 of Table 2. The identified granularity levels are: Java non-comment lines of code (NCLOC), Java method, Java class, and Java file.

As for ATD dimensions ATDD^SQ, we elicited 6 core dimensions, namely Inheritance, Exception, Java Virtual Machine Smell (JVMS), Threading, Interface, and Complexity (see column 4 in Table 2). The Inheritance dimension (9 rules) clusters rules evaluating inheritance mechanisms between classes, such as overrides and inheritance of methods or fields. The Exception ATDD^SQ (6 rules) groups rules related to the Java throwable class “Exception” and its subclasses. JVMS (5 rules) contains rules which assess potential misuse of the Java Virtual Machine, e.g., the incorrect usage of the specific Java class “Serializable”. Rules associated with the Threading dimension (5 rules) deal with the potential issues arising from the implementation of multiple execution threads, which could potentially lead to concurrency problems. The Interface dimension (5 rules) encompassess rules assessing fallacies related to the usage of Java interfaces. Finally, the Complexity dimension (2 rules) encompasses rules derived from prominent complexity measures, e.g., McCabe’s cyclomatic complexity (McCabe, 1976). This phase lasts approximately 1.5 h, and includes both the AR^SQ identification and classification.

Phase 2: software portfolio identification

Subsequent to Phase 1, we can proceed with the identification of software portfolios, i.e., the experimental subjects of our empirical investigation. In order to collect AR^SQ values, we opt to use the SonarCloud platform, which enables us to efficiently and effectively gather the data required for the ATDx analysis (see also ‘Phase 0: Selection of the SonarQube Tool’). Hence, we want to identify portfolios that (i) are implemented in Java and (ii) make available pre-computed AR^SQ values via SonarCloud. In order to do so, we mine SonarCloud via its web-based API and (i) collect information about all public projects hosted on SonarCloud and (ii) identify the SonarCloud organizations⁴ having the highest number of Java-based software projects. This leads us to identify two different software ecosystems, namely (i) Apache (https://www.apache.org), covering general-purpose software components like the well-known Apache HTTP server, Apache Hadoop, and Apache Spark, and (ii) ONAP (https://www.onap.org), focusing on orchestration, management, and automation of network and edge computing services. In this study we choose to target two different ecosystems as experimental subjects⁵ in order to mitigate possible external threats to validity. Indeed, focusing on Apache and ONAP allows us to study ATDx results for software portfolios developed for different contexts, and having different development processes, cultures, and technical backgrounds.

Among all the Java projects in each ecosystem, we filter out those without a corresponding GitHub repository. This filtering steps allows us to (i) have full traceability towards the source code of the system (useful for further inspections) and (ii) retrieve the names and email addresses of projects’ contributors to be contacted for the survey (see ‘Phase 5: Identification of Relevant Contributors’). In order to avoid potential selection bias, we do not perform any other filtering step of the selected software projects, e.g., by removing those with relatively low number of Java classes or few violations of the rules in AR^SQ.

The final set of software projects is composed of 126 Apache projects and 111 ONAP projects, for a total of 3.6 millions of non-commenting lines of Java code across 237 software projects. Table 3 and Fig. 6 show the summary statistics of the selected ecosystems. From Table 3, we can observe that the smallest software project is included in the Apache ecosystem, and is implemented by only 90 Java NCLOC. From Fig. 6, we can see that this small project constitutes an outlier with respect to the other project of the ecosystem. In the ONAP ecosystem instead, the smallest software project is constituted by 753 NCLOC. The presence of small projects in the ecosystems is justified by the presence of “periferal” or “utility” software projects in the ecosystems, as further discussed in ‘Discussion’. The largest software project considered is also present in the Apache ecosystem, and includes 383K Java NCLOC. By considering the distributions reported in Fig. 1, we observe that both ecosystems present some software projects possessing a high outlier size, that will considerably contribute to the total number of AR^SQ violations of the two ecosystems (as further detailed in the following section). Regarding the mean size of projects of the two ecosystems, we note that the ONAP ecosystem possesses overall projects of bigger size (cf. columns Mdn of Table 3). The median instead is higher in the Apache ecosystem, due to the presence in the ecosystem of some projects of exceptionally high size, as previously discussed. The variability in size is overall higher in the Apache ecosystems (cf. columns σ of Table 3), which is also reflected in the coefficient values (CV) of the two ecosystems.

Phase 3: AR^SQ dataset building

After the identification of the software portfolios, we proceed with building the dataset of AR^SQ values for each portfolio. As a preliminary activity, we check the SonarQube quality profiles used by Apache and ONAP in order to ensure that all rules in AR^SQ are included (see Phase 1). This activity led to the exclusion of 20 rules from the AR^SQ set; the final set of AR^SQ rules is presented in Table 2. This quality assurance step is needed only in the context of our empirical evaluation and it is necessary to ensure that all rules in AR^SQ rules contribute to the calculation of the ATDD^SQ and ATDx^SQ values. After the consolidation of the AR^SQ set we retrieve the AR^SQ values for each project included in the identified portfolios. This process is executed via automated queries to the SonarCloud API. We obtained a total of 22.8 K AR^SQ rule violations across the 237 projects. For each rule violation, additional metadata is mined e.g., the Java class where the violation occurs, the affected lines of code, and the textual description of the issue. Such information is then used for further analysis during the report generation phase (see ‘Phase 6: ATDx Report Generation’), and is provided as complement to the ATDx report shared with the contributors.

An overview of the mined AR^SQ rule violations is reported in Table 4. As we can observe from the table, the total number of AR^SQ rule violations is much higher in the Apache ecosystem if compared to the ONAP one. We attribute this result to the presence, in the Apache ecosystem, to some large projects (cf. columns “Max.” of Table 3), which are also characterized by a high number of AR^SQ violations (see column “Max.” of Table 4). Both ecosystems include projects that do not present AR^SQ violations. The Apache and ONAP ecosystems have a median number of AR^SQ violations equal to 12 and 11, respectively. The standard deviation (σ) of AR^SQ violations instead results much higher for the Apache ecosystem instead. As before, this result can be attributed to the presence of few projects of considerable size in the Apache ecosystem (see Fig. 6). the same consideration can be made for the coefficient of variation (CV), as the projects belonging to the Apache ecosystem display a higher heterogeneity in sizes if compared to the ONAP projects.

Phase 4: ATDx analysis

By following our empirical evaluation design, the ATDx analysis is run independently for each portfolio, i.e., the analysis is based on the intra-portfolio comparison of AR^SQ values. As detailed in ‘Empirical Evaluation Design’, this ensures that the clustering on which ATDx relies is executed by considering exclusively software projects sharing a similar context, hence reflecting the envisioned usage scenario of ATDx.

Figures 7 and 8 gives an overview of the ATDx analysis results. While the ATDD values vary across the projects of the two ecosystems, both of them exhibit low ATDx values (with a median of 0.28 for Apache, and 0.25 for ONAP). In order to gain further insights into this finding, we consider the values of the various ATDD^SQ dimensions, and observe that none of the ATDD^SQ values reaches the maximum of the scale (i.e., 5—see ‘ATDx Formalization’). This has to be attributed to the potential empirically unreachable ATDD^T maximum values property of ATDx, which is further discussed in ‘Discussion’. While it would be possible to convert the scale adopted in order to improve the presentation and intuitiveness of the results (e.g., by converting local maxima to absolute ones), we refrain to do so, in order to support the transparency and understandability of the results.

All together the ATDD values contribute in equal parts to the final ATDx value (as the ATDx value is calculated as the average of all ATDD values). This means that the highest ATDD values increase the most the overall value of ATDx. In particular, in our evaluation Interface is the dimension that increases the most the ATDx value of both portfolios, followed by the Exception dimension (see Figs. 7, 8). The other ATDD^SQ dimensions increase less the ATDx value for both portfolios; nevertheless, some outliers are present, specially in the Complexity and Exception dimensions, meaning that some projects present an exceptional number of violations of rules belonging to such dimensions. Overall, the obtained results are in line with previous studies on other software metric indexes, e.g., the one by Malavolta et al., (2018).

Phase 5: identification of relevant contributors

In parallel to the AR^SQ dataset building and ATDx analysis phases, we identify the relevant contributors for our study, i.e., the contributors who will be invited to participate in our survey. As detailed in ‘Empirical Evaluation Design’, we are interested in contributors who contributed to at least two software projects of a portfolio in the past 12 months. In order to identify such contributors, we first mine the GitHub repositories to identify contributors who pushed commits to the master branches of the software projects in the past 12 months. Subsequently, we identity all overlapping contributors, i.e., contributors who resulted to be active in two or more projects included in a portfolio in the past 12 months. This process leads to the identification of 233 relevant contributors, 72 for the Apache ecosystem, and 161 for the ONAP ecosystem. No contributor is identified as a relevant contributor for both the Apache and ONAP ecosystems. For each identified relevant contributor, we store their contact information, along with the projects their contributed to, which will then be used to generate their personalized ATDx report in the subsequent phase of the empirical evaluation.

Phase 6: ATDx report generation

After the ATDx analysis, and the identification of the relevant contributors, we proceed with the generation of a personalized report for each relevant contributor. In total, 233 personalized reports have been generated. The generated reports follow the Markdown format and are hosted in a dedicated GitHub repository (https://github.com/S2-group/ATDx_reports). Using the Markdown format for the reports allows us to (i) show the ATDx results in a familiar environment for the projects’ contributors and (ii) directly link the personalized report in the email inviting the contributors to participate to the survey.

An example of the content contained in a personalized report is shown in Figs. 9–11. Each report is composed of three main parts, namely:

1. An introductory text providing the contributor with a concise explanation of the ATDx approach, the related background information (e.g., a brief definition of the ATDD^SQ dimensions), and a summary of the report content (see Fig. 9);

2. An overview of the ATDx analysis results for all projects of the contributor, provided in form of radar charts, to allow a swift comparison of ATDx analysis results across the projects (see Fig. 10);

3. A documentation of the ATDx analysis results for each project, including the top-10 classes containing the highest AR^SQ values, mapped to their ATDD^SQ dimensions (see Fig. 11).

Additionally, in order to provide the contributors with further context regarding the projects included in the report, each radar chart is complemented with additional information about the system under analysis, specifically: (i) a link to the original GitHub repository of the software project, (ii) a link to its SonarCloud dashboard, and (ii) a link to the complete raw data resulting from the ATDx analysis.

Phase 7: report distribution and survey invitation

After the generation of the reports, we share the results to the 233 relevant contributors identified in Phase 5. In addition to the distribution of the personalized reports, during this phase, we also invite contributors to participate to the online survey described in Table 1. Striving for a high response rate, we kept the invitation email as short and engaging as possible and customized its contents based on their receivers and the project they contributed to. Two different rounds of invitation, executed in two subsequent weeks, are used to stimulate the relevant contributors to participate in the survey.

Phase 8: online survey

In the last step of our empirical evaluation, we gather the data required to answer our research questions via the online survey. This survey is implemented by rigorously adhering to the structure presented in ‘Empirical Evaluation Design’. We stop the data collection 4 weeks after the last round of invites are sent out. This allows us to finalize the results to be considered, while providing relevant contributors an adequate amount of time to participate to the survey.

Participants demographics

In total, 47 out of 233 relevant contributors participated in our survey (i.e., ∼20% of the contributors participated in the survey). Participants have a median (average) of 12 (12.3) years of software development experience, with a minimum (maximum) of 2 (26). Most participants (97%) declared to have contributed to more than one OSS project, with the majority (38%) contributing to 6–10 OSS projects. All participants declared to be familiar with the analyzed software projects, with (i) 50% of them being very familiar with the projects (i.e., “occasional contributors”), (ii) 40% of them being extremely familiar with the projects (i.e., “regular contributors”), and 4% of them being moderately familiar (i.e., “have looked at its artifacts, read its code, and can contribute easily”). Based on the gathered demographic data, we are reasonably confident that all participant have a good level of development experience and enough familiarity with the projects to properly understand the ATDx results shown in their personalized reports.

(RQ1) On ATDx representativeness

In order to assess the representativeness of our approach, we examine the responses to questions Q4–Q7 of our survey (see Table 1). Figure 12 provides an overview of the responses given by the participants.

Question Q4 regards the extent to which ATDx analysis results reflect the actual ATD of a software project, by considering individually each project. The response distribution of this question reveals that most participants find the ATDx results representative (72%), with most participants agreeing with the statement formulated in Q4 (51%), or strongly agreeing with it (21%). Only a small portion of the participants does not find the ATDx results representative to various extents (8%), with only one participant strongly disagreeing with the statement. By considering the rather sporadic open-ended comments provided by participants for this question (6 data points), we note a characteristic lack of awareness of the ATD present in their projects, e.g., “Not sure how much technical debt we have” (4 data points), and a few acknowledgments of the results representativeness, e.g., “Results match my expectations”(2 data points). Overall, by considering the answers provided by the participants, we conclude that ATDx is representative when individual projects are considered.

Question Q5 focuses on comparing ATDx analysis results across all projects within each personalized report. Based on the results gathered with this question, we note that most participants find the ATDx results representative when compared across projects, albeit to a lower extent than when considering the results of individual projects. In particular, while the majority of participants agrees with the representativeness of ATDx results (62%), answers are also characterized by a higher disagreement (11%), and the highest number of neutral answers among all Likert-scale survey questions (28%). The few comments provided by participants for this question (5 data points), point to difficulties in comparing the ATD across different software projects. We conjecture that the lower agreement with respect to Q4 could be attributed to inherent challenges in comparing the ATD present in different software projects, that would also motivate the high number of neutral responses measured for this question.

Question Q6 regards the representativeness of ATDD^SQ dimensions used in the empirical evaluation. By looking at Fig. 12, we observe that overall Q6 yields the highest agreement rate (75%), and the lowest neutral (17%) and disagreement rates (8%). The participants (7 data points) suggest in the open-ended comments: (i) adding more dimensions, (ii) adding specific dimensions (e.g., “tests”, “cloned code”), or (iii) adding more details about the dimensions already included.

The final question related to RQ1 (Q7 in Table 1) regards potentially-missing ATDD^SQ dimensions of the specific ATDx instance used in the empirical evaluation. This question is optional and only three participants answered it. Nevertheless, the provided answers are informative and propose the following additional dimensions: “duplicated classes”, “testing”, and “cloned code”.

(RQ2) On ATDx actionability

With RQ2 we aim to assess the degree to which the ATDx analysis results stimulate developers to take action towards addressing their ATD. In the survey, this RQ is covered by two separate questions: a Likert scalequestion (Q8) and an open-ended question (Q9).

As shown in Fig. 12, participants generally agree with the statement that the results displayed in the radar charts inspire them to take action (60%). The remaining participants tend to either disagree with the statement (21%) or take a neutral stance (19%). By considering the additional comments provided by the participants to support their answer (6 data points), we observe the need for a finer-grained level of information in the ATDx report to address the identified ATD, as the provided documentation may not be sufficient to trigger concrete action on the analysis results. Examples of requested additional information include: “Give more information on problems” and “Add more technical debt aspects”.

Based on this finding, we conclude that the current information documented in the ATDx reports (namely ATDD^SQ values, top classes with AR^T violations, and JSON files containing the raw SonarQube analysis results) is perceived as actionable. Nevertheless, participants also suggested interesting points for improvement, e.g., by providing (i) the ability to zoom in and out of ATD hotspots at different levels of abstraction, (ii) ATD visualizations, (iii) hints about ATD resolution strategies. Question Q9 is about the scenarios in which the ATDx analysis can be used in practice. Even though only 10 participants answered this question⁶, participants mention some interesting usage scenarios about visualization (e.g., “as a UI in SonarQube”), refactoring (e.g., “find code to fix” code review), and communication (e.g., “talk about problems in issue tracker”). Also, participants highlight the lack of a user interface to visualize the analysis results in an interactive manner. Driven by the results collected for RQ2, we envision to improve the reporting of ATDx analysis results, in order to improve its actionability, and directly support a set of selected usage scenarios, e.g., by enabling the composition of the analysis with continuous integration pipelines, issue trackers, and enabling a finer-grained scrutiny of the result via a dedicated dashboard.

Conclusion validity

Conclusion validity regards if the experimental measurements are measuring the theoretical constructs they are intended to measure. As the results of our empirical evaluation are gathered via a survey, a possible thread to conclusion validity is the face validity of the survey (Weiner & Craighead, 2010), i.e., the extent to which the survey conveys the concept it purports to measure. In order to mitigate potential threats to face validity, supporting text explaining the goal of the survey, concepts related to ATD, and the ATDx approach purpose and functioning, were integrated in all material shared with the survey participants, namely the survey invitation message, the ATDx report, and the survey itself. Additionally, to ensure that the survey questions were sufficiently clear to participants, and no important aspect was missing in the questions, each closed-ended question was accompanied by an open-ended question(“Comments?”), were participants could add clarifications to their answers, doubts, and remarks.

To avoid potential threats related to the extent to which the survey answers are fitted to answer our RQs, we designed the survey questions by deriving them directly from the RQs and the goal of our empirical investigation. This led to the formulation of different survey questions, covering the various aspects the RQ purported to assess. To ensure full traceability of the mapping between survey questions and RQs, the complete process leading to the formulation of each survey question is documented in ‘Empirical Evaluation Design’, while the explicit mapping between survey question and RQ is also schematically reported in Table 1.

Potential threats related to low statistical power are mitigated by documenting separately the distributions of answers to each single question of the survey, allowing for independent scrutiny and interpretation of the gathered results. Additionally, the 20% response rate results to be aligned with other survey-based software engineering investigations (Malavolta et al., 2020). Hence, we are confident that this threat may have only marginally influenced our results, if at all.

Internal validity

Internal validity regards if the observed results are actually due to the treatment. Regarding the experimental subject utilized for the ATDx analysis, i.e., the Apache and ONAP ecosystems, we note that the projects considered may also contain non-Java source code, even if tagged as “Java” software projects on SonarCloud. To mitigate potential threat to internal validity, we consider for the ATDx analysis exclusively SonarQube rules pertaining to Java. To avoid instead potential bias when selecting the AR^SQ rules, three researchers were involved in Step 1 of the ATDx building process, their level of agreement was measured statistically, and disagreements were jointly discussed with the help of a third researcher. The same mitigation strategy was also applied for the definition of the Java-based 3-tuples in step 2.

Regarding the survey adopted, an internal threat to validity regards the potential influence that the invitation, ATDx report, and survey text may have had on survey answers. In order to mitigate this threat, all text was kept as neutral and formal as possible. Additionally, survey participants were informed that the survey was completely anonymous, so to allow them to freely express themselves.

A threat to validity which could have affected our results regards maturation (Wohlin et al., 2012). Specifically, the positioning of optional open-ended questions towards the end of the survey may have influenced their response rate. Nevertheless, we prioritized clarity and flow of the survey over this potential threat, and mitigated it by ensuring that answering all survey questions would require as little time as possible.

An incorrect understanding by the participants of the notion of ATD, and its difference from TD, might have negatively influenced the internal validity of our study. To mitigate this threat, we shared with the participants an intuitive definition of ATD at the beginning of both the invitation mail, the ATDx report, and the survey. Such definition was supported by the description of the different architectural technical debt dimension (ATDD) we used, in order to provide the participants with further context on the multiple facets of ATD considered in the study.

Finally, potential selection biases were mitigated by defining a priory a rigorous selection process to identify the portfolios and survey participants used in our empirical evaluation (see ‘Empirical Evaluation Design’). Additionally, to ensure the soundness of the selected participants, a set of demographic questions, including the familiarity with the shared software projects, was included in the survey.

Construct validity

Construct validity regards if our empirical evaluation is appropriate to answer the RQs. A prominent threat to construct validity, presented in ‘Discussion’, regards the evaluation of a specific instance of ATDx in order to evaluate the approach. As this step is required, we could not completely avert this threat, which has to be considered while interpreting the results. To mitigate its influence, we based our ATDx implementation on a widely popular static analyser, a starting set of design-related rules presented in the academic literature (Ernst et al., 2017), and two prominent OSS ecosystems, one of which was already adopted in various other TD studies (Digkas et al., 2017; Lenarduzzi, Saarimaki & Taibi, 2019; Tan, Lungu & Avgeriou, 2018; Li et al., 2020).

Another potential threat to construct validity is constituted by the adoption of OSS software ecosystems as portfolios for the ATDx analysis. To mitigate potential threats related to the selection of the portfolios, we ensured that they included a considerable number of software projects (237 in total), belonged to established OSS organizations (Apache and ONAP), and utilized SonarQube in their continuous integration pipeline. Additionally, for our survey we selected exclusively participants who contributed to at least two software projects of the portfolios, hence allowing them to compare ATDx analysis results across different projects.

A related threat regards the use of the Apache ecosystem as one of the two portfolios considered for our experimental evaluation. In fact, the Apache portfolio covers heterogeneous application domains and is maintained by many different contributors. Similarly, industrial portfolios can comprise hundreds or even thousands of projects (Jeffery & Leliveld, 2004), and hence display a similar variety in terms of project heterogeneity and distribution of developers across projects. Nevertheless considering the Apache ecosystem inhibits us to study one specific characteristic of ATDx, namely the possibility to tailor the results to a specific business domain. This threat was partially mitigated by considering also the ONAP ecosystem, which specifically focuses on network and edge computing services. In any case we note that the lower agreement rate to RQ5 (see Fig. 12) might be partially due to the heterogeneity of projects considered, leading to difficulties in comparing the ATD across the projects the participants contributed to.

In order to mitigate potential threats to mono-operation and mono-method bias described by Wohlin et al. (2012), in our survey design, we formulated it as a mix of open-ended and closed-ended questions, with different questions mapped to each RQ.

External validity

External validity regards whether and to what extent our observations can be generalized. A potential threat to external validity concerns the representativeness of the portfolios selected for our empirical evaluation. As reported in the previous section, we mitigated potential threats to external validity by ensuring that only relevant portfolios, and their contributors, were considered. In addition, the tool on which our experimental instance of ATDx is based, namely SonarQube, is one of the most frequently used static analysis tools for Java-based software projects (Janes, Lenarduzzi & Stan, 2017), making us reasonably confident about the relevance of its rules in real-world projects. Despite our best efforts to mitigate external validity threats, such could potentially influence our obtained results, especially if proprietary portfolios or other source code analysis tools are considered. Future research will naturally further strengthen the external validity of the results reported in this research, e.g., by experimenting with ATDx in industrial settings, and by considering additional source code analysis tools.