Secure software development: leveraging application call graphs to detect security vulnerabilities

Data breach prevention and privacy-enhancing technologies

Researchers have created various frameworks and encryption schemes to address vulnerability in data communication. Coutinho et al. (2018) introduced an advanced neural cryptographic model that trains artificial neural networks to establish a theoretically unbreakable one-time pad (OTP) encryption algorithm. This model operates autonomously without needing human input to enhance communication security. Vivar et al. (2020) comprehensively analyzed security issues in smart contracts and recent advancements in available public tools. Meanwhile, Bai et al. (2024) proposed a password-based access control system that maintains security while ensuring system availability. Xu et al. (2024) developed the Advanced Network-Hiding Access Control (AHAC) framework, which minimizes exposure in network settings, thereby improving secure data access. Jiang et al. (2024) introduced an innovative Reversible Data Hiding in Encrypted Domains (RDH-ED) scheme based on Variable Threshold Image Secret Sharing (VTSIS), which requires no preprocessing and supports high security, a high embedding rate, and complete reversibility.

Valera-Rodriguez, Manzanares-Lopez & Cano (2024) examined Fully Homomorphic Encryption (FHE) using the Microsoft SEAL library, tackling privacy and security challenges across digital ecosystems. Jin et al. (2024) designed a token-based encryption scheme that optimizes search and update times, supports multi-user modes, reduces memory usage, and addresses history storage challenges. Chen et al. (2024) proposed an ECC-based AKA protocol for secure message exchanges between devices and servers, ensuring user anonymity, forward security, and two-way authentication with minimal computational overhead.

Detection of password misuse vulnerabilities

Password misuse continues to be a major vulnerability due to possible mistakes by developers in following cryptographic security standards. Zheng et al. (2024) tackled this issue by introducing CRYPTOLINT, a lightweight vulnerability detection system that employs static program slicing techniques to identify cryptographic misuse. Shuai et al. (2014) created the Cryptographic Misuse Analyzer (CMA), which uses static analysis to identify password API calls in Android applications. This system constructs control flow and function call graphs, performs dynamic analysis, and logs cryptographic function invocations to detect possible password misuse vulnerabilities.

To focus on cryptographic misuse in iOS applications, Li et al. (2014) developed iCryptoTracer. This system integrates static and dynamic analysis to identify algorithmic issues and offer detailed cryptographic insights.

Vulnerability detection tools and techniques

Multiple tools have been established to address vulnerabilities more comprehensively, using automated testing methods to ensure strong security against diverse cryptographic misuse risks. The POET system (Rizzo & Duong, 2010) employs fuzzing technology to automate the detection of Padding Oracle attacks in web applications. At the same time, FIAT (Barenghi et al., 2012) uses fault injection techniques to improve security testing on password-based systems. Also, Aumasson & Romailler (2017), Peng et al. (2019), Feng et al. (2022), Kamalbayev et al. (2021) proposed differential fuzzing technology, which uses the CDF automated tool to assess cryptographic applications for security vulnerabilities.

In summary, our proposed detection method employs a layered and modular integrity model, combined with dynamic and static analysis techniques, to create a comprehensive view of cryptographic function flows within Windows applications. Using a CFDT, the system evaluates the entire sequence of cryptographic function calls to identify potential vulnerabilities impacting application integrity. This approach addresses several limitations found in previous methods, such as overlooking runtime vulnerabilities and having limited path coverage, thus offering a more precise and robust solution for detecting cryptographic misuse (Forain, de Oliveira Albuquerque & de Sousa Júnior, 2022; Bi et al., 2019; Li et al., 2024; Chen et al., 2024).

The “Scheme Architecture: Proposed approach” section provides an experimental analysis to validate this approach’s effectiveness. In this section, we apply our detection model to various Windows applications to evaluate its ability to discover cryptographic misuse vulnerabilities and compare its performance with existing methods. The following section outlines the experimental setup, test cases, and analysis of the results, illustrating our methodology’s practical impact and benefits.

Cryptographic application integrity model

The core of our approach for detecting misuse vulnerabilities is based on establishing a solid cryptographic application integrity model. This model offers a structured framework for assessing the security and integrity of cryptographic functions within software applications, ensuring these functions adhere to the expected flow to uphold cryptographic-level security. The model is structured hierarchically and layer by layer to accomplish this goal, organizing cryptographic functions according to their roles and interactions. Figure 1 provides an overview of the framework used for identifying misuse vulnerabilities, while Figs. 2 through 4 demonstrate the application of the model using specific cryptographic libraries like CryptoAPI and OpenSSL. These figures illustrate how each layer of the model contributes to the secure execution of cryptographic processes.

The model consists of three layers, each addressing various elements of cryptographic operations:

• 1)

  Library support layer: This foundational layer includes essential data structures and operations as building blocks for more advanced cryptographic tasks. Examples of this layer include items such as X509 certificates, PEM information formats, random number generators, and ASN1 encoding libraries. These components fulfill the basic needs of cryptographic libraries by providing key functions and data formats necessary for secure data management.

• 2)

  Cryptographic algorithm layer: This middle layer contains various cryptographic algorithms that different libraries implement. The specific types of algorithms and the number of functions available can differ based on the library used (e.g., CryptoAPI or OpenSSL). Typically, this layer includes symmetric and asymmetric encryption algorithms, hashing, and digital signatures. It is critical to guarantee that applications can access secure and well-tested cryptographic algorithms essential for data protection and integrity.

• 3)

  Protocol layer: This layer builds upon the foundational capabilities of the first two layers by implementing higher-level protocols such as SSL/TLS, which are critical for secure communication tasks like identity verification and key exchange. It connects low-level cryptographic operations and the application’s needs, providing a safe and coherent protocol structure for end-user applications.

Each layer is personalized to effectively operate cryptographic modules, which are self-contained groups of functions that collaborate to provide specific functionalities within a cryptographic library. Figure 2 depicts the overall organization of this integrity model, while Figs. 3 and 4 offer more detailed views of the architectures of the CryptoAPI and OpenSSL cryptographic libraries, respectively.

Within the cryptographic modules, each function call is illustrated by a node, and these nodes are categorized based on their invocation patterns. This classification assists in monitoring how often each function is called and whether their invocation sequences conform to expected standards. The nodes are categorized as follows:

• α Node: Denotes a function that should be called exactly once at a specific location within the module, guaranteeing that essential cryptographic operations are carried out without duplication.

• β Node: Denotes a function that may be invoked multiple times at a designated position, indicating flexibility regarding how often the function can be used within the cryptographic process.

• γ Node: Denotes interchangeable functions within the same category, where the order of invocation can differ without affecting the intended flow of cryptographic functions.

The flexibility these nodes offer allows our model to adapt to the usual variations in the usage of cryptographic functions, enhancing its applicability across diverse libraries and application demands. A visual overview of these node types and their invocation patterns can be found in Table 1, which details the various sequence types and their security implications.

In practice, a cryptographic module is represented as an ordered sequence of these nodes, reflecting the intended flow of essential cryptographic functions within a specific application. The absence of any required node in this sequence indicates a missing or incorrect function, which could potentially jeopardize the application’s security. For example, the sequence depicted in Eq. (1) demonstrates a typical pattern of cryptographic function calls:

(1) $$\mathit{\alpha }\left(\mathit{A}\right)\to \mathit{\alpha }\mathit{\gamma }\left(\mathit{B}\right)\to \mathit{\beta }\left(\mathit{C}\right)\to \mathit{\alpha }\mathit{\gamma }\left(\mathit{D}\right)\to \mathit{\alpha }\left(\mathit{E}\right).$$

In this sequence, every type of node is essential in ensuring that cryptographic operations are carried out securely and predictably. Any node left out or used incorrectly could result in security weaknesses, as shown in Table 1.

Extraction of combined dynamic and static cryptographic function invocation graph

Using this model, we can confirm whether the calls to cryptographic functions in an application follow the expected flow outlined in the model. We extract the CFCG to evaluate the sequence of cryptographic functions in an application. The CFCG offers a high-level view of function invocation during cryptographic activities, capturing key details about the interactions and order of function calls (see Fig. 5 for an example).

The integrity model’s hierarchical and modular structure provides a flexible yet powerful framework for identifying misuse vulnerabilities across various applications and cryptographic libraries. This model forms the groundwork for additional analysis and comparison, which will be explored in the subsequent sections.

Next, we will extract and analyze the flow of cryptographic function invocations within an application. We will compare this with the sequences set out in our integrity model to spot any deviations that may signal misuse vulnerabilities.

It is essential to fully understand how cryptographic functions are carried out in an application to detect misuse vulnerabilities. We employ a combined static and dynamic analysis approach to create a CFCG, which maps the complete flow of cryptographic function invocations. This hybrid method addresses the shortcomings of static- or dynamic-only techniques, such as missing runtime dependencies or inadequate coverage of potential function pathways.

Both static and dynamic analyses play distinct roles in developing a reliable and accurate CFCG:

• Static analysis: The static component examines the application’s codebase before it runs, tracing function calls and control flow to generate an initial structure for the CFCG. This preliminary graph captures possible execution paths, including direct and indirect function calls. Static analysis is valuable for revealing dependencies and function calls that may not be triggered during a single execution but are essential for understanding the overall cryptographic flow.

• Dynamic analysis: In contrast, dynamic analysis occurs during runtime, collecting real-time information about function executions, including indirect calls and dynamically loaded functions that may not be identified through static analysis alone. This process enables us to obtain usage data on cryptographic functions, documenting the sequence and frequency of function calls in practice.

Figure 5 displays a sample CFCG that illustrates an application’s flow of cryptographic functions. In this graph, nodes represent individual cryptographic functions, while edges indicate the calling relationships among these functions. By merging the data from static and dynamic analyses, we can enhance this initial CFCG to more accurately depict the actual function invocation paths within the application, assisting in identifying potential misuse vulnerabilities.

Steps for extracting the cryptographic function call graph

The CFCG extraction process consists of a defined series of steps designed to capture all relevant function calls and paths, as illustrated in Fig. 6. The steps are outlined as follows:

• 1)

  Preprocessing the application: The first step is to prepare the application for analysis. If the application contains packed or encrypted files, we use unpacking tools to ensure all code is accessible for static and dynamic inspection. This preprocessing phase guarantees no function calls are missed due to obfuscation methods or packing.

• 2)

  Static control flow analysis: In this step, we conduct a thorough static code analysis using control flow analysis tools such as IDA Pro SDK. This results in the creation of the preliminary CFCG, which maps out all potential function calls, including explicit calls and indirect jumps within the control flow. The generated graph offers a high-level view of the application’s potential execution paths. This phase also identifies implicit calls or conditional branches that might be overlooked by dynamic analysis due to its reliance on specific runtime conditions.

• 3)

  Dynamic binary instrumentation: We proceed to the dynamic analysis phase with the initial graph established. In this stage, we apply dynamic binary instrumentation by executing the application in a controlled environment and monitoring cryptographic function calls in real-time. This approach captures critical runtime information, including the sequence of function calls, execution frequency, and any dynamically loaded functions. The runtime traces enable us to refine the initial CFCG to accurately represent the execution flow, incorporating indirect jumps and implicit calls.

• 4)

  Graph refinement and integration: After collecting static and dynamic data, the CFCG is refined to generate a final, comprehensive graph that effectively represents the cryptographic function calls within the application. Based on insights gained from dynamic analysis, the CFCG is updated to add new nodes (representing cryptographic functions) and edges (representing calling relationships) that static analysis did not initially capture. This final step ensures the CFCG fully integrates anticipated and observed function flows, resulting in a highly accurate model of the application’s cryptographic behavior.

Construction of cryptographic function dominator tree

The CFCG is a thorough representation, but it can be pretty detailed due to the numerous cryptographic function calls and their interconnections. To make the analysis more accessible and improve the identification of misuse vulnerabilities, we create a CFDT derived from the CFCG. The CFDT emphasizes meaningful relationships and control flows, making identifying vulnerabilities within cryptographic functions simpler by organizing the tasks based on their logical execution sequence.

A dominance tree is a control flow data structure that illustrates how specific nodes (or function calls) “dominate” others according to their place in the execution hierarchy. This structure is critical for analyzing the flow of cryptographic functions, as it clarifies which functions must be executed before others, enabling a detailed comparison of the expected execution order with the actual execution order.

Dominance relationships and their importance

In a dominance tree, the arrangement of each node indicates its execution dependencies, which means that one function must finish before another can start. The CFDT structures these relationships to simplify, verifying whether cryptographic functions are executed securely and predictably. Figure 7 presents an example of a dominance tree. It illustrates how various cryptographic functions are organized based on their dominance relationships, thus creating a clear pathway from the application’s entry point to its exit.

The CFDT is based on several essential definitions that describe the relationships between function nodes:

• Dominance: A function node ${\mathit{v}}_{\mathit{p}}$ is said to “dominate” another node ${\mathit{v}}_{\mathit{q}}$ if every potential path from the program’s entry point to ${\mathit{v}}_{\mathit{q}}$ must pass through ${\mathit{v}}_{\mathit{p}}$. This relationship is essential because it allows us to monitor key functions that secure the cryptographic process and block unauthorized access.

• Strict dominance: If a node ${\mathit{v}}_{\mathit{p}}$ dominates ${\mathit{v}}_{\mathit{q}}$ and ${\mathit{v}}_{\mathit{p}}\ne {\mathit{v}}_{\mathit{q}}$, then ${\mathit{v}}_{\mathit{p}}$ is regarded as “strictly dominating.” ${\mathit{v}}_{\mathit{q}}$. Strict dominance means that ${\mathit{v}}_{\mathit{q}}$ relies on ${\mathit{v}}_{\mathit{p}}$ and cannot operate independently, introducing an additional layer of organization to the cryptographic call flow.

• Immediate dominance: Node ${\mathit{v}}_{\mathit{p}}$ is said to “immediately dominate.” ${\mathit{v}}_{\mathit{q}}$ if it strictly dominates ${\mathit{v}}_{\mathit{q}}$ and there are no other nodes positioned between them in the dominance hierarchy. Immediate dominance is especially valuable for analyzing direct dependencies among functions, enabling us to determine sequences where secure function calls should take place without interference or omission.

• Post-dominance: The concept of post-dominance is the opposite of dominance, where a node ${\mathit{v}}_{\mathit{p}}$ is said to “post-dominate” ${\mathit{v}}_{\mathit{q}}$ if every path from ${\mathit{v}}_{\mathit{q}}$ to the program’s exit point must pass through ${\mathit{v}}_{\mathit{p}}$. Post-dominance guarantees that specific critical cryptographic functions are carried out before the application concludes, preventing incomplete security operations.

Figure 7 demonstrates a sample code’s CFDT and post-dominance tree, illustrating how the hierarchical relationships assist integrity analysis.

Detection algorithm design

After building the CFCG and the CFDT, the following task is to implement a detection algorithm to find misuse vulnerabilities within the flow of functions in the cryptographic application. Our algorithm uses the CFCG and CFDT to establish a comprehensive process for comparing the actual sequences of cryptographic functions with those that should be present in a secure application. By conducting this comparison, the algorithm can identify any irregularities or deviations in function flow that may point to vulnerabilities.

Algorithm 1 describes the steps in this process and offers a structured approach for analyzing the integrity of cryptographic functions and detecting potential misuse vulnerabilities.

Experimental setup

The experiments took place in a controlled setting to replicate common cryptographic use cases found in Windows applications. We chose test cases from three primary categories:

• 1)

  Networked applications: These programs need secure network communication, such as messaging and file-sharing applications.

• 2)

  Data-processing applications: This category includes software that handles sensitive data, such as encryption tools and document editors with cryptographic features.

• 3)

  Malicious code samples: We selected specific samples from recognized malware databases to assess the strength of our detection algorithm against vulnerabilities in cryptographic functions.

Table 2 outlines the hardware and software environment used for testing, detailing the system configuration to guarantee that our experiments can be replicated and are reliable. By employing a mix of legitimate and malicious applications, we aimed to confirm the algorithm’s ability to differentiate secure applications from those that may have vulnerabilities.

Execution of the detection algorithm

The detection process was carried out following the steps outlined in Algorithm 1. Each application was analyzed to extract its CFCG and to create a CFDT, as described in “Extraction of Combined Dynamic and Static Cryptographic Function Invocation Graph” and “Construction of Cryptographic Function Dominator Tree”. This extraction and analysis were conducted using the static and dynamic analysis tools detailed in Fig. 6. We ensured thorough path coverage using static and dynamic analyses, effectively capturing all pertinent cryptographic function flows.

The algorithm performed the following steps for each application:

• 1)

  CFCG extraction and CFDT construction: Through static and dynamic analysis, the algorithm produced a CFCG, which was subsequently converted into a CFDT. This process offered an in-depth perspective on the relationships between cryptographic functions, as illustrated in Figs. 5 and 7.

• 2)

  Sequence analysis and comparison: The algorithm implemented its detection rules, contrasting the actual function call sequences in the CFDT with the expected sequences outlined in the cryptographic integrity model.

• 3)

  Identification of misuse vulnerabilities: Any deviations from the expected sequences were marked as misuse vulnerabilities, with specific patterns such as missing dominant nodes or altered function orders signaling potential issues.

Malicious code samples are detected within a virtual machine environment to protect the host system.

Analysis of detection results for a malicious sample

We evaluated our detection system using a malicious application identified on VirusTotal. Table 3 summarizes the detection results, emphasizing the misuse of vulnerabilities in cryptographic functions related to SSL secure connections. Our detection system revealed a significant flaw in how the sample handles cryptographic functions necessary for SSL connections.

Upon further analysis of the cryptographic module for SSL, the detection system noted that the application establishes an SSL connection and retrieves the certificate information from a remote server. However, as shown in Algorithm 2, the application fails to verify the received certificate, an essential step in validating the server’s identity. This omission makes the SSL connection vulnerable to man-in-the-middle (MITM) attacks, as it does not implement a secure authentication process.

Algorithm 2 visually represents the reverse analysis of the cryptographic function calls within this sample. In the code sequence, following the retrieval of the certificate information (j_SSL_get_peer_certificate()), the program only displays the details of the certificate’s subject and issuer without verifying its authenticity. This lack of verification creates a vulnerability in application integrity, permitting any SSL certificate, whether valid or not, to be accepted. As a result, the identity authentication process in this malicious application is ineffective, exposing it to potential security threats.

Analysis of detection results for CNKI Reader and Xunlei download

In addition to analyzing malicious samples, we also used our system to examine legitimate applications with known vulnerabilities, such as CNKI Reader’s CAJViewer (Feng et al., 2022) and Xunlei Download Manager (Kamalbayev et al., 2021). These vulnerabilities have been documented and reported in the China National Vulnerability Database (CNVD), which offers comprehensive details on the misuse vulnerabilities identified in these applications. In both instances, notable cryptographic weaknesses within their upgrade processes render them vulnerable to security threats.

CNKI Reader (CAJViewer) vulnerability analysis

The CAJViewer application displayed a vulnerability related to integrity misuse during its upgrade process. The detection results revealed no SSL secure connections in the update sequence. In particular, the upgrade executable (AutoUpgrade.exe) did not include a secure cryptographic function sequence required for authenticating the server during updates, as detailed in Table 4. This oversight indicates a serious misuse of cryptographic integrity and emphasizes the lack of a complete SSL authentication mechanism, which is essential for secure software updates.

Table 4:

Misuse vulnerability information for CNKI Reader and Xunlei download.

Vulnerability information	CNVD vulnerability ID	CNVD certificate ID
1-3 CNKI Reader CA (Viewer) (http://www.er arbitrary file download execution vulnerability)	CNVD-2018-02906	CNVD-YCGA-201801007490
1-3 Xunlei download software upgrade arbitrary file download vulnerability	CNVD-2018-06136	CNVD-YCGA-201803083461

DOI: 10.7717/peerj-cs.2641/table-4

Figure 8 depicts the update process for CAJViewer, which comprises two main steps:

• 1)

  The update program, AutoUpgrade.exe, initially requests an XML configuration file from the server. This file guides the update process, including essential information such as file names, sizes, and MD5 hashes.

• 2)

  Following the XML file parsing, the application downloads the updated files, verifies their integrity using MD5 checks, and then replaces the existing program version with the newly downloaded version before launching it.

Since the update process lacks SSL server authentication, it is susceptible to MITM attacks, which enable an attacker to intercept and alter the update process. Figure 9 illustrates a simulated MITM attack scenario where the client machine’s access to the legitimate CNKI update server is redirected to a fraudulent server. The server controlled by the attacker supplies a malicious XML configuration file and an executable containing the WannaCry ransomware virus. Consequently, when the client starts the update, the ransomware is downloaded and executed, jeopardizing the system.

Key findings and conclusions from detection results

Based on the outcomes of these assessments, several conclusions can be made about the effectiveness of the system and the severity of various types of misuse vulnerabilities:

• 1)

  Effective detection of misuse vulnerabilities: The detection system can accurately identify both parameter misuse and application integrity misuse vulnerabilities. In the case of parameter misuse vulnerabilities, the system successfully identifies misused cryptographic function addresses, parameter names, and values. Regarding application integrity misuse vulnerabilities, it retrieves cryptographic function call sequences from the target program and indicates missing critical functions.

• 2)

  Risk levels of misuse vulnerabilities: Various types of misuse vulnerabilities present different levels of risk. In general, application integrity misuse vulnerabilities that lack a complete cryptographic mechanism (such as SSL) are considered the most serious, as they make applications highly vulnerable to attacks. For example, vulnerabilities linked to crucial management or parameter misuse could allow attackers to decrypt sensitive information. Nonetheless, these parameter misuse vulnerabilities usually need specific conditions for exploitation, while the lack of a fundamental mechanism (as seen in the integrity misuse cases) poses an immediate threat.

Conclusion

This article proposes a detailed method for identifying misuse vulnerabilities in cryptographic applications through a structured analysis incorporating static and dynamic techniques. We developed a CFCG and a CFDT to visualize an application’s entire flow of cryptographic function calls, which helps spot security deviations.

Our proposed methodology overcomes the shortcomings of conventional detection methods that typically depend on only static or dynamic analysis, which may overlook indirect calls, dynamic dependencies, or detailed execution paths. Using our integrated approach, we achieved increased path coverage and improved accuracy in identifying cryptographic misuse vulnerabilities. The CFDT structure enabled us to efficiently identify and analyze critical security pathways, boosting the reliability and robustness of the detection process.

The experimental evaluation confirmed our algorithm’s effectiveness across various test scenarios, including both legitimate applications and malicious examples. The algorithm successfully determined misuse vulnerabilities across multiple application types, ranging from network communication tools to malware samples, thus offering insights into a broad array of cryptographic function flows. The detection rate was high, with few false positives, indicating that our method can reliably distinguish between secure applications and those at risk of misuse vulnerabilities.

Key findings from our research include:

• Enhanced vulnerability detection: The implementation of the CFDT provided a structured view of cryptographic functions, allowing for the accurate identification of missing or incorrectly ordered function calls essential for application security.

• Broad path coverage with low false positives: By merging static and dynamic analysis, our method captured indirect calls and dynamically loaded functions, minimizing the rate of false positives and establishing the algorithm’s suitability for complex software systems.

• Scalability and flexibility: The algorithm’s capability to adapt to various application types and cryptographic libraries demonstrates its potential for widespread application in detecting misuse vulnerabilities across multiple fields.

The proposed methodology presents a practical and dependable solution for enhancing cryptographic security in software applications. This approach can discover vulnerabilities that might remain unnoticed by thoroughly analyzing the flow and sequence of cryptographic function calls, thereby advancing software security standards.

Future work

Looking ahead, there are numerous avenues for future development. Subsequent efforts could focus on extending the algorithm’s adaptability to additional operating systems and cryptographic libraries, thereby widening its application scope. In addition, integrating machine learning techniques could enhance the algorithm’s capability to detect complex patterns of cryptographic misuse, further decreasing false positives and improving detection accuracy. As cybersecurity continues to evolve, we believe that methodologies like ours, which integrate comprehensive analysis with in-depth path validation, will be essential in promoting secure software development practices.

In summary, the detection approach introduced in this study is a valuable resource for safeguarding cryptographic applications from misuse vulnerabilities. By strengthening software with trustworthy cryptographic integrity checks, we can enhance the protection of sensitive data and contribute to a more secure online environment.