- Academic Editor
- Muhammad Aleem
- Subject Areas
- World Wide Web and Web Science, Programming Languages, Software Engineering
- © 2023 Leger et al.
- This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. For attribution, the original author(s), title, publication source (PeerJ Computer Science) and either DOI or URL of the article must be cited.
- Cite this article
Detecting bugs represents one of the most time-consuming tasks in software development (National Institute of Standards and Technology, 2002), and the development of Web applications is no exception. To alleviate this task, alongside the proposal of programming languages and Integrated Development Environments (IDE), a large number of debuggers that provide many different features (Balzer, 1969).1 Even though multiple debugger alternatives exist, most practitioners still rely on classic breakpoint-based debuggers (Perscheid et al., 2017). Such debuggers allow developers to pause the execution of a program, allowing them to step through the program execution (i.e., forward, into, or out of a given instruction). The advantage of this technique is to observe the values for selected program variables (Stallman, Pesch & Shebs, 2010; Dr. Racket, 2022). Other debuggers provide more advanced features like navigation through a program execution history (Pothier, Tanter & Piquer, 2007; Bousse et al., 2015; Barr et al., 2016), the re-execution of a program from a given execution point (UndoDB, 2022; Srinivasan et al., 2004; Bhansali et al., 2006; Choi & Srinivasan, 1998), or the remote monitoring of an execution (Session Stack, 2022; Raygun, 2022; TrackJS, 2022). However, the use of advanced debuggers faces two problems. First, developers consider debuggers complex to use, opting to use log approaches with print-like statements (Beller et al., 2018). Second, most existing debuggers are postmortem. That is, the analysis of the program can only occur after the execution has taken place, and only for a single execution path (i.e., a set of state values). As a result, classic debuggers only show the occurrence of a bug, forcing developers to execute their application multiple times to try to detect the bug over multiple value instances. This makes finding the cause of a bug difficult and time consuming, as the debuggers detect an instance of a problem, but provide no information about the underlying reasons for the program that occurred. We argue that offering the possibility of going back-in-time through the application’s execution, to replay a program using different values in an intuitive way, can improve understanding of the causes behind a bug.
Additionally, we propose a specialized Graphical User Interface (GUI) for DeloreanJS to ease the usability of its features, in response to the complexity that back-in-time debuggers might introduce (e.g., adding interactive timelines to navigate through the program’s execution history (Pothier, Tanter & Piquer, 2007)).
Using the back-in-time features, DeloreanJS allows developers to (1) improve understanding of a bug, and (2) experiment with hypothetical scenarios. DeloreanJS helps developers understand bugs as they can repeatedly modify variable values associated with a bug and resume an execution from the same timepoint, saving a large number of executions (i.e., time). Additionally, developers can experiment with hypothetical scenarios of a Web application execution through the interactive user interface, allowing the exploration and evaluation of diverse timepoints with different variable values.
Furthermore, to validate the usability of DeloreanJS, we conducted an empirical evaluation based on the System Usability Scale (SUS) approach (Brooke, 1996) (see the ‘Usability Evaluation’ section). Our evaluation consists of 15 computer science undergraduate students, with varying expertise levels, evaluating DeloreanJS in comparison to the built-in debugger in Mozilla Firefox (Mozilla Firefox, 2022). As a proof of DeloreanJS’ usability, all participants using DeloreanJS, recommended it; while 35% of the participants using Firefox’s debugger, recommended it.
In summary, our proposal constitutes an advancement in debugger research in three different dimensions, which we posit as our main contributions:
Back-in-time debugger for a mutable object-oriented language. DeloreanJS extends state-of-the-art features of back-in-time debuggers to enable the use of mutable states in the object-oriented programming paradigm.
Usable user interface for a back-in-time debugger. Advanced debugging features usually increase the complexity of using a debugger. DeloreanJS posits a user-friendly GUI to ease the navigation through timepoints and the generation of new execution timelines.
The rest of this article is organized as follows. The ‘Related Work’ section compares DeloreanJS to debuggers with existing approaches that have similar features. We then describe DeloreanJS and its crucial components in ‘Deloreanjs’. The ‘Validation: Deloreanjs in Action’ section presents our proposal in action through five examples. ‘Usability Evaluation’ presents the evaluation of DeloreanJS alongside a usability study based on the SUS approach. We end this article with a conclusion and describe the limitations of our proposal.
Availability. A proof-of-concept implementation of DeloreanJS with the tests presented in this article is available at http://pleger.cl/sites/deloreanjs (DeloreanJS, 2022). The source code is available from the following GitHub repository: http://github.com/fruizrob/delorean (revision 5f98bc6). Our proposal currently supports Google Chrome (Google Chrome, 2022) and Mozilla Firefox (Mozilla Firefox, 2022) browsers without any need for extensions or plugins.
Table 1 shows a comparison of debugger features in approaches that consider the execution history of an application. This table shows 15 debuggers with their supported approaches and features. For each debugger, some features are supported (black circles), and some are not supported (white circles). In the last row, we compare these debuggers to our proposed debugger. Considering the approach of these debuggers, we can classify them to four groups:
Runtime verification tools (Meredith, 2012) are not strictly defined as debuggers, nonetheless, these tools have a behavior similar to that of DeloreanJS given that errors can be detected at run time. Available runtime verification tools include PQL (Martin, Livshits & Lam, 2005), PTQL (Goldsmith, O’Callahan & Aiken, 2005), and JavaMOP (Meredith et al., 2011; Chen & Roşu, 2007). These tools allow developers to express the complex patterns of an application’s execution, for example, detecting access to an item that is not available in an array because another execution thread removed this item. Unfortunately, similarly to previous debuggers, these tools cannot resume an execution from a specific point of the computation history with different variable values.
To go back-in-time to a specific point in the execution of a Web application, the timepoint abstraction is crucial. This is because a timepoint captures and stores the control state of a Web application in terms of the: program counter, stack, and (partially) heap. Figure 4 shows how we capture the program counter and stack using continuations (Friedman & Wand, 1984). Variables from the stack and heap are captured using static analysis (Cousot & Cousot, 1977).
Capturing the program counter and stack
Figure 5 shows the workflow of Listing 1. Line 5 shows a continuation kont created before adding the y variable. This execution capture occurs on line 9 when the add function is called. The result of add is passed to show, and the number 6 (6 = (x = 5) + (y = 1)) is shown. Line 10 invokes the continuation stored in kont with the parameter 20, resulting in the return value of the anonymous function between lines 5–6 as 20 and not 1. As a result, 25 (25 = (x = 5) + (y = 20)) is displayed. Note that the if expression statement (line 6), and the if statement (line 10) are used to differentiate between the creation of a continuation and its invocation. A continuation is a function when it is created (line 5); and when the continuation is invoked, it is bound to the value passed as a parameter (e.g., the value 20 in our example).
Capturing variables with their values
The static analysis allows DeloreanJS to instrument the source code to store (during the execution) the values of watched variables. Figure 6 exemplifies the two steps in our static analysis. Considering a lexical scope strategy, Step 1 captures and stores the values of defined watched variables (e.g., v1 and v2). In Step 2, the static analysis also captures and stores the variables that modify watched variables, i.e., that have dependencies to watch variables (e.g., a and b). As a result, DeloreanJS creates a dependency tree for each watched variable, where a node contains a variable that (transitively) modifies the watched variable (e.g., the two trees shown in Fig. 6). This last step follows a reactive programming (Wan & Hudak, 2000) strategy and is necessary to ensure watched variables evolve consistently with the variable values when an application resumes from a timepoint. For example, in Fig. 6, if the a variable is not captured, then the v2 variable would have a different value when the application resumes its execution from a timepoint. To implement these two steps, DeloreanJS first creates an Abstract Syntax Tree (AST) from the source code. Using the AST, DeloreanJS applies the Visitor (Gamma et al., 1994) design pattern to visit every node of the AST to: (1) create a dependency tree for each watched variable, and (2) capture and store it at runtime any modification to variables in the dependency tree.
If watched variables are associated with objects, developers can select between a shallow or deep copy of objects in DeloreanJS’ user interface. The first option only copies the references to other objects, while the second option clones these objects. Both options contain a tradeoff that is necessary to consider. On the one hand, if the shallow copy option is used, developers may not keep an exact version of an object through the time, i.e., developers can resume an execution from a timepoint with inconsistent memory (e.g., object properties with future values). On the other hand, if the deep copy option is used, memory usage significantly increases, and it is possible that references of nested objects may not be changed (e.g., modification of an object reference in a place that our debugger is not supervising). In the current implementation, the DeloreanJS’ user interface allows developers to choose between the two approaches.
Inserting and using timepoints
Developers can explicitly insert timepoints using the delorean.insertTimepoint(String) method, and DeloreanJS can implicitly insert timepoints when watched variables and their dependencies change their values. To insert implicit timepoints, the debugger instruments the source code before the execution, to add a timepoint every time that a variable of any dependency tree changes its value (Fig. 6).
Figure 7 shows what happens when (1) a timepoint is inserted, and (2) the inserted timepoint is used. When a timepoint is inserted DeloreanJS creates and stores a Timepoint object, which contains a new continuation and an object that stores the watched variables with their dependencies. When a developer selects a specific timepoint (e.g., TP) using DeloreanJS’ user interface, the debugger invokes the continuation stored in the timepoint and subsequently modifies the values of the watched variables along with their dependencies.
Validation: DeloreanJS in action
This section introduces the main functionalities of DeloreanJS and its inner workings through five examples extracted from a MIS (Laudon & Laudon, 2016) that manages student grades at a university. To gradually introduce the debugger, the complexity of the examples is incrementally increased.
Bug detection and fixing
A common task for a MIS is to calculate the final grade of a student, according to an evaluation strategy assigned to a course. We want to be able to identify errors, in a meaningful way, whenever this calculation cannot take place. Listing 2 shows an example of how to manage errors during the calculation. In this example, we calculate the final grade for a student of the course “Algebra.”. Here, an exception is triggered because the variable “courseName” contains an incorrect name for the course (“Alggebra”), which does not have an evaluation strategy assigned to it. Although the goal of DeloreanJS is to detect bugs, we can also use our tool to fix bugs at runtime. To do so, a developer must first add the courseName to the list of watched variables using the user interface; then the developer inserts an explicit timepoint (line 4) to be able to go back-in-time to it. When an exception is triggered, the program stops its execution. As we are attempting to invoke a function that is not in the evalStrategies array (line 9), the developer can select a timepoint such as StrategyNotFound using the DeloreanJS user interface and change the courseName value to the correct course name: “Algebra.”. Finally, the developer can resume execution from the selected timepoint for a successful calculation.
Improve the understanding of a bug
Another desirable feature of the MIS is to generate a report that contains the average grade of all courses at a university. This feature is implemented in Listing 3. Similar to the code in Listing 2, the course “Algebra” is misspelled. However, this scenario is more complex as the exception is triggered within the execution of the loop, and has many possible iterations (e.g., there can be more than 1,000 courses per semester). Knowing which iteration and why it triggers an exception in a loop can be an extremely time-consuming task for developers. The use of breakpoints, available in existing debuggers, does not necessarily ease the task at hand as breakpoints do not react to an exception, but react to the execution of a statement. In the case of a loop, such statements may be executed several times, having to stop at each of them. For this example, DeloreanJS allows developers to save the execution state from many executions (i.e., loop iterations), and reuse the execution context of the application (i.e., go back to a specific iteration and try different values). This is helpful information for developers to understand the bug. In Listing 3, a DeloreanJS timepoint is inserted for each iteration of the loop while the program is executing. When an exception is triggered, a developer can go back in time to any iteration of this loop to find the causes of a bug, improving their understanding of the reason for the bug. Again, in this example, we observe that DeloreanJS allows developers to watch and modify objects and their properties (e.g., courseNames) to explore execution alternatives.
Clarify unexpected results
Experiment with hypothetical scenarios
Experimenting with several hypothetical scenarios without the need to re-run an application from the beginning may save time for testers. Developers can use DeloreanJS to experiment with different execution scenarios by reusing timepoints with different values for watched variables. We illustrate this feature through three potential reports that this MIS can show depending on the value of the realMean variable, as shown in Listing 5. Using DeloreanJS with explicit timepoints triggered by exceptions, a tester can (re)use the timepoint TestingDifferentResults to modify the value of realMean and explore the behavior of the system when it displays different reports.
Revisiting: improve the understanding of a bug
This does not strongly highlight its benefits. DeloreanJS’ user interface allows developers to activate the implicit timepoints option, so that timepoints are automatically added each time that a watch variable is modified. With implicit timepoints, a developer can navigate in the computation history of a program through the selection of timepoints that represent value modifications of watch variables. Other proposals (Pothier & Tanter, 2009; Barr et al., 2016) have shown the benefits of navigating through a program’s execution history to understand a bug; this is mainly because it is possible to find when a variable is bound to an unexpected value. For example, Fig. 3 shows the execution history through the timepoints with their associated timestamps; in each timepoint, developers can watch variable and object property values at that execution point. We illustrate the use of implicit timepoints for navigation through the second example of this section, when we attempt to generate a report that contains the average grade of all courses at a university. Without adding any call to a DeloreanJS method, the developer is able to watch the evolution of variable values in each iteration of the (long) loop, and, of course, resume the execution from a selected timepoint.
Figure 8 illustrates the navigation that a developer can use with DeloreanJS. The figure shows the evolution of a variable named average in two different execution points, 7 and 8 ms (ms), that are contained in two timepoints. Whereas the value of average is 0 at 7 ms, this value changes to 59.1 at 8 ms. As many timepoints can be created at the exact millisecond, the user interface groups these timepoints in one point to simplify the interface.
We have presented DeloreanJS through different concrete examples, which show the use of explicit and implicit timepoints to deal with bugs. Using timepoints, we have shown how to modify values of variable or object properties using a Web interface while an application is running. Although we use the first three examples with explicit timepoints, the usefulness of DeloreanJS comes from employing implicit timepoints, as developers can navigate through the evolution of values in the execution history of a Web application.
Graphical user interface
As mentioned in Section 4, DeloreanJS’ GUI is mainly inspired by Visual Studio Code (Visual Studio Code, 2022), which provides a familiar user interface for developers.3 To interact with a timeline, we borrow the interface used in TOD (Pothier & Tanter, 2009) and other IDEs (Field et al., 2022). Additionally, as Fig. 3 shows, we include the timepoint interactions, the support of multiple timelines, and a panel to modify the property values contained by a timepoint. Readers can check out DeloreanJS’ GUI on its website (DeloreanJS, 2022).
Percentage of participants that detected a bug. Figure 9 compares the percentage of participants that were able to detect a bug using each of the platforms debugger: DeloreanJS, and FireFox. For both debuggers, most participants (over 75%) could detect the bugs for all tasks. Note that all participants using DeloreanJS could detect the bug in task 2 (Listing 6) while only 93% of the participants could detect the bug using FireFox’s debugger. The difference in success rates may be because the piece of code in task two presents an unexpected behavior (NaN as a result) and not a runtime exception. Participants using DeloreanJS could use timepoints to find the moment when the result becomes NaN.
|Firefox||Undergraduate Students||15||University of the Andes (Colombia) - Universidad Católica del Norte (Chile)|
|DeloreanJS||Undergraduate Students||15||University of the Andes (Colombia) - Universidad Católica del Norte (Chile)|
|1||Detect a simple bug|
|2||Detect a bug into a loop|
|3||Experiment with different and hypothetical scenarios|
|4||Detect a bug using implicit timepoints|
|5||Detect a bug in advanced structures|
Average time to detect a bug. Figure 10 shows the average time per task that participants used to detect the bug. In the first task, participants using DeloreanJS, detected the bug significantly faster than the participants using Firefox’s debugger, with a net difference of 4 min. In the remaining tasks, the average time to solve each task is similar for both debuggers. It is possible that the difference in time required to locate the bug in the first task is related to the fact that the first time participants used either debugger, and Firefox’s debugger seems more complex (cf. Section 5.3.1).
To evaluate and compare DeloreanJS with Firefox’s debugger in respect to their usability, we employed the SUS approach (Brooke, 1996). With the SUS approach, a set of participants that used a product, service, or application, are asked to score ten items using a Likert scale (Albaum, 1997) of five levels (from “Strongly agree” to “Strongly disagree”). The ten items are presented as statements that a participant scores:
“I think that I would like to use this system frequently”
“I found the system unnecessarily complex”
“I thought the system was easy to use”
“I think that I would need the support of a technical person to be able to use this system”
“I found the various functions in this system were well integrated”
“I thought there was too much inconsistency in this system”
“I would imagine that most people would learn to use this system very quickly”
“I found the system very cumbersome to use”
“I felt very confident using the system”
“I needed to learn a lot of things before I could get going with this system”
To calculate a global score, we follow a three-step procedure. The global score is in the range of 0–100, which determines its usability according to Fig. 11.4
Add up the total score for all odd-numbered questions, then subtract five from the total to get total-odd.
Add up the total score for all even-numbered questions, then subtract that total from 25 to get total-even.
Add total-odd and total-even, and the result is multiplied by 2.5.