A low-cost unity-based virtual training simulator for laparoscopic partial nephrectomy using HTC Vive

Details and challenges in LPN

LPN has emerged as a viable alternative to open nephrectomy while minimizing patient morbidity. The primary emphasis of this study is the treatment of exophytic renal tumors, which have benefited from improvements in laparoscopic experience. The Da Vinci Surgical System is usually used for surgical practices as it is safe and accurate. However, the machine is very costly. The main difficulties during the operating process that require practice are:

• 1) To prevent unnecessary harm to the kidney and its adjacent areas, the tumor’s precise location must be determined.

• 2) To stop unmanageable bleeding, the renal artery, renal vein, and ureter are clipped and mobilization of descending colon is ensured.

• 3) Adhesions between the kidneys can be successfully separated to avoid operation failure.

• 4) Laparoscopic tool insertion needs to be done cautiously since inexperienced handling can harm the intestines and major blood arteries.

The surgeon can practice and complete a variety of partial nephrectomies efficiently if they can manage this broad range of indications. Figure 1 is added here to show the steps involved in a typical nephrectomy procedure. The patient must be placed correctly, and the camera and other equipment must be inserted through trocars as the initial step. Figures 1A and 1B shows the mobilization of descending colon; it is necessary to forcibly ligate either the inferior mesenteric artery or the left colic artery in order to mobilise the descending colon. As a result, the inferior mesenteric artery stops contributing to the colon’s blood supply. The next step involves the identification of renal artery, veins and other features. Figures 1C and 1D represents clipping of renal artery and cutting the renal vein respectively. Then the final step is to dissect cancerous tissues around the kidney as shown in Fig. 1E. Our team has reproduced some of the same processes using the HTC Vive, which is significantly less expensive than the Da Vinci machines, by simulating some of the same phases using a simple mesh representation. We have focused on the important step of accurately separating cancerous tissues from the rest of the structure because the others can be practised in various training sessions. The surgeons starting their first LPN practise can use our solution.

3D modeling

With the help of simplex meshes, which are topologically dual triangulations, we have modeled the kidney and the cancerous part. Depending on the local energy, these meshes are more adaptable. It is common practice to employ this method for effective physics-based modeling. It enables reliable and effective object deformation.

Joining components: Figure 4 shows the kidney model consisting of different 3D components (renal artery, vein, and cancerous part) extruded and then joined together using Blender’s Join operation. Figures 4A and 4B present the 3D structure of a healthy kidney with a solid and wireframe view, respectively. Pertinent materials have been used to understand better and match the organ’s color palette. Figure 4 shows a solid view in Fig. 4C and a wireframe view in Fig. 4D after the malignant area has been incorporated onto the mesh surface.

Subdivision modifier: Subdivision surface modification divides the mesh triangles into subsurfaces. Only a few properties were changed; the mesh’s core topology remained unchanged. The models are depicted in Figs. 4E and 4F after the modifier has been applied. Since the mesh triangles are clearly sparse, the rocky mesh surface in Fig. 4C is not plausible whereas the model presentation in Fig. 4E is smoother and more aesthetically pleasing. As the CPU or GPU must compute more triangles in Fig. 4E than the model Fig. 4C, this smooth structure significantly burdens the system. The model in Fig. 4C was chosen for simulation because it balances computational expense and an aesthetically realistic model.

Force-based soft body simulation

In this particular technique, the soft body’s structure is defined by a triangular mesh, and the constraints exert the necessary pressure to produce the volumetric effect. The forces such as spring, damping, pressure, and offset directional force are considered when modeling each surface point. These forces exert the pressure on a specific area while keeping the total number of vertices constant. As a result of the interaction of these forces, the entire item would shift without altering its original structure.

We define the 3D soft body, SB3D, as

$$S{B}_{3}D=S,F,V,N,C$$whereas S here defines the soft body,

• $F=\{F_j||j=0,\dots ,m-1\}$ represents the set of forces for each point j.

• $V=\{V_j||j=0,\dots ,m-1\}$ represents the set of velocities for each point j.

• N represents the count of total edges.

• C represents constraints set being observed for object deformation.

Combined force effect: The composite force, $F_j$ is centered on elasticity control force, pressure control force, damping control force, and a direction offset component from the point j on the body. The composite force $F_j$ is defined by the Eq. (1) as:

(1) $$F_{\mathrm{j}}=F_{\mathrm{s}\mathrm{j}}+F_{\mathrm{d}\mathrm{j}}+F_{\mathrm{p}\mathrm{j}}+F_{\mathrm{o}\mathrm{j}}$$where

• $F_{sj}$ represents the spring force at point j,

• $F_{dj}$ represents the damping force at point j,

• $F_{pj}$ represents the pressure force at point j, and

• $F_{oj}$ represents the offset force at point j.

Consider this simulation system with the numerical integration of Hooke’s spring force described through Eq. (2).

(2) $$F_{\mathrm{j}}=m_{\mathrm{j}}.\frac{{\mathrm{\partial }}^2\overrightarrow{r_{\mathrm{j}}}}{\mathrm{\partial }{t}^2}$$

The pressure force applied will deform the body shape acting in the direction of normal vector $\hat{n}$ as described in the Eq. (3).

(3) $$P=\overrightarrow{P}.\hat{n}[\frac{N}{m^2}]$$where P is a pressure value and $\hat{n},$ is a normal vector to the surface on which pressure force is acting. For calculating specific pressure forces, we multiplied pressure by the surface area, giving us Eq. (4).

(4) $$\overrightarrow{F_{\mathrm{P}}}=\overrightarrow{P}.\mathrm{\Delta }A[N]$$

Algorithm 1 represents the soft body functioning of this model.

Detecting user input: A ray is tossed into the scene from the camera location. Unity’s physics engine processes the location data of the object. We move across the mesh surface after pressing the controller trigger buttons to make a cut along the triangles and calculate the locations of the points. An elastic effect results from adding the total forces at the contact locations. The Eq. (5) represents the drag force that each surface point encounters until it reverts to its initial position.

(5) $$v_{\mathrm{d}}=v(1-d\mathrm{\Delta }t)$$when the damping force is high, the mesh loses some bounce. Figure 5A presents programmable spring and damping forces variables. The user can easily alter the settings for these variables, giving them full control over how firm the created soft body impression will be.

Moving vertices under the pressure force: The AddPressureForce() function iteratively adds the required pressure force to each of the newly discovered locations. To ensure that the vertices are constantly driven into the surface and follow the normal to the contact points, we can add a slight direction offset to this loop. Figure 5B depicts pressure and offset force variables.

Through Eq. (6), we can find the exact distance between vertices and the direction in which force is applied.

(6) $$F_{\mathrm{v}}=\frac{F}{1+d^2}$$

One plus the squared distance ensures the full-force strength even if the distance is zero. Once we know the force, we can use the relation $F=\frac{m}{a}$ and $a=\frac{\mathrm{\Delta }v}{\mathrm{\Delta }t}$ to calculate the approximate change in velocity. Considering the mass m = 1 for each vertex, we finally have the Eq. (7).

(7) $$\mathrm{\Delta }v=F\mathrm{\Delta }t$$

Now, the vertices can be moved with an absolute velocity updated every frame. The mesh is then given a new shape by adding the moved vertices. The normals for the newly discovered vertices need to be recalculated. Each vertex position is adjusted by the Eq. (8)

(8) $$\mathrm{\Delta }p=v\mathrm{\Delta }t$$

Dealing with mesh transformation: An elastic surface is represented by nothing more than a collection of vertices; it lacks a true volume, unlike solid real-world objects. Vertices start to move during cutting as soon as we apply pressure to them with the desired spring and damping effect. Collision and bending constraints in PBD are then used to handle this movement. Since the mesh colliders for the kidney model are unaffected by these forces, the natural shape of the object does not alter throughout the simulation.

Constraint based physics approach

Model points in PBD are connected by the triangular mesh edges and shared connection points. The positions of the points and the force’s direction (velocity is always tangent, constraint force is still normal) are determined directly during the simulation. Because of this, PBD can effectively address the instability issues. Equations (9) and (10) provide the central idea of sequentially resolving the system with constraints of equality and inequality in different time steps.

(9) $$C_{\mathrm{j}}(x_{\mathrm{i}\mathrm{j}},...,x_{\mathrm{i}{\mathrm{n}}_{\mathrm{j}}})=0$$

(10) $$C_{\mathrm{j}}(x_{\mathrm{i}\mathrm{j}},...,x_{\mathrm{i}{\mathrm{n}}_{\mathrm{j}}})\ge 0$$when simulating, neighboring mesh triangles will respond to bending restrictions under the same stretching influence if one mesh triangle reacts to the stretch amount in the direction normal to the force. The stretching constraint function is described for each connecting edge through Eq. (11).

(11) $$C_{\mathrm{s}\mathrm{t}\mathrm{r}}(x_1,x_2)=|x_1-x_2|-d_{12}$$where $d_{12}$ is the length of the edge at rest position, the bending constraint generated for any two triangles is described through Eq. (12).

(12) $$C_{\text{bend}}(x_1,x_2,x_3,x_4)=arccos(n_1.n_2)-{\phi }_{12}$$where $n_1$ and $n_2$ represent the normal vectors in the rest position and ${\phi }_{12}$ is the dihedral angle between the triangle 1 and 2. The Eq. (13) for minimizing the total energy defined as:

(13) $$E^{\mathrm{t}\mathrm{o}\mathrm{t}}=\sum (k_{\mathrm{s}\mathrm{t}\mathrm{r}}{C}_{\mathrm{s}\mathrm{t}\mathrm{r}}^2+k_{\mathrm{b}\mathrm{e}\mathrm{n}\mathrm{d}}{C}_{\mathrm{b}\mathrm{e}\mathrm{n}\mathrm{d}}^2)$$where $k_{str}$ and $k_{bend}$ are global stiffness parameters.

This section outlines the fundamental operation of Algorithm 2: the solver iterations compute and project the position constraints. The calculated points are relocated to their projected locations along with updating the point velocities. Distance limitations represent the connective component of the malignant tumor in the kidney. Depending on the stiffness parameter and rest length, each connection is breakable. At a specified break threshold, algorithms tear apart the highlighted locations. However, the deformation does not appear visually plausible because there is just a single layer of a soft structure in our method and no simulation of fascia tissues. The fascia resembles a torn piece of article once the malignant component has been removed from the kidney. The faster solution time brought on by, the smaller number of mesh triangles is still a significant benefit.

Force feedback comparison

The haptic HTC Vive device gives the user a powerful sense of presence. The continuous movement provided by the responsive haptic feedback enhances cognition.

The headset offers a 110-degree field of view, a refresh rate of 90 Hz, and a resolution of 1,080 $\times$ 1,200 pixels. According to Table 1, the refresh rate is nearly identical to that of most sensory devices. However, we are enjoying a cost-benefit here with the HTC Vive Pro VR device.

The SteamVR plugins in Unity allow us to manipulate haptic sensors in communication with organ tissues. Figure 6 demonstrates the haptic device integrated workflow.

The geometry track and the details of the initial interaction are immediately processed. The controller placement and haptic proxy both keep shifting after that. The quick refresh rate gives a credible visual representation of soft tissue deformation in a malignant portion. The PBD-based collision model occasionally removes the triangular meshes to maintain stability. This distortion provides immediate vibrational and visual feedback, which we call the system’s haptic force feedback. The sensory device performance has proved satisfactory while deforming the soft structure in this simulated situation.

Simulator performance comparison using unity profiler

The first simulation experiment was carried out on a system equipped with an Intel(R) Core(TM) i7-7700 CPU running at 3.60 GHz, an x64-based processor, and 8.00 GB of installed memory. The second testing experiment used a high-end graphics system, the GeForce GTX 1060, with Max-Q Design 6.00 GB GDDR5 memory on the card, 1,280 shading units, and 80 texture mapping units. This Max-Q design has the advantage of less power consumption as compared to other graphics cards.

The “CPU/GPU Usage Profiler” in Unity shows information about the duration of various operations and events, including rendering, physics, animations, and scripting. This enumerates every significant region where our system spends most of its time. Figure 7A displays the CPU used for displaying the simulation at a certain timestamp, and Fig. 7B depicts the same simulating system rendering on the GPU. It is clear from Fig. 7B that the majority of the GPU-intensive rendering time is consumed by the “Game Player Loop” and its hierarchy.

Tables 2 and 3 show the complete component details for Unity’s renderer and memory profilers.

When a significant number of trainers are required, GPU necessitates a high processing unit with a lot of memory, which is fairly expensive and difficult to organize by the hospital authorities. The CPU computation speed was fairly good with a tiny time step, but the user had to make certain sacrifices in terms of deformation performance. The experience was significantly enhanced by the GPU’s capacity to perform parallel processing. With GPU, our simulation’s overall computation time was reduced with a rapid convergence rate.

Face and content validity for simulator performance evaluation

To evaluate the overall performance and usability of the simulator, we have used some face and content validity measures, as presented in Table 4. Each measure has a score range (0–9) (0 indicates a poor score, and nine indicates an excellent score).

The nephrology department at Shaikh Zayed Medical Complex Lahore recruited seven final-year students and five interns to participate in this study. The study‘s objectives and the proper use of the simulator were briefly explained to the participants. We are quite appreciative of their assistance in this. Each evaluation metric was given a score according to a scoring system. In Fig. 8, the mean score values from the student and intern groups, representing the face and content validity tests, are plotted.

All participants stated the virtual world was easy to use and that the HTC Vive haptic device helped them practice their skills. Both groups provided favorable feedback about the platform’s precision and interactivity, the metrics with the highest scores. The student group suggested that more realistic graphics be made. The intern group found instrument mapping needed to be more successful when carrying out the process. We treated the laser pointer as the laparoscopic equipment for simulation purposes. The simulator cannot currently teach any experienced surgeons, both groups agreed. It can, however, help new surgeons become more proficient at cutting tissue. Future testing methods and procedures will be increasingly sophisticated to produce more accurate results. Under the direction of medical professionals, we want to improve the anatomy, force feedback, and simulator design and stability.

Game-based system advantages and limitations

Using a constraint-based approach like PBD in combination with Unity has given us considerable advantages, along with some limitations.

HTC Vive as a surgical trainer

Numerous studies have demonstrated that virtual simulators do not offer realistic force or haptic feedback, nor are they extremely accurate and constant. Another big disadvantage of these high-fidelity VR simulators is their exorbitant price. Additionally, the annual cost of maintaining hardware and software is substantially higher. Even though these gadgets significantly impact surgical training as a whole, their high cost can make them an undesirable option compared to alternative resources that are similarly priced. An affordable option that offers a vibrating effect for tactile movement is the HTC Vive head-mounted display. This effect has been considered haptic feedback from the system. We have contrasted our suggested system’s cost and functionality with some of the more common virtual simulators offered commercially. A cost comparison analysis is displayed in Table 5. Our research has demonstrated that, with some performance and feedback functionality compromises, the HTC Vive can be the most reasonably priced VR-based surgical trainer when used with an open-source game engine.