Deep reinforcement learning task scheduling method based on server real-time performance

Task analysis model

The user sends a task request to the cloud platform for handling, and the task information corresponding to user requests is stored in a task queue to form a user task request queue. A task request queue contains a series of tasks, which can be expressed as $\mathrm{Q}=\left\{T_1,T_2,\cdots ,T_n\right\}$, where $T$ represents a task, and $n$ means on number of tasks in the task request queue. Tasks do not interfere with each other, cannot be preempted, and can be executed in parallel when meeting the current virtual machine load and own resource requests. An essential attribute of task information is the task length. The task length $\mathrm{L}$ is subjected to a normal distribution with mean $\mu$ and standard deviation $\sigma$, that is, $\mathrm{L}\sim N\left(\mu ,{\sigma }^2\right)$. Of course, the arrival time of tasks $T^a$, is also a vital attribute of task information, which determines the order in which the DRL model allocates virtual machines to tasks. As a classic distribution, the Poisson distribution is widely used in queueing theory and can be used to describe the pattern of task arrivals. Hence, the arrival time $T^a$ of tasks defers to the Poisson distribution with parameter λ, that is, $T^a\sim P\left(\lambda \right)$. At the same time, the response time $T^{res}$ required for task execution on the cloud platform needs to be no greater than the deadline $T^D$ to ensure that the task can be responded to and processed normally. Tasks with a response time exceeding $T^D$ will be considered failed tasks during runtime performance parameter evaluation.

(1) $$\begin{array}{c}{T}^{res}=T^w+T^e\le T^D\end{array}$$where $T^w$ represents the waiting time required to fulfill task execution, and $T^e$ is the execution time of the task. The waiting time $T^w$ and the execution time $T^e$ will be influenced by the scheduling decisions made by the DRL model.

To validate the robustness of the proposed task scheduling method and to show the diversity of task requests, we randomly sample the resource requests of tasks for CPU, memory and disk I/O in the range of $\left[0,\theta \right)$. As a result, in this scheduling model, each task is represented as

(2) $$\begin{array}{c}{T}_i=\left\{l_i,T_i^a,req{C}_i,req{M}_i,reqI{O}_i,T_i^D\right\}\end{array}$$

(2a) $$\begin{array}{c}subjecttoreq{C}_i,req{M}_i,reqI{O}_i\in \left[0,\theta \right)\end{array}$$where $l_i$ is the task length of the $i$-th task, $T_i^a$ is the arrival time of the $i$-th task, $req{C}_i$ is the CPU resource request of the $i$-th task, $req{M}_i$ is the memory resource request of the $i$-th task, $reqI{O}_i$ is the disk I/O resource request of the $i$-th task, and $T_i^D$ is the deadline of the $i$-th task, $\theta$ denotes the load threshold of the virtual machine.

The waiting time $T^w$ of the task is determined by the virtual machine state chosen by the DRL model. This virtual machine state will be affected by previously executed tasks, especially the end time of the task.

(3) $$\begin{array}{c}{T}_i^E=T_i^a+T_i^{res}\end{array}$$where $T_i^E$ is the end time of the $i$-th task, $T_i^a$ and $T_i^{res}$ are the arrival time and response time of the $i$-th task respectively.

Thereby, the waiting time of the $i$-th task is

(4) $$\begin{array}{c}{T}_i^w=\{\begin{array}{c}0,iffirsttaskorcondition\hfill \\ \mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}\left\{T_k^E|condition\right\}-T_i^a,otherwise\end{array}\end{array}$$

(4a) $$\begin{array}{c}\mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}V{M}_k=V{M}_i=j,k\in \left[1,i-1\right]\end{array}$$

(4b) $$\begin{array}{c}{T}_k^a<T_i^a,T_i^a<T_k^E,T_k^{res}\le T_k^D\end{array}$$where $condition=fulfillreq{C}_i,req{M}_i,reqI{O}_{i}and(loa{d}_j^t+loa{d}_i)\le \theta$, $V{M}_k$ and $V{M}_i$ are the virtual machines selected by DRL to execute the $k$-th task and the $i$-th task. The above formula $V{M}_k=V{M}_i=j$ means that the virtual machines selected by the two tasks are the same, both are the $j$-th virtual machine. And $loa{d}_j^t$ is the real-time load of the $j$-th virtual machine at time $t$, and $loa{d}_i$ is the load generated by the execution of the $i$-th task.

When a task is run on a virtual machine, the task will produce a certain load on the virtual machine, and the virtual machine load will have a certain impact on the performance of the virtual machine, that is, the processing speed $V$ of the virtual machine. In moment $t$, the $i$-th task commences execution on the selected $j$-th virtual machine. The task will be processed at the real-time processing speed $V_j^t$ corresponding to the real-time load of the virtual machine at moment $t$, thus obtaining the execution time $T_i^e$ of the current task.

(5) $$\begin{array}{c}{T}_i^e={\displaystyle \frac{l_i}{V_j^t}}\end{array}$$where $l_i$ is the task length of the $i$-th task, and $V_j^t$ is the real-time processing speed of the $j$-th virtual machine selected at the moment $t$ when the $i$-th task starts running.

Real-time performance platform model

The real-time performance platform is composed of a group of virtual machines, which can be denoted as $\mathrm{V}\mathrm{M}\mathrm{s}=\left\{V{M}_1,V{M}_2,\cdots ,V{M}_m\right\}$, where $\mathrm{V}\mathrm{M}$ represents a virtual machine, and $m$ indicates the total count virtual machines on the current cloud platform. A virtual machine is an independent and isolated computing environment that allows users to feel like they are using a real physical machine. Similar to physical machines, virtual machines include CPU, disk, memory, and processing speed. The processing speed of virtual machines is a crucial factor considered, which is extended to virtual machine performance in this article. The DRL model utilizes current state information to make decisions for each task, assigning them to the most appropriate virtual machine for execution. In this article, the processing speeds of different virtual machines are assumed to be consistent. Still, due to varying loads during task execution, the processing speeds under different loads naturally differ. Meanwhile, load thresholds $\theta$ are set for virtual machines. Once this threshold is exceeded, tasks cannot execute and enter the virtual machine waiting queue until the virtual machine load meets the needs of the task, which is

(6) $$\begin{array}{c}{T}_{i}runningstatus=\{\begin{array}{c}Execute,ifcondition\\ Wait,otherwise\end{array}\end{array}$$

For the literature (Toumi, Brahmi & Gammoudi, 2022), the resource contention rate directly affects the server computational performance in the cloud, i.e., when tasks are executed in parallel, the contention rate for the resources increases which leads to the rapid degradation of the server performance. The general linear function is difficult to reflect the complex relationship between load and performance, whereas the power function as a nonlinear function has good nonlinear fitting ability. Consequently, combining the above discussion and considering the mutual interference of parallel execution of tasks, this article expresses the relationship between virtual machine performance and load as follows

(7) $$\begin{array}{c}{V}_j^t=V_j\times \left(1-\sqrt{loa{d}_j^t}\right)\end{array}$$where $V_j^t$ represents the real-time processing speed of the $j$-th virtual machine at time t, $V_j$ is the originally defined processing speed of the $j$-th virtual machine, and $loa{d}_j^t$ is the real-time load of the $j$-th virtual machine at time $t$.

In this scheduling model, each virtual machine is represented as

(8) $$\begin{array}{c}V{M}_j=\left\{V_j,loa{d}_j,C_j,M_j,I{O}_j\right\}\end{array}$$where $V_j$ is the original definition of the processing speed for the $j$-th virtual machine, and $loa{d}_j$ represents the load of the $j$-th virtual machine, with values ranging between [0, 1]. $C_j,M_j,I{O}_j$ respectively represent the CPU, memory and disk I/O resources of the $j$-th virtual machine. For this article, $C_j,M_j,I{O}_j$ = 1.

Real-time load model

According to the current environment status, the DRL model determines the appropriate virtual machines for task allocation and execution. Considering the diversity and dynamics of task demands and the continuity of tasks, the subsequent arrival and execution of tasks are influenced by the impact of previously arrived tasks on the performance of virtual machines. To accurately capture the real server load dynamics, in this article, the virtual machine global load score in Elsakaan & Amroun (2024) is adopted as the load caused by task requirements on the virtual machine. Therefore, the load generated by the resource request of CPU, memory, and disk I/O by the $i$-th task is

(9) $$\begin{array}{c}loa{d}_i=\alpha \ast req{C}_i+\beta \ast req{M}_i+\gamma \ast reqI{O}_i\end{array}$$

(9a) $$\begin{array}{c}subjectto\alpha +\beta +\gamma =1\end{array}$$where $\alpha$, $\beta ,$ and $\gamma$ are the weights of CPU, memory, and disk I/O resource requests in the generated load respectively.

The main objective of load balancing is evenly distributing tasks among virtual machines within the system. And ensure that the load borne by each virtual machine on the cloud platform is relatively balanced while ensuring the successful execution of tasks, thereby enhancing the overall performance of the system. Consequently, the real-time load of the $j$-th virtual machine at the moment $t$ and the corresponding real-time performance platform average load and load variance are

(10) $$\begin{array}{c}loa{d}_j^t=sum\left\{loa{d}_i|T_i\in r_j^t\right\}\end{array}$$

(10a) $$\begin{array}{c}subjectto{T}_i^{res}\le T_i^D,T_i^a<t<T_i^E\end{array}$$

(11) $$\begin{array}{c}loa{d}_{avg}^t={\displaystyle \frac{1}{m}\sum _{j=0}^{m}loa{d}_j^t}\end{array}$$

(12) $$\begin{array}{c}loa{d}_{var}^t={\displaystyle \frac{1}{m}\sum _{j=0}^m{\left(loa{d}_j^t-loa{d}_{avg}^t\right)}^2}\end{array}$$where $r_j^t$ is the set of tasks running on the $j$-th virtual machine at time $t$, $\mathrm{t}\in \left[T_1^a,\cdots ,T_n^a\right]$.

SRP-DRL task scheduling framework

The scheduling framework of SRP-DRL is illustrated in Fig. 1. The scheduling framework is composed of four main modules: (a) the task queue, (b) the DRL model, (c) the real-time performance platform model, and (d) the real-time load model. The corresponding workflow of the scheduling model is as follows. First, the user sends requests to the task queue of the scheduling model. Second, the DRL model takes the task information from the task queue as input. Third, the DRL model obtains real-time performance platform server status information. Fourth, DRL makes scheduling decisions built on the status information of the real-time performance platform and assigns a virtual machine to the task. Fifth, the task runs on the selected virtual machine and evaluates the task running performance parameters under the deadline (DDL) conditions. Simultaneously, the real-time load state of the virtual machine group before the real-time performance platform executes the task is fed into the load model. Finally, the DRL model obtains the reward value returned depending on a comprehensive evaluation of the task running performance parameters under the DDL conditions and the real-time load state, which guides the model’s training.

We choose the DRL model DQN to complete task scheduling, reduce task response time, and improve server load balancing capability while considering the server’s real-time performance and delay constraints. DQN is an effective algorithm in DRL. In utilizing deep neural networks, the method aims to approximate the optimal action-value function $Q^{\ast }$, aiming to obtain the optimal policy for maximizing cumulative rewards.

(13) $$\begin{array}{c}Q\left(s_t,a_t\right)\leftarrow Q\left(s_t,a_t\right)+\alpha \ast \left(r_t+\gamma \ast \underset{a\in A}{max}Q\left(s_{t+1},a\right)-Q\left(s_t,a_t\right)\right)\end{array}$$where $Q\left(s_t,a_t\right)$ represents the $Q$-value of executing action $a_t$ under state $s_t$, $\alpha$ is the learning rate, $r_t$ is the immediate reward obtained by the agent executing action $a_t$, $\gamma$ is the discount factor, $s_{t+1}$ is the newly observed environmental state after executing action $a_t$, $A$ denotes the action space of the DRL model, and $\underset{a\in A}{max}Q\left(s_{t+1},a\right)$ corresponds to the action value of the action that maximizes the $Q$-value in state $s_{t+1}$. Algorithm 1 describes the pseudocode for task scheduling based on DQN.

DRL model

Detailed design of SRP-DRL

In implementing cloud task scheduling using DRL algorithms, the design of the state space, action space, and reward function is crucial, as these factors will directly impact the performance of the algorithmic model. The SRP-DRL method model is elaborated upon in detail, and its model diagram is depicted in Fig. 2. Furthermore, the algorithm pseudo-code is presented for SRP-DRL to obtain task execution performance parameters based on real-time load and performance.

State space. The state space involved in the reinforcement learning model is crucial for the performance and adaptability of the algorithm. The proposed SRP-DRL algorithm considers the effect of server load on performance and introduces a real-time performance-aware strategy to enhance the model’s perception of environmental states. In the SRP-DRL algorithm at time $t$, the model’s state space $S$ is defined as $s_t=\left\{V_1^t,V_2^t,\cdots ,V_m^t,loa{d}_1^t,loa{d}_2^t,\cdots ,loa{d}_m^t\right\}$.

Action Space. The action space of a model is the aggregation of virtual machines from which tasks can be selected, specifically expressed as $A=\left\{V{M}_1,V{M}_2,\cdots ,V{M}_m\right\}$. The action decision output of the model is a one-hot encoded value in the action space, that is, one for the virtual machine selected by the model for the task and 0 for the rest. For example, when the $m$-th virtual machine is selected, then $a=\left\{0,0,\cdots ,1\right\}$.

Reward Function. To guide the reinforcement learning agent toward optimizing load balancing, the reward function is defined as

(15) $$R=\{\begin{array}{l}-1,\text{\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}Violation\hspace{0.17em}of\hspace{0.17em}DDL\hspace{0.17em}}\\ \frac{0.1}{T^{res}},\text{\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}Not\hspace{0.17em}violating\hspace{0.17em}DDL\hspace{0.17em}and\hspace{0.17em}}loa{d}_j^t-loa{d}_{avg}^t\\ \frac{\left|loa{d}_j^t-loa{d}_{avg}^t\right|}{T^{res}},\text{\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}Not\hspace{0.17em}violating\hspace{0.17em}DDL\hspace{0.17em}and\hspace{0.17em}}loa{d}_j^t\le loa{d}_{avg}^t\end{array}$$where $T^{res}$ represents the response time of task execution, $loa{d}_j^t$ is the real-time load of the $j$-th virtual machine selected by the task at moment $t$ before executing the current task, and $loa{d}_{avg}^t$ is the average load of the real-time performance platform at the moment t when the current task is not executed.

In SRP-DRL, the real-time load and performance of the server dynamically change every time, which means that the operating performance parameters such as response time, load, etc., for the same task under different server loads will also be different. Hence, it is necessary to obtain the running performance parameters of the task depending on the real-time load and performance state of the server where the task is selected for execution to instruct the model training. Algorithm 2 describes the pseudocode of the SRP-DRL method to obtain task running performance parameters.

Algorithm 2:

PerformanceParametersGeneration.

Input:  $T_i$, $a_t$, historical task performance parameters, and historical task run queue

Output: Task running performance parameters

1    Compute the $loa{d}_i$ occupied by $T_i$ execution;

2    // $\theta$ is load threshold

3    Gain real-time load of virtual machine $loa{d}_j^t$ based on $a_t$;

4    if first task OR (fulfill $req{C}_i,req{M}_i,reqI{O}_i$ AND ( $loa{d}_j^t$ + $loa{d}_i$) $\le \theta$):

5        Obtain task running performance parameters;

6    else:

7        Get the task running queue of the virtual machine;

8        Arrange the task running queue in ascending order according to task end time;

9        Calculate task execution time $T_i^e$;

10       // Traverse the task running queue

11       While unfulfilled $req{C}_i,req{M}_i,reqI{O}_i$ OR ( $loa{d}_j^t$ + $loa{d}_i$) > $\theta$:

12           Update ${load}_j^t$, remaining resources and $T_i^e$;

13       Calculate task running performance parameters;

DOI: 10.7717/peerj-cs.2120/table-4

Parameter settings and data generation

In our experiments, the weighted resource requests of tasks is considered as the load generated when tasks run on virtual machines. In the literature (Cheng et al., 2022), the authors chose to use a two-layer neural network model to approximate the optimal action-value function based on simplicity and computational efficiency, and the results show that this choice can improve scheduling efficiency. To make full use of the advantages of this model, it is introduced into our research, and at the same time, the overall load balancing performance can be improved after making necessary hyperparameter adjustments. The experience replay buffer size is 800, the batch size for mini-batch stochastic gradient descent is 60, the learning rate is 0.001, the experience replay learning threshold is 500, and the target network parameter update frequency is 50. For the cloud platform, the virtual machine load threshold $\theta$ is 0.95, the initial speed of virtual machines is 500. Regarding the task execution process, the most crucial thing is the deadline of the task, which in the experiments was 0.5, dictated by the average length of tasks and the original processing speed of virtual machines.

The relevant attributes of the randomly generated tasks in the experiment are all subject to specific probability distributions, as shown in Table 2. The task length $l_i$ obeys a normal distribution with an expectation of 200 and a variance of ${\sigma }^2$; the task arrival time $T_i^a$ obeys a Poisson distribution with a parameter of $\lambda$, $\lambda$ is expressed as the task arrival rate in the text; the resource requests of the task $req{C}_i,req{M}_i,reqI{O}_i$ all obey the uniform distribution with a lower limit of 0 and an upper limit of the load threshold $\theta$.

Evaluation indicators

In the experiment, we evaluate the performance of the SRP-DRL scheduling method from task average response time, task success rate, and server average load variance.

Task average response time. Response time is an essential indicator for measuring load balancing. It can evaluate the server running status and performance of the real-time performance platform. It is defined as follows.

(16) $$\begin{array}{c}{T}_{avg}^{res}=\sum _{i=0}^n{T}_i^{res}\end{array}$$

Task success rate. Whether the task responds successfully depends on whether the remaining resources and load status of the current server are sufficient to meet the task’s resource requirements and whether the task’s response time can meet the task’s DDL constraints, that is, $T^{res}\le T^D$. The task success rate can be expressed as

(17) $$\begin{array}{c}TSR={\displaystyle \frac{tas{k}_{suc}}{n}}\end{array}$$where, $tas{k}_{suc}$ is the number of successfully executed tasks, and $n$ is the total number of task requests in the task queue.

Server average load variance. Load variance can detect the fluctuation degree of server load and intuitively reflect the load balancing effect of the real-time performance platform. Under the condition of ensuring a high task success rate, the smaller the load variance, the more balanced the server load. On the contrary, it means the load is more unbalanced, which is defined as

(18) $$\begin{array}{c}avg\mathrm{\_}loa{d}_{var}={\displaystyle \frac{1}{n}\sum _{i=0}^{n}loa{d}_{var}^t}\end{array}$$

Baseline algorithm

In this article, the load-balancing performance of SRP-DRL is evaluated by comparing it to four baseline algorithms.

Random. The task randomly selects a virtual machine for execution. The algorithm is simple and fast, but task execution performance cannot be guaranteed.

Round-Robin. Tasks are assigned to virtual machines in turn in order. The algorithm allocates resources fairly but may result in poor resource utilization.

EITF. The task selects the earliest idle virtual machine to execute the task. The algorithm reduces waiting times but may result in less efficient use of idle resources.

BEST-FIT. The task dynamically selects the most appropriate virtual machine for execution based on task requirements and virtual machine status. The algorithm can effectively utilize resources, but the implementation is more complex and may increase system overhead.

Comparison results and analysis

Different number of tasks. At this experiment, we compare the load-balancing performance of five scheduling methods with varying numbers of tasks. The number of tasks is increased from 8,000 to 24,000 with an interval of 4,000. The task arrival rate is 20, the number of virtual machines is 10, and the standard deviation $\sigma$ of the task length is 20. The results of the experiment are shown in Fig. 3.

Concretely, Fig. 3A illustrates the task response time, Fig. 3B presents the task success rate, and Fig. 3C shows the average load variance. In Figs. 3A–3C, as the number of tasks increases, the average task response time, task success rate, and average virtual machine load variance do not change significantly. Experiments show that the number of tasks has little impact on the load-balancing performance of the current five task scheduling variances. As the experiment results are shown in Fig. 3, SRP-DRL is significantly better than other task scheduling methods in average response time and success rate. However, the average load variance of the EITF algorithm is lower than that of SRP-DRL. This is because the success rate of the EITF algorithm is very low, resulting in slight load fluctuation on the virtual machine during the task scheduling process, ultimately leading to a lower average load variance for the EITF algorithm. Hence, the comprehensive load balancing performance of SRP-DRL is better than that of other task scheduling methods.

Different task arrival rates. For this experiment, the load-balancing performance of five scheduling methods was compared under diverse task arrival rates. The task arrival rate ranged from 20 to 40 with intervals of 5, the number of virtual machines was 10, the number of tasks was 8,000, and the standard deviation of task length $\sigma$ was 20. The findings of the present experiment are shown in Fig. 4.

The quantity of tasks arriving within a specific time frame is determined by the task arrival rate. The higher the task arrival rate, the more tasks come in a unit of time. In particular, Fig. 4A indicates task response time, Fig. 4B shows task success rate, and Fig. 4C represents average load variance. As shown in Figs. 4A–4C, with the increase in task arrival rate, the average task response time slowly increases, the task success rate is gradually decreasing, and the average load variance of the virtual machine is gradually decreasing except for Random and EITF. This is because with more tasks arriving in a unit of time and the limited processing capacity of virtual machines, the waiting time for tasks increases, affecting task response time and ultimately leading to task failure beyond the deadline. Increasing the number of tasks arriving in a unit of time allows for a more effective distribution of tasks among virtual machines, avoiding overloading or underloading certain virtual machines, thereby affecting the average load variance of the entire virtual machine group. Specifically, the average load variance of the SRP-DRL algorithm is lower than that of the EITF algorithm at a task arrival rate of 40. As shown in the experimental results in Fig. 4, the comprehensive load balancing performance of SRP-DRL compares favorably with other task scheduling methods in the three indicators of average response time, success rate, and average load variance.

Different number of virtual machines. In this experiment, the study compared the load-balancing performance of five scheduling methods with other numbers of virtual machines. The number of virtual machines increased from 10 to 18 with an interval of 2, the task arrival rate was 40, the number of tasks was 8,000, and the task length standard deviation $\sigma$ was 20. The performance of the experimental results is shown in Fig. 5.

For details, Fig. 5A presents task response time, Fig. 5B illustrates task success rate, and Fig. 5C indicates the average load variance. From Figs. 5A–5C, with the increase in the number of virtual machines, the average task response time decays gradually, the task success rate increases. In contrast, the average load variance of virtual machines increases slowly, except for the fluctuation of Random and EITF. For this reason, as virtual machines increase, the system’s ability to process tasks increases, reducing task wait times and affecting average task response time and success rate. However, maintaining load balance between virtual machines becomes increasingly challenging, leading to fluctuations in the average load variance of the virtual machine group. It is shown in Fig. 5 of the experimental results that SRP-DRL also outperforms other task scheduling methods in terms of combined load balancing performance regarding average response time, success rate, and average load variance.

Different task length deviations. Within this experiment, a comparative study on the load-balancing performance of five scheduling methods was conducted for various task length standard deviations. The task length standard deviation $\sigma$ ranged from 10 to 50 with intervals of 10, the task arrival rate was 30, the number of tasks was 8,000, and the number of virtual machines was 10. The outcome of the experiment is shown in Fig. 6.

Specifically, Fig. 6A indicates task response time, Fig. 6B shows task success rate, and Fig. 6C illustrates the variance in average load. In Figs. 6A–6C, with the increase in task length standard deviation, the average load variance of virtual machines gradually increases except for Random. The cause of this is task lengths become more diverse with an increasing standard deviation of task lengths. The load generated by task execution fluctuates wildly in resource occupancy time (that is, task execution time), making it increasingly difficult to maintain load balancing among virtual machines. This is shown in Fig. 6 of the experimental results, which shows that the integrated load balancing performance of SRP-DRL was similarly superior to other task scheduling methods in all three metrics of average response time, success rate, and average load variance. Indeed, the performance of EITF on the average load variance metric is consistent with the results in Fig. 3C.