Sim-DRS: a similarity-based dynamic resource scheduling algorithm for microservice-based web systems

Rule-based approach

Yan, Chen & Shuo (2017) proposed an elastically scalable strategy based on container resource prediction and message queue mapping to reduce the delay of service provisioning. Leitner, Cito & Stöckli (2016) proposed a graph-based model for the deployment cost of microservices, which can be used to model the total deployment cost depending on the call patterns between microservices. Magalhaes, Rech & Moraes (2017) proposed a scheduling architecture consisting of a Web server powered by a soft real-time scheduling engine. Gabbrielli et al. (2016) proposed JRO (Jolie Redeployment Optimiser) tool to generate a suggested SOA (service-oriented architecture) configuration from a partial and abstract description of a target application. Filip et al. (2018) proposed a mathematical formulation for describing an architecture that includes heterogeneous machines to handle different microservices. Zheng et al. (2019) presented SmartVM, a business Service-Level-Agreement (SLA)-aware, microservice-centric deployment framework to handle traffic spikes in a cost-efficient manner. Fard, Prodan & Wolf (2020) proposed a general microservice scheduling mechanism and modeled the scheduling problem as a complex variant of the knapsack problem, which can be expanded for various resource requests in queues and solved by multi-objective optimization methods. Mirhosseini et al. (2020) developed a scheduling framework called Q-Zilla from the perspective of solving the end-to-end queue delay, and the SQD-SITA scheduling algorithm was proposed to minimize the delay caused by microservice distribution.

Rule-based approaches are straightforward and are efficient in simple environments. The scheduling problem can be solved by constructing rules through domain knowledge, using software architecture and simple data modeling theory, which is effective in an environment that meets certain assumptions. However, they rely heavily on prior domain knowledge, have a low degree of mathematical abstraction, and may be labor-intensive, imprecise, and have poor results in high variability scenarios.

Heuristic approach

Li et al. (2018) proposed a prediction model for microservice relevance using optimized artificial bee colony algorithm (OABC). Their model takes into account the cluster load and service performance, and has a good convergence rate. Stévant, Pazat & Blanc (2018) used a particle swarm optimization to find the best placement based on the performance of microservices evaluated by the model on different devices to achieve the fastest response time. Guerrero, Lera & Juiz (2018b) presented an NSGA-II algorithm to reduce service cost, microservice repair time, and microservice network latency overhead. Adhikari & Srirama (2019) used an accelerated particle swarm optimization (APSO) technique to minimize the overall energy consumption and computational time of tasks with efficient resource utilization with minimum delay. Lin et al. (2019) proposed an ant colony algorithm that considers not only the utilization of computing and storage resources but also the number of microservice requests and the failure rate of physical nodes. Guerrero, Lera & Juiz (2018a) proposed an NSGA-II-based approach to optimize system provisioning, system performance, system failure, and network overhead simultaneously. Bhamare et al. (2017) presented a fair weighted affinity-based scheduling heuristic to reconsider link loads and network delays while minimizing the total turnaround time and the total traffic generated. Lin et al. (2019) used an ant colony algorithm to solve the scheduling problem. It considered not only the computing and storage resource utilization of physical nodes, but also the number of microservice requests and failure rates of the nodes, and combined multi-objective heuristic information to improve the probability of choosing optimal path.

These approaches generally abstract resources scheduling into an optimization problem through appropriate modeling methods and solve the problem in a certain neighborhood. And heuristic approaches have been proven to be efficient in finding good scheduling solutions in a high-dimensional space (Guerrero, Lera & Juiz, 2018a), especially under the circumstances of balancing many conflicting objectives. However, they suffer from low performance and search from a random state, which lead to noneffective use of a priori knowledge of existing good solutions.

Learning-based approach

Alipour & Liu (2017) presented a microservice architecture that adaptively monitors the workload of a microservice and schedules multiple machine learning models to learn the workload pattern online and predict the microservice’s workload classification at runtime. Nguyen & Nahrstedt (2017) proposed MONAD, a self-adaptive microservice-based infrastructure for heterogeneous scientific workflows. MONAD contains a feedback control-based resource adaptation approach to generate resource allocation decisions without any knowledge of workflow structures in advance. Gu et al. (2021) proposed a dynamic adaptive learning scheduling algorithm to intelligently sorts, allocates, monitors, and adjusts microservice instances online. Yan et al. (2021) used the neural network and attention mechanism in deep learning to optimize the passive elastic scaling mechanism of the cloud platform and the active elastic mechanism of microservices by accurately predicting the load of microservices, and finally realized the automatic scheduling of working nodes. Lv, Wei & Yu (2019) used machine learning methods in resource scheduling in microservice architecture, pre-trained a random forest regression model to predict the requirements for the microservice in the next time window based on the current unload pressure, and the number of instances and their locations were adjusted to balance the system pressure.

The learning-based solutions are still in their infancy. The main advantage of these approaches is that they can generate scheduling decisions adaptively and automatically, without any human intervention. However, these approaches require a considerable number of samples to build a reasonable decision model for a microservice system (Alipour & Liu, 2017). The high demand of samples is always challenging because only a very limited set of samples can be acquired during a short time period for resource scheduling in production systems.

System components

Specifically, DRS-MWS has the following components.

Optimization objectives

At any time point t, we wish to optimize three objectives: (i) the resource consumption (C^(t)) for supporting users’ requests; (ii) the system jitter (J^(t)) due to the deploying adjustment of the microservice instances; and (iii) the invocation expense (E^(t)) for calling different microservice instances along the microservice invocation chain of an application. It is worth mentioning that we treat a microservices as a black-box function, and the latency of queueing and executing requests within the microservice is not considered in this paper.

To achieve this optimization, we define our resource scheduling first by managing the number of instances for each microservice and then deciding how to deploy each microservice instance to an appropriate VM. More specifically, suppose that an MWS has n VMs and m different microservices, and a workload $W^{(t)}=\{w_1^{(t)},w_2^{(t)},\cdots ,w_m^{(t)}\}$ is generated at a time point t. We define a matrix S_m×n to denote our scheduling decision, where each element s_ij ∈ S indicates the number of instances for microservice ms_i that will be deployed to VM v_j, and for example, we have three VMs (v₁, v₂, and v₃) and three microservices (ms₁, ms₂, and ms₃) in Fig. 1, and the current resource scheduling at time point t can be defined as:

(2) $$S^{(t)}=\left[\begin{array}{ccc}1& 1& 0\\ 1& 0& 1\\ 0& 1& 1\end{array}\right].$$

Resource consumption, denoted by C^(t), is measured as the sum of the resource demands for deploying each microservice instance on its target VM:

(3) $$C^{(t)}=\sum _{i=1}^m\sum _{j=1}^n(s_{i,j}^{(t)}\cdot m{s}_i.r),$$where $s_{i,j}^{(t)}$ denotes the number of microservice instances ms_i deployed on VM v_j, and ms_i.r represents the normalized resource demand for deploying ms_i on a VM.

System jitter, denoted by J^(t), is an important performance metric used to measure the robustness of the system at time point t, which can be further defined as the change degree of microservice instances’ deployment for an MWS environment. For example, given any two continuous time points t and t − 1, J^(t) results from subtracting scheduling decisions S^(t) and S^{(t − 1)}:

(4) $$\begin{array}{c}{J}^{(t)}=\sum _{i=1}^m\sum _{j=1}^n|s_{i,j}^{(t)}-s_{i,j}^{(t-1)}|\end{array}.$$

Invocation expense, denoted by E^(t), is defined as the associated cost for considering both the microservice invocation chains of applications and the latency overhead among different cloud providers. Because the latency has been defined in the previous section, we need to define the former as microservice invocation expense.

Given a set of applications A = {A₁, A₂, ⋯} in an MWS, we define the correlation, denoted as cor(ms_i,ms_j), between any two microservices ms_i and ms_j as:

(5) $$cor(m{s}_i,m{s}_j)=\{\begin{array}{cc}1\hfill & \hfill \mathrm{i}\mathrm{f}(m{s}_i,m{s}_j)\in E_{A_k},\mathrm{\forall }{A}_k\in A\hfill \\ 0\hfill & \hfill \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array},$$where EA_k represents the microservices required to implement A_k.

The microservice invocation distance, denoted by a matrix D_m×m, is thus defined to indicate the alienation between any two microservices, and each element d_i,j in D is defined as:

(6) $$d_{i,j}=\{\begin{array}{cc}{e}^{-cor(m{s}_i,m{s}_j)}\hfill & \mathrm{i}\mathrm{f}i\ne j\\ 0\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}.\end{array}$$

As shown in Fig. 1, there is only one application A = {ms₁, ms₂, ms₃} that belongs to the MWS, and the microservice distance D is thus represented as:

(7) $$D=\left[\begin{array}{ccc}0& e^{-1}& 1\\ 1& 0& e^{-1}\\ 1& 1& 0\end{array}\right].$$

Based on the latency L_n×n among different VMs, the scheduling decision S^(t)_m×n at time point t, and the microservice distance D_m×m, we now define the invocation expense E^(t) as a scalar value:

(8) $$\begin{array}{c}{E}^{(t)}=U\cdot (D\cdot S^{(t)}\cdot L)\cdot U^T\hfill \end{array},$$where U_1×n is an auxiliary matrix whose elements are all equal to 1. For example, as shown in Fig. 1, given L defined in Eq. (1), S^(t) defined in Eq. (2), and D defined in Eq. (7), we have:

(9) $$\begin{array}{c}{E}^{(t)}=\left[\begin{array}{ccc}1& 1& 1\end{array}\right]\left(\left[\begin{array}{ccc}0& e^{-1}& 1\\ 1& 0& e^{-1}\\ 1& 1& 0\end{array}\right]\left[\begin{array}{ccc}1& 1& 0\\ 0& 1& 1\\ 0& 1& 1\end{array}\right]\left[\begin{array}{ccc}0& 50& 50\\ 50& 0& 20\\ 50& 20& 0\end{array}\right]\right)\left[\begin{array}{c}1\\ 1\\ 1\end{array}\right]=280e^{-1}+520\end{array}$$

Problem formulation

We formally define DRS-MWS as a three-objective optimization problem:

(10) $$\{\begin{array}{c}\underset{\mathrm{\forall }{S}^{(t)}}{min}{C}^{(t)}(A,MS,V,S^{(t)})\hfill \\ \underset{\mathrm{\forall }{S}^{(t)}}{min}{J}^{(t)}(A,MS,V,S^{(t)})\hfill \\ \underset{\mathrm{\forall }{S}^{(t)}}{min}{E}^{(t)}(A,MS,V,S^{(t)})\hfill \end{array}$$

(11) $$s.t.\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{e}\mathrm{d}\mathrm{u}\mathrm{l}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\le \gamma \cdot \mathrm{\Delta }t$$

(12) $$\begin{array}{ccc}\hfill & \sum _{j=1}^m(s_{i,j}^{(t)}\cdot m{s}_j.r)\le \delta \cdot v_i.r\hfill & \mathrm{\forall }t,i=1,2,\cdots ,n\hfill \end{array}$$

(13) $$\begin{array}{ccc}\hfill & \sum _{j=1}^n{s}_{i,j}^{(t)}={\lceil {\displaystyle \frac{w_i}{m{s}_i.g}\rceil }}^{(t)}\hfill & \mathrm{\forall }t,i=1,2,\cdots ,m\hfill \end{array}$$where Eq. (10) states that at any time point t, given an MWS consisting of a set of applications (A), a set of microservices (MS), and a set of VMs (V), the goal of DRS-MWS is to find a resource scheduling policy S among all valid policies under workload W^(t) to minimize the resource consumption (C^(t)), system jitter (J^(t)), and invocation expense (E^(t)) simultaneously. The constraint Eq. (11) is that any solution to the problem must terminate after a γ · Δt amount of time. The constraint Eq. (12) states that the resource consumption of any VM for deploying microservices must not exceed a certain proportion of its total resource capacity. Finally, the constraint Eq. (13) states that the workload on each microservice needs to be served appropriately.

Motivation

Evolutionary algorithms (EA) are among the most popular for solving the DRS-MWS problem (Guerrero, Lera & Juiz, 2018a). EA implementation requires a definition of the solution and the construction of several technical components including initial population, genetic operators such as crossover and mutation, fitness function, selection operator, offspring generation, and execution parameterization (Mitchell, 1998). Most of the previous approaches assume that no a prior information about the solution is available, and often use a random initialization method to generate candidate solutions (i.e., the initial population). According to (Rahnamayan, Tizhoosh & Salama, 2007), population initialization is a crucial step in evolutionary algorithms because it can affect the convergence speed and also the quality of the final solution. Due to the dynamics and the hard time constraint in the DRS-MWS problem, previous studies have shown that adopting a random initial population often leads to non-convergence in the optimization process and eventually a low-quality solution (Wang et al., 2009).

Instead of using a complete randomization technique for initial population generation, we attempt to leverage existing solutions to similar problems to compose the initial population. The rationale behind this idea is that good solutions to similar problems may share some common structures. Figure 2 illustrates the optimization process of Sim-DRS, which contains two phases: offline training and online scheduling. In the training phase, it first generates a set of synthetic workloads, and then applies a Non-dominated Sorting Genetic Algorithm II (NSGA-II) to reach the pareto optimality for three optimization objectives mentioned in Eq. (10). It constructs a database of good solutions by adding such one-to-one <workload-solution> pairs. In the scheduling phase, given a real workload W^(t) generated by users at each time point t, Sim-DRS first applies a similarity-based algorithm to generate an initial population mixed with random and previously-known similar solutions from the database by calculating the similarities between W^(t) and existing synthetic workloads. It then uses a standard NSGA-II algorithm to optimize three objectives defined in Eq. (10) and select the best one.

Solution database construction

To measure the correlation between any two workloads $W_i=\{w_1^i,w_2^i,\cdots ,w_m^i\}$ and $W_j=\{w_1^j,w_2^j,\cdots ,w_m^j\}$, we first define workload similarity (or similarity in short):

(14) $$\mathrm{\Gamma }(W_i,W_j)=e^{-\sqrt{\sum _{k=1}^m{(w_k^i-w_k^j)}^2}}.$$

As shown is Fig. 2, we need to construct a solution database containing previously-known good solutions for different workloads in the training phase. The solution database construction (SDC) algorithm is provided in Algorithm 1.

As shown in Algorithm 1, for every workload W_i in the synthetic workload set W, we randomly generate p different solutions as the initial population (line 3), and then apply a Non-dominated Sorting Genetic Algorithm II (NSGA-II) to find the optimal solution (line 4). The <workload-solution> pair is added to the database as a known fact (line 5). Note that the training phase is conducted offline, which allows an extensive execution of the NSGA-II algorithm without any time constraint. Based on the definition of similarity, we then apply a k-Means clustering algorithm to these pairs by clustering the workloads and generate k different groups, where k, which is often designated based on an empirical study, is used to characterize different workload patterns (line 7).

Figure 3 illustrates an example of the solution database containing three groups of workloads G₁, G₂, and G₃, each of which consists of three workloads. Three centers, namely W₃, W₅, and W₇ in this example, represent their clusters, respectively.

Similarity-oriented initial population generation

Given a workload W^(t) generated by users at each time point t during the dynamic scheduling phase, the similarity-based initial population generation algorithm in our Sim-DRS approach aims to find good solutions from the solution database according to the distance measurement, as shown in Algorithm 2.

The similarity-based initial population generation algorithm begins with searching for the group G* in the solution database that has the largest similarity with W^(t) (line 2), where $W_G^c$ represents the cluster center of any workload group G. For example, given a workload W^(t) in Fig. 3, we need to calculate γ(W^(t),W₃) for G₁, γ(W^(t),W₅) for G₂, and γ(W^(t),W₇) for G₃, respectively.

Given G *, we construct a roulette wheel selection (RWS) process, a proportional selection strategy which has a similar selection principle as roulette wheel, to construct a population with good solutions (line 3). More specifically, suppose that group G * contains k different <workload-solution> pairs G* = {a₁,a₂,⋯,a_k}, and an individual pair a_i = <W_i,S_i> has the distance value of Γ(W^(t),W_i). Then, the probability for a_i to be selected is:

(15) $$\begin{array}{c}ps(a_i)={\displaystyle \frac{\mathrm{\Gamma }(W^{(t)},W_i)}{\sum _{j=1}^k\mathrm{\Gamma }(W^{(t)},W_j)}}i=1,2,\cdots ,k.\hfill \end{array}$$

After obtaining the roulette wheel: PS = {ps(a₁),ps(a₂), ⋯,ps(a_k)}, we repeatedly select candidate workload-solution pairs p times from G* using the RWS strategy with PS (line 5).

Once a solution S_i to the workload W_i is selected, we need to adjust it to the current workload W^(t) if W^(t) ≠ W_i (lines 6–19). More specifically, for each microservice ms_j, we calculate the difference diff between the workload w^t_j for W^(t) and wⁱ_j for W_i (line 7), and then convert the workload difference (diff) into the microservice instance difference (adj) (line 8). To adjust the scheduling decision sⁱ_j,* for ms_j on every VM, we randomly choose a VM indexed by x (x∈ [1,n]) and add/remove a microservice instance to/from v_x (lines 12–16). This process repeats multiple times until |adj| times of adjustments have been performed successfully (lines 11–17). After each adjustment on S_i, we add it to the set of good solutions GS for the initial population (line 20). The algorithm stops after p times of selections and adjustments, and finally returns the set of good solutions GS.

Sim-DRS algorithm

The pseudocode of Sim-DRS is provided in Algorithm 3. Sim-DRS initially generates h_s (α · populationSize) number of good solutions using our similarity-based initial population generation algorithm (line 2), and generates h_r ((1 − α) · populationSize) number of solutions using the standard random algorithm (line 3). Finally, h_s and h_r are merged together to form the initial population (line 4).

Our Sim-DRS approach is based on the Non-dominated Sorting Genetic Algorithm-II (NSGA-II) (Deb et al., 2002) (lines 5–25) as introduced in the previous section. The crossover operation randomly exchanges the same number of rows of two individuals to produce new ones (line 12). To avoid the local minimum value and cover a larger solution space, a mutation operator is also used (line 14). Note that in order to satisfy the constraint stated in Eq. (11), we need to randomly adjust the values of an individual after applying the crossover and mutation operations (line 16–17).

According to the problem formulation, Sim-DRS considers three objective functions, namely, C^(t), J^(t), and E^(t), as the fitness functions to measure the quality of a solution (lines 5 and 22). It sorts the solutions at Pareto optimal front levels, and all the solutions in the same front level are ordered by the crowding distance (lines 22–25). Once all the solutions are sorted, a binary tournament selection operator is applied over the sorted elements (line 11): two solutions are selected randomly, and the first one on the ordered list is finally selected (line 27).

Experimental setup

Benchmark

We choose TeaStore (von Kistowski et al., 2018) as our benchmark application to evaluate the performance of different algorithms including Sim-DRS. TeaStore is a state-of-the-art microservice-based test and reference application, and has been widely used for performance evaluation of microservice-based applications. It allows evaluating performance modeling and resource management techniques, and also offers instrumented variants to enable extensive run-time analysis.

As shown in Fig. 4, the TeaStore consists of five distinct services and a registry service. All services communicate with the registry for service discovery and load balancing. Additionally, the WebUI service issues calls to the image provider, authentication, persistence and recommender services. The image provider and recommender are both dependent on the persistence service. All services communicate via representational state transfer (RESTful) calls, and are deployed as Web services on the Apache Tomcat Web server.

Experimental results

Given fixed time constraints, we run four different scheduling algorithms independently. Table 3 tabulates the objective values when the workload subjects to the Poisson distribution. As expected, Sim-DRS has better performance than the three comparison algorithms in terms of three objectives. Specifically, in terms of resource consumption, our algorithm achieves an average performance improvement of 10.91% over ACO, 11.40% over PSO, and 10.55% over NSGA-II; in terms of system jitter, our algorithm achieves an average performance improvement of 42.77% over ACO, 41.53% over PSO, and 38.12% over NSGA-II; in terms of invocation expense, our algorithm achieves an average performance improvement of 9.70% over ACO, 10.58% over PSO, and 10.80% over NSGA-II. It is worth noting that Sim-DRS shows significant improvements over the other algorithms for the system jitter objective, which indicates that the solutions generated by our algorithm are more stable with a higher level of robustness of the MWS. Another important observation from Table 3 is that Sim-DRS achieves more significant improvements over the other algorithms when the time constraint is stricter (i.e., tighter scheduling time). This is consistent with our similarity assumption stated in the Motivation section.

For a better illustration, we plot the performance measurements of ACO, PSO, NSGA-II, and Sim-DRS in terms of three objectives when the workload subjects to the random distribution in Figs. 5A–5C, respectively. In each figure, the x-axis lists the scheduling time constraint and the y-axis represents the measurements of the three objectives. We can also conclude from Fig. 5 that Sim-DRS achieves stable improvements compared with the other three algorithms.

In summary, we can safely draw the conclusion from the experimental results that our algorithm outperforms all other algorithms in terms of three objectives, namely resource consumption, system jitter, and invocation expense, in both workloads that follow Poisson and random distribution. Note that the improvement is much significant in terms of system jitter, which means that our algorithm is more practical in the production WMS. This is because our algorithm requires less deployment efforts and is able to make the system more stable.

Figure 6 illustrates the objective measurements of different algorithms with the time constraint of 2 s for 20 scheduling decisions. We observe from Fig. 6 that Sim-DRS outperforms all other three algorithms under every scheduling decision point, followed by ACO, PSO, and NSGA-II. The difference between these algorithms is not significant at the beginning and end of the decision time points, but the difference at the middle points is. Since the workloads follow the Poisson distribution, such results indicate that Sim-DRS is more stable and robust under different circumstances in comparison with other algorithms.

Finally, Fig. 7 shows the response time of user requests for different algorithms with the workload following Poisson distribution. We observe from Fig. 7 that the response time measurements also follow Poisson distribution for all four algorithms, which is reasonable because there should be a positive correlation between the response time and the number of user requests. We also observed that our algorithm is better at responding to requests when the workload is heavier, which means that Sim-DRS can obtain a better performance in terms of three objectives while still ensuring a good response time.