Deep gradient reinforcement learning for music improvisation in cloud computing framework

Dataset collection

An essential part of the learning process is the data that will be entered into the model. When these data are improperly selected, a model may yield unfavourable outcomes. Even the choosing process needs to be done with caution. The chosen data should be pertinent to the scenario and problem description. The first stage is choosing songs that are comparable to each other, which means they employ the same instruments and are in the same genre. The outcome of selecting tracks that are not related is unappealing music. Occasionally, the datasets that are gathered may come from different sources. It is important to carefully alter them into a comparable format with the same quantity, kind, values, and names of characteristics. This proposed approach focuses on the classical music genre, which is composed by a variety of performers. These are gathered from several public datasets and should be transformed into the necessary MIDI format.

The harmonised chorales of J.S. Bach that make up the Bach Chorales collection are renowned for their melodic coherence and structural consistency. This homogeneity is essential because it offers a reliable foundation for training machine learning models. Bach is well known for his compositions’ complex harmonic progressions and melodic lines. This intricacy provides an extensive dataset full of examples of complicated ideas in advanced music theory. Bach Chorales are typically offered in MIDI format, which is perfect for the needs of this project. MIDI files include comprehensive timing, duration, and pitch information for training music-generating models. This study intends to improve melodic framework identification and improvisation to improve music creation. Bach Chorales provide a great opportunity to investigate and play around with these frameworks because of their distinct and unambiguous melodic structures. This research uses a wealth of classical music compositions consistent in style and complexity and appropriate for deep learning applications in the music industry by choosing the Bach Chorales dataset.

Dataset 2: This analysis section uses the MAESTRO dataset (Hawthorne et al., 2018). It is a collection of 1,276 MIDI data recordings of virtuoso piano performances, totalling 7.04 million notes, made with Yamaha Disklaviers. In contrast to the MIDI data gathered for KernScores, this data features dynamic and expressive timing.

Distinctive music notes selection

Because the dataset is made up of different songs, there are more distinctive note combinations that must be predicted while creating music. Every note that needs to be anticipated is comparable to a class in a machine learning classification issue. To classify the inputs in a categorization problem, they are taken and modified through several hidden layers. Before training the model, the number of concealed layers, connections in every layer, and the layer resulting from training must all be determined. Because of this, it is crucial to understand how many nodes—equal to the number of classes—are present in the layer that produces the output.

Music sequence segmentation

While the songs in the dataset can differ in length, the model only accepts a limited number of inputs determined by its architecture. This could make using them as inputs problematic. Poor outcomes will arise from improper feeding of the datasets. To address this issue, the dataset is split into same-sized sequences, which are then used to train the model. Each of these sequences is fed into the model for a single instance. Subsequently, the model teaches itself to anticipate which note will come next in the input sequence. The segment size is a parameter that needs to be carefully set because it will determine the number of inputs in the model design.

Bi-directional gated recurrent units

Bi-directional gated recurrent unit (Bi-GRU) is implemented to detect the melodic frameworks in the proposed approach. This deep learning-based model is developed on the audio attributes to determine the chord progression of a submitted music segment. The bidirectional GRU model is trained using the characteristics extracted from the music files. GRU is a prevalent option for problems involving periodic and series-to-series categorisation. This unique type of RNN can take feature dependencies into account over an extended training period. This model includes an attention state that aids in retaining or losing details, such as one’s capacity to comprehend music.

The attention preserves significant aspects of the music that are necessary for later consideration. Because of this feature, Bi-GRU performs well when dealing with serialised data, such as audio signals. Dividing the concealed state into two halves is an essential part of Bi-GRU. The first portion is stored for a long time whereas the second component is reserved for only a brief period. Similar to how the human mind processes music, some information about the audio features is preferentially devoured; others are retained, while others are disregarded or lost.

One component of the model architecture is a progressive network. The input layer is connected to the Bi-GRU layer to lessen bias, which is subsequently attached to the dropout layer. It is possible to add more layers to improve the functionality and feature extraction of the Bi-GRU model. Including a few dropout layers in between is crucial to ensure the model can learn correctly. To gather all the data from all the nodes into a single layer that can eventually be connected to the output layer, the model’s last layer is compressed into one and linked to an intense layer. The output layer for the last calculations requires an activation function.

Activation functions are used at this process stage to determine a node’s output in neural networks given a certain set of inputs. The activation function of digital networks is achieved by using a chip circuit dependent on the binary input and using the neural networks of the linear perceptron. Predicting which note will occur next is likewise a multiclass classification because the number of notes in the songs is always high and the number of categories is also considered. Thus, in this specific model, the activation function is softmax. Another excellent option for the problem’s activation function is the tanh function.

The equal-length sequences obtained during the data preprocessing stage train the developed model. To anticipate the following note to be played, the entire dataset, separated into many sequences, is run through these sequences. The difference between the expected and real notes is computed to determine the loss. The various weights and biases in the model are now adjusted using the computed loss to make the subsequent prediction more accurate the next time. By doing this, the loss is reduced for the future.

Deep gradient-based reinforcement learning

This work uses deep gradient-based reinforcement learning for real-time music improvisation by optimising the rewards. A policy that improves the anticipated prospective total reward is learned through reinforcement learning. The prospective reward can be R_θS. The adjusted discount at any instant of time k is represented in Eq. (1), (1)${\displaystyle R_k=\sum _{a=0}^K{\mu }^a{r}_{a+k}.}$

In the above equation, r_k denotes the standards of the generated next note. µrepresents the rate of discount that is applied. The model continually learns the actions that must be performed and its corresponding value function. The action to be taken by the agent is represented as θ_αx(x_k|h_k) and the function is denoted as F_αx(h_k). Accordingly, the agent is made to proceed with the actions x_k for the different states h_k. The value of the discounts generated is computed using the Eq. (2): (2)${\displaystyle d^{\delta }=R\left[\sum ^{K_{k=0}}{\nabla }_{{\alpha }_x}\log {\theta }_{\alpha }\left(x_k|h_k\right)\sum ^{K_{n=0}}{\left(\delta \beta \right)}^f{\gamma }_{k+f}^F\right].}$

In Eq. (2), the value of ${\gamma }_k^F$ can be determined by using the representation in Eq. (3) which specifies the resultant of the time variation in the value function, (3)${\displaystyle {\gamma }_k^F=x_k+\delta F\left(h_{k+1}\right)-F\left(h_k\right).}$

The slope value of the reward function is computed using the formulation in Eq. (4), (4)${\displaystyle R_s=R\left[\sum _{k=0}^K{\nabla }_{{\alpha }_x}\left(H\left(\left|\right|F_{{\alpha }_x}\left(h_k\right)-R_k|{|}^2\right)\right)\right].}$

The training procedure of the DGBRL method alternates between the creation and reward stages. During the creation phase, manual and machine-generated equivalent tokens are successively produced by the creation agent in a given state, h_k. Following the musical improvisation, the reward agent calculates the reward for every move at every step. Subsequently, these rewards are utilised to compute R_k and are employed to update the reward function $F_{{\alpha }_x}\left(h_k\right)$ and the creation agent θ_αx(x_k|h_k).

Further, the model adapts as it processes and learns from a new token, either by absorbing the new data or creating a new category. Absolute mobility is the ability of the model to measure the precise change in the upcoming tokens. The DGBRL’s inherent reward R is calculated as the amount of the shift observed between the state before the new input data (h_k−1) and the modified state (h_k) using the representation in Eq. (5): (5)${\displaystyle R=\sqrt{{\left(h_k^x-h_{k-1}^x\right)}^2}.}$

To avoid the input values that contribute minimally towards the improvisation of the musical notes, a minimum limit is being set on the reward value based on the endurance e as shown in Eq. (6): (6)${\displaystyle R_l=\frac{e}{\left|R-e\right|+e}.}$

Consequently, input tokens that yield negligible changes are considered uninteresting and unrewarding. The more unpredictable and less comparable an input is to previously identified patterns, the less rewarding the difference is from it. Using the above equation, the DGBRL’s music improvisation phase can guarantee that the most satisfying decision is always taken by selecting the most rewarding following stimuli.

The problem of music improvisation can be defined in a finite amount of time, for a certain number of steps, evaluating all potential future inputs, provided that pitch region and rhythm are periodic. Music improvisation at every phase entails figuring out what would happen if you observed each option. When the model runs consistently, it generates the option that will yield the greatest reward, plays the corresponding pitch, and records the outcome. Theoretically, this leads to investigating all local maxima; however, as the points become less significant with every presentation, the agent is never trapped.

Generation of MIDI format

The music file’s output is extracted and must be converted to MIDI format. Since the production of music generation is not a value that can be compared and verified for correctness, it cannot be directly evaluated. It is necessary to hear the music to determine whether or not it is pleasing to the ear and whether it generates the required music in the appropriate genre with the proper instruments. This is accomplished by converting the output file to MIDI format using online converters or open-source programs. This output can now be transferred and listened to like any other music file and can be played with a music player.

Experimental setup

Owing to its ease of use, uniformity, device autonomy, and availability of excellent tools and software for deep learning (DL) and machine learning (ML), Python was chosen as the programming language for this research setup. Python 3.11.5 is the version employed. The backend framework used in this proposed system was TensorFlow 2.10.0. Pandas, sci-kit-learn, and Keras are the three main machine-learning libraries adopted.

The dataset is split into train, test, and validation sets using the standard train_test_split technique in the Scikit-learn module. Twenty percent of the train data went to the validation split, and twenty percent went to the test split, as is customary. After that, the BiGRU model was trained for 1,000 epochs, leveraging 75% dataset. The model was prepared using the Stochastic gradient descent optimiser with a learning rate of 0.001 and a loss function of categorical cross-entropy.

The scope of the chosen problem and model efficiency determines which loss function and optimiser to use in the model implementation. The problem in the current research is a multi-class classification problem, which is ideally suited for the use of categorical cross-entropy. It should be mentioned that the goals are also expressed as integers. Since the encoded data takes up significantly less space than one-hot encoding, the target is minimal. The stochastic gradient descent technique is more suitable based on the dynamic approximation of the initial and higher-order instances. It provides an optimisation approach that can handle scant slopes on chaotic issues by combining the best features of the RMSProp along with Adam algorithms.

Model implementation

Model deployment is one of the most important phases of a machine-learning application. The model must be highly efficient and extensible to support thousands of users. As a result, the cloud, along with containerization architecture, was used to deploy the proposed system. In any case, a significant portion of computing in recent years has occurred in the cloud. The deployment expands flexibly and enables the system to be used by a small number of or millions of users. In this research, this system only recognises the uploaded music’s melodic structure but improvises the raga’s vicinity and converts it to MIDI format.

The use of the cloud and containerisation services has significantly advanced over time into sophisticated DevOps technology, which handles the distribution of load to satisfy unpredictable demands from global clients. Thus, DevOps in a solid cloud infrastructure is employed to simultaneously cope with even millions of requests. Docker employed technology to deliver container software by utilising virtualisation at the Operating System level to achieve containerisation. With Docker, developers can quickly pack, deploy, and operate any programme as an independent, compact container that can function almost anytime. Deploying Docker containers in a cloud environment is also simple.

The orchestration of containers was done using Kubernetes. Kubernetes is a freely available container management technology that allows an extensible web hosting framework for cloud apps to run. It provides mobility and quicker, more manageable deployment timeframes. The prediction model’s containers were implemented on the Google Cloud platform’s Kubernetes group. Every node comprises several hosts, each of which may hold one or more containers.

Datasets

The dataset used in the implementation is the Bach Chorale dataset, a univariate sequential time series dataset with six features. It primarily comprises Single-line melodies of 100 Bach chorales. The six different features of the dataset are start-time, pitch, duration, key signature, time signature and fermata. Each chorale consists of four sections that are monolingual and univocal. The dataset utilised in the implementation can be accessed using the following link: https://archive.ics.uci.edu/dataset/25/bach+chorales.

Experimental results

Initially, experiments were conducted to detect the melodic framework in the input audio songs, which were further used for music improvisation using deep gradient-based reinforcement learning. Bi-GRU was employed to identify the raga in the songs. The audio features were extracted, and the number of raga classes in specific notes was also analysed. To conclude the performance supremacy of Bi-GRU, a few existing methods were also considered for performance comparison. The conventional deep learning methods such as RNN, long short-term memory networks (LSTM), bi-directional long short term memory networks (Bi-LSTM) and transfer learning techniques such as ResNet and DenseNet models were utilised for analysing the efficiency of Bi-GRU in detecting the melodic frameworks. Table 1 shows the accuracy exhibited by each model in detecting the raga in the audio input.

It takes significant computational power to train and implement a complex music creation model in a cloud environment using Bi-GRUs and deep reinforcement learning. Because of the complexity and scale of the neural network designs involved, such as transformers or hybrid models that combine CNNs and RNNs, the model initially requires high-performance GPUs for effective training. These GPUs speed up matrix calculations, which are essential to deep learning. Moreover, big datasets and the maintenance of many model instances during training stages require sufficient RAM, especially for batch processing and data augmentation tasks.

It can be observed from Table 1 that the RNN technique produced an accuracy of 79.6% in particularly detecting the ragas in the music. The LSTM method was 82.3% accurate in detecting the melodic ragas. Bi-LSTM exhibited higher accuracy than LSTM with 87.5%, which was also higher than the efficiency of the ResNet model with 84.7%. DenseNet showed higher efficiency than the other models considered for comparison, which is 91.4%. However, it was conclusive from the accuracy assessments that the Bi-GRU technique employed in the present research can detect the ragas with 95.6% accuracy, which is the highest among the other deep learning techniques compared. The results of the analysis are presented graphically in Fig. 2.

Once the melodic frameworks are identified, the next step is to improvise the music using the reward model of the DGBRL method. The agent involved in the implementation employs five different reward models trained previously for 100 epochs at differing learning rates of 0.01, 0.001, 0.05, 0.005 and 0.02. In the DGBRL technique, there are two learning phases. The initial instruction of the reward models is done using the approximation of the highest probability. After that, all reward systems are employed for deduction, and the generation model is trained using the melodic ragas identified by the Bi-GRU model for a discount factor value of 0.5. The resulting model’s weights are initialised using the pre-trained reward model with an approximate learning rate of 0.01. The input and improvised music samples are presented in Fig. 3.

After training the model for several epochs with different learning rates, the music improvisation exhibited by the model is evaluated using specific parameters. The parameters considered for the evaluation of music improvisation in this work include pitch frequency (PF), standard pitch delay (SPD), average distance between peaks (ADP), note duration gradient (NDG) and pitch class gradient (PCG).

Table 2 shows the values of the parameters obtained for various reinforcement learning-based algorithms.

The reinforcement learning methods such as the model-based RL algorithm, rule-based RL algorithm, State-Action-Reward-State-Action (SARSA) algorithm, proximal policy optimisation (PPO), and Trust Region Policy Optimization (TRPO) are taken for performance analysis of the approach for music improvisation in the proposed system. The pitch frequency of the model-based and rule-based algorithms lies in the negative notes, whereas for SARSA, PPO and TRPO algorithms, pitch frequency lies in positive notes, with DGBRL having the most negligible value.

On the contrary, the SPD value is positive for model and rule-based RL algorithms and stays in negative notes for SARSA, PPO, TRPO and DGBRL algorithms. The standard pitch delay is found to be lesser for the DGBRL technique. The average distance between the peaks is also less for DGBRL, with a −0.07 value, and the highest for the rule-based RL technique, with +1.67. Further, the values of two important gradients are also investigated. The values of NDG and PCG are minimal for the DGBRL method, with 0.0041 and 0.025. The NDG and PCG values are high, with 0.0082 and 0.067 for the model-based RL algorithm; however, they are lesser than those obtained for the rule-based RL algorithm. Figures 4 and 5 provide the graphical representations of the parameter assessments of NDG and PCG values.

Figure 6 compares the frequency and time domains of the AI-enhanced and original versions, showing both pattern similarities and differences. Initially, the time domain was slightly delayed in training, but there was no high variation between the original and improvised datasets.