Joint intent detection and slot filling with syntactic and semantic features using multichannel CNN-BiLSTM

Classical approaches

Joint learning classification for intent detection and slot filling has been widely studied by many researchers. Jeong & Lee (2008) conducted one of the earliest studies on this topic (Jeong & Lee, 2008). They used a triangular-chain conditional random field (CRF) to model the dependencies between slots and intent. In another study conducted by Wang (2010), a maximum entropy model was used for intent detection and CRF for slot-filling tasks. In addition, (Celikyilmaz & Hakkani-Tur, 2012) presented a multilayer hidden Markov model for joint learning tasks. Although classical models can successfully capture the dependencies between slots and intent, they face scalability challenges, particularly when dealing with large datasets (Cohn, 2007; Jia, Liang & Liang, 2023; Mairesse et al., 2009). Classical models often involve mathematical computation during training and prediction. As the size of the dataset increases, these computations become computationally expensive, making it difficult to scale models efficiently. They also encountered issues with feature creation, as features needed to be predefined by humans before model training could occur (Weld et al., 2022; Xu & Sarikaya, 2013; Zhang & Wang, 2016). The manual feature engineering process is time-consuming and limits the ability of these models to adapt to the complexity of natural language (Ferrario & Naegelin, 2020). These limitations have led to the development of alternative approaches, particularly deep-learning approaches.

Deep learning approaches

The first study to exploit deep neural networks in a joint learning model for intent detection and slot filling was conducted by Xu & Sarikaya (2013). They utilized a CNN for feature extraction and a CRF for analysis. Guo et al. (2014) employed recursive neural networks to analyze the dependency structures of utterances, incorporating features such as n-grams and named entities, and used tree-derived features for intent detection and slot filling. Although their model achieved state-of-the-art performance, it lacks bidirectional connections, leading to slow and ambiguous models. Many subsequent studies have employed different methodologies using RNNs for joint learning classification owing to their suitability for capturing temporal dependencies. In some studies, RNNs were used as word-level classifiers, with intermediate hidden states used for slot filling tasks and the weighted sum of the hidden states for intent detection. In other studies, RNNs functioned as sentence classifiers, using the last hidden state was used for intent detection. Zhou et al. (2016) proposed a hierarchical LSTM (HLSTM) with two layers: the bottom-layer served as a sentence classifier, with the last hidden state was employed for intent detection, while the upper layer performed word-level classification for slot labeling. Liu & Lane (2016b) introduced a joint model that using LSTM cell. In their proposed model, the intent and slots are detected at each time step as the input arrived, and the overall intent was obtained using the last hidden state. Hakkani-Tür et al. (2016) tagged each utterance with a special token before passing it to a BiLSTM to obtain a latent semantic representation of the entire input utterance for intent detection along with intermediate hidden states for slot labeling. Liu & Lane (2016a) proposed another joint model that used a context vector generated from the weighted sum of RNN hidden states to obtain both intent and slots. Zhang & Wang (2016) combined a gated recurrent unit and max pooling layer in their joint model. Chao, Ke & Xiaofei (2020) employed a BiLSTM model with POS tag as embeddings, while Firdaus et al. (2018) used a bidirectional gated recurrent unit model with hybridized non-contextual and POS tag embeddings. Most recently, Hardalov, Koychev & Nakov (2023) proposed a transformer based model incorporating named entity recognition as a feature.

Hybrid models

Our approach is closely related to the work in Firdaus et al. (2019), where two non-contextual embeddings were used. Glove and word2vec were concatenated and served as features for a CNN to capture the sentence representation. This was then fed into a BiLSTM to obtain contextual information within utterances. However, this method’s reliance on non-contextual embeddings is a notable drawback. In another study conducted by Qin et al. (2021), a hybrid model combining BERT and BiLSTM was proposed to establish bidirectionality between intent detection and slot filling tasks. However, this study considered only contextual embeddings.

In nutshell, while classical models laid the foundation for intent detection and slot filling, their scalability and manual feature engineering requirements limit their applicability to large datasets and complex language tasks. Deep learning approaches, particularly those leveraging RNNs and CNN, have significantly advanced the field by automating feature extraction and capturing temporal dependencies. However, these models still face challenges such as limited contextual understanding and ambiguity in bidirectional connections. Hybrid models have attempted to bridge these gaps by combining various types of embeddings and model architectures. Nevertheless, existing hybrid models often rely on non-contextual embeddings, which fail to fully exploit the contextual nuances of languages. Moreover, the integration of syntactical features such as POS tags remains underexplored. Our proposed model addresses these issues by integrating contextual, non-contextual, and syntactic embeddings using a multichannel CNN architecture. This approach not only enhances feature representation but also disambiguates polysemous words, leading to improved performance in intent detection and slot filling tasks. To the best of our knowledge, this is the first study to use an MCNN architecture that leverages contextual, non-contextual, and POS tag embeddings in different channels to improve the performance of joint learning classification for intent detection and slot filling.

Proposed model

Figure 1 illustrates the architecture of the proposed joint model for intent detection and slot filling. This model integrates both the CNN and BiLSTM, as used in Wu et al. (2024). The proposed model comprises three input layers, three convolutional layers with subsequent max pooling, a shared BiLSTM encoder, and two dense layers that implement softmax functions. These components jointly detect the intent of the user’s input utterance and the associated slots by assigning them with multiclass labels (B, I, O), where “I,” “O,” and “B” signify Inside, Outside, and Beginning of slots, respectively. Details of the model are described in the following subsections.

Input layer (embedding layer)

The proposed model utilizes three types of embeddings: non-contextual, contextual, and POS tag embeddings each as an independent input channel for the network. This layer converts each word in the input sentence into its corresponding vector.

Convolutional layer

After encoding all the sentences into vectors and applying zero padding to ensure a uniform length across all embedding channels. Convolution operations are then applied to extract local features from the embedding channels. This allows the model to identify and leverage important patterns and dependencies in the data. In this study, we applied convolution operations with different filter sizes to POS embeddings (P), contextual embeddings (Q), and non-contextual embeddings (R). Conventionally, convolution operations involve sliding filters over embeddings to detect patterns and to generate feature maps.

Let w_n ∈ R^hws, w_c ∈ R^hws, w_p ∈ R^hws be filters applied for the non-contextual, contextual, and POS embedding, respectively, where ws is the window size h is the embedding dimension for each word. Thus, the generated features can be expressed as: (2)${\displaystyle Z_i=f\left(w.x_{i:i+ws-1}+b\right)}$ where ( ⋅) is the convolution operator, f is a non-linear activation function (ReLU in this case), b is the bias. This function is applied to each window $\left[x_{1:ws},x_{2:ws+1},\dots ,x_{n-ws:n}\right]$ in each channel. Thus, the feature map for each embedding type is given as: (3)${\displaystyle \text{POS features:}{z}_p=\left[z_1^p,z_2^p,\dots ,z_{n-ws+1}^p\right]}$ (4)${\displaystyle \text{Contextual features:}{z}_q=\left[z_1^q,z_2^q,\dots ,z_{n-ws+1}^q\right]}$ (5)${\displaystyle \text{Non-contextual features:}{z}_r=\left[z_1^r,z_2^r,\dots ,z_{n-ws+1}^r\right]}$

However, it is important to note that various filters can be employed to capture diverse features from embedding matrices (Rakhmatulin et al., 2024).

Max pooling layer

The pooling operation is designed to reduce the resolution of the feature maps by applying a pooling function to several units within a defined local region, known as the pooling size. This process helps to generalize the features obtained from the convolutional layer (Zhao & Zhang, 2024). Specifically, the purpose of using the max pooling layer in this study was to capture the most significant features from the convolutional layer by selecting the maximum value from each feature map segment (Kim, 2014). Thus, the maximum values for POS and contextual and non-contextual features are expressed as follows: (6)${\displaystyle {\tilde{Z}}_p=\mathit{max}\left[z_1^p,z_2^p,\dots ,z_{n-ws+1}^p\right]}$ (7)${\displaystyle {\tilde{Z}}_q=\mathit{max}\left[z_1^q,z_2^q,\dots ,z_{n-ws+1}^q\right]}$ (8)${\displaystyle {\tilde{Z}}_r=max\left[z_1^r,z_2^r,\dots ,z_{n-ws+1}^r\right]}$

After pooling the maximum features for each channel, we obtain the final maximum feature by concatenating the three features using the following equation: (9)${\displaystyle Z={\tilde{Z}}_P^1\oplus \cdots \oplus {\tilde{Z}}_P^m\oplus {\tilde{Z}}_q^1\oplus \cdots \oplus {\tilde{Z}}_q^n\oplus {\tilde{Z}}_r^1\oplus \cdots \oplus {\tilde{Z}}_r^o}$ where ⊕ is the concatenator operator and m, n, o are the filters for POS tag, contextual, and semantic features respectively.

Shared BiLSTM layer

Bidirectional long short-term memory network (BiLSTM) is a type of recurrent neural network (RNN) architecture known for its ability to capture long-term dependencies in sequential data. BiLSTM is an extension of the basic LSTM that integrates forward and backward LSTM structures to capture deeper contextual information (Qi et al., 2024). Essentially, BiLSTM employs two separate LSTM networks to analyze input sequences. The forward LSTM processes the sequence starting from the first token, whereas the backward LSTM processes it from the final token. Subsequently, the outputs of both LSTM networks are merged at each time step. This arrangement ensures that the output at any given time step encompasses both the preceding and succeeding context from the input sequence, thereby facilitating a more effective capture of long-term dependencies within the sequence. The essence of using BiLSTM in this study is to capture sequential dependencies, which are essential for analyzing natural languages (Pogiatzis & Samakovitis, 2020). Thus, the concatenated output of the convolutional layer Z is passed through the BiLSTM layer to generate the output h_t. (10)${\displaystyle \overrightarrow{h_t}=LSTM\left(z_t,\overrightarrow{h_{t-1}}\right)}$ (11)${\displaystyle \stackrel{⃖}{h_t}=LSTM\left(z_t,\stackrel{⃖}{h_{t-1}}\right)}$ (12)${\displaystyle h_t=\overrightarrow{W_t}.\overrightarrow{h_t}+\stackrel{⃖}{W_t}.\stackrel{⃖}{h_t}+b}$ The output of the BiLSTM is given as: (13)${\displaystyle H=\left[h_1,h_2,\dots ,h_n\right]}$

Output layer

Model versions

Our proposed model for the joint learning classification of intent detection and slot filling encompasses seven versions, each of which is tailored to different settings of embeddings passed to different channels in the MCNN model. These versions are described below.

• MCNN-BiLSTM-1: This version primarily aims to assess the influence of pretrained word embeddings. The non-contextual word embedding channel was initialized randomly, whereas the other channels were ignored. This means that only randomized word embeddings were considered for training.

• MCNN-BiLSTM-2a: In this setting, the non-contextual embedding channel is initialized with pretrained word2vec embeddings with other ignored channels.

• MCNN-BiLSTM-2b: In this setting, the non-contextual embedding channel is initialized with domain-specific word2vec embeddings with other ignored channels.

• MCNN-BiLSTM-3: In this case, only the channel for the BERT contextual embeddings was initialized, and all others were ignored.

• MCNN-BiLSTM-4: Two channels were used for training. Specifically, the contextual embedding channel was initialized using BERT embeddings and the non-contextual embedding channel was initialized using word2vec embeddings.

• MCNN-BiLSTM-5: Two channels were used for training. The non-contextual embedding channel was initialized with word2vec embeddings, and the POS embedding channel was initialized with POS tag embeddings.

• MCNN-BiLSTM-6: In this version, contextual embedding and POS tag embedding channels are initialized using BERT and POS tag embedding, respectively.

• MCNN-BiLSTM-7a: In this version, all three channels are used for training. Specifically, we used domain-specific word2vec embeddings trained on the ATIS and SNIPS datasets as inputs for non-contextual embeddings. This study aimed to assess the impact of general embeddings compared to domain-specific embeddings.

• MCNN-BiLSTM-7b: This variant is similar to that of MCNN-BiLSTM-7a. However, in this case, instead of domain-specific word2vec embedding, we used a general purpose pretrained word2vec embedding as an input to the non-contextual embedding channel.

Experiment study

In this section, we describe the datasets used in our experiments. Subsequently, we present a detailed experimental methodology for assessing the effectiveness of the proposed approach. Finally, we conducted a comparative analysis of the baseline methods.

Dataset

To verify the validity of our proposed model, we conducted experiments on the most widely used datasets in NLU research, namely, ATIS (Hemphill, Godfrey & Doddington, 1990) and SNIPS (Coucke et al., 2018). These datasets were selected in this study for their complementary characteristics: ATIS provides a focused, domain-specific challenge, whereas SNIPS offers a broader and more diverse set of challenges. Together, they provide a comprehensive test bed for evaluating the effectiveness and generalizability of our proposed model in comparison with the existing models. Table 1 presents the key characteristics of the ATIS and SNIPS datasets.

Table 2 presents an example of a semantic frame from the ATIS dataset, using the sentence “I want to fly from Baltimore to Dallas round trip.” The slots use the IOB (in-out-begin) format for slot tagging. In this example, the intent is to find a flight, with ‘Baltimore’ tagged as the departure city, ‘Dallas’ as the arrival city, and ‘round trip’ indicating the type of trip.

Experimental set-up

We conducted a grid search to select the hyperparameters that would yield the best performance for our model. The grid search involved systematically varying the hyperparameters and evaluating the performance of the model on a validation set. The following selections were made based on the empirical testing and previous research findings:

Comparative methods

To examine the effectiveness of our proposed model, we initially identified the best-performing settings of our model and subsequently compared them with the following baseline models:

• RNN-LSTM (Hakkani-Tür et al., 2016): This model uses a BiRNN with LSTM to perform joint learning classification using lexical features represented by 1-hot encoding.

• BiRNN-attention (Liu & Lane, 2016a): This model employs an encoder–decoder architecture with an attention mechanism with random embedding initialization.

• CNN-BiLSTM (Wang, Tang & He, 2018): Uses CNN to extract features from word2vec word embeddings and the BiLSTM layer as an encoder and decodes it with an attention-based RNN. This baseline model was used as a representative deep learning-based method that exploits the CNN and BiLSTM models.

• BiGRU/BiLSTM-MLP (Firdaus et al., 2018): A multitask ensemble method with features obtained from glove, word2vec, and POS tags.

• BiLSTM-self-attention (Qin et al., 2019): This model uses stack propagation, which utilizes intent information to guide the slot filling task.

• BiLSTM+Attention (Chao, Ke & Xiaofei, 2020): A joint model with POS scaling attention to help the model focus on verbs and nouns that are important in representing user behavior and object operations, respectively.

• SASGBC (BERT only) (Wang, Huang & Hu, 2020): This model uses BERT to encode an input sequence, integrate intent information with slot gates, and establish a contextual semantic relationship with self-attention.

• BiLSTM+attention (Qin et al., 2021): This model exploits the pretrained BERT model together with BiLSTM and co-interactive attention, and initializes the embedding layer with glove embeddings.

• BERT (Hardalov, Koychev & Nakov, 2023): A contextual BERT model is used to design a joint model with name entity recognition(NER) features.

It should be noted that following the common practice in the literature (Firdaus et al., 2018; He et al., 2021), this study directly used the findings of the aforementioned state-of-the-art methods presented in their original publications and compared them with our proposed model.

Statistical analysis of the proposed model performance in comparison to the baseline

To validate the significance of the improvements in our proposed model compared to state-of-the-art baseline methods, we used a statistical package for social sciences to conduct a statistical significance assessment using a t-test with a significance level of 0.05. A t-test yielding a p-value less than 0.05 indicates that the observed differences are statistically significant, suggesting less than 5% probability that these differences occurred by chance, thereby confirming a substantial difference between the outcomes of the two groups.

Tables 6 and 7 present the results of our statistical analyses on the accuracy and F1-score metrics, respectively, based on a five-fold validation on the ATIS dataset. Notably, the computed p-values in these tables were below the critical significance level of 0.05, indicating statistical significance. The statistical analyses of accuracy and F1-score metrics are presented in Tables 8 and 9, respectively. The results show that the performance of the proposed model is statistically significant based on the F1-scores (Table 9). However, in Table 8, the baseline model proposed by Hardalov, Koychev & Nakov (2023), which uses an enriched BERT model, shows no statistically significant difference from the proposed model. This outcome can be attributed to the fine-tuning of the entire BERT transformer, capturing a richer task-specific context. However, it is computationally more expensive compared to using static BERT embeddings, as employed in our proposed model.