Multi-modal sentiment analysis framework for Urdu language opinion videos

Ethical considerations

All videos in the dataset were sourced from publicly available online platforms and did not contain any private or sensitive content. This study exclusively utilized such publicly accessible videos and did not involve direct interaction with human subjects or the collection of any additional private data. Therefore, formal approval from an Institutional Review Board (IRB) or ethics committee was not required, in accordance with our institution’s policies.

We used official Application Programming Interfaces (APIs) (such as the YouTube Data API) and web scraping techniques in compliance with platform terms of service and privacy guidelines. No personally identifiable information beyond the speakers’ public online personas is included in our dataset. The dataset is released on Kaggle under an appropriate research-use license, ensuring that the rights of content creators are respected.

In our analysis and presentation of examples, we translate or transliterate utterances and avoid disclosing any names or specific personal details to protect the speakers’ identities, even though their faces appear in the visual modality. Since the videos are user-generated and already public, privacy concerns are considered minimal. We emphasize that the data is used solely for research on sentiment analysis and not for any form of surveillance or misuse. Overall, we have been mindful of cultural and ethical considerations throughout the data collection and sharing process.

With our dataset established, the following section details the methodological framework employed for extracting and analyzing multi-modal sentiment features from textual, acoustic, and visual data.

Transcription

All videos were transcribed using an online tool (Transkriptor) to obtain text from the spoken content. The initial transcriptions were then proofread by three native Urdu speakers to correct errors. The Urdu language often incorporates words from other languages (especially English) in informal speech. For example, (ایکسپیکٹکررہاتھاکہیہباکسآفسپےبیٹرپرفارمکرےگی) (transliterated: “expect kar raha tha kay yeh box office pe better perform karay gi,” meaning “I was expecting that it would perform better at the box office”) mixes English and Urdu. In many text processing approaches, foreign words are removed during preprocessing. However, in our case removing the English words would alter the meaning of the sentence. Therefore, we opted to keep such words in the transcripts and, when necessary, translated them into Urdu to maintain the context.

Segmentation

Each video’s transcription was saved in a spreadsheet with the filename matching the video file. For each spoken utterance, we recorded its start time and end time in the spreadsheet. These timestamps were then used to segment the videos into individual utterance clips using the moviepy library. The audio track was also extracted for each segment using the same library. Through this process, the continuous videos were broken down into 4,515 segments aligned with the annotated utterances.

Annotation

Three native speakers with postgraduate degrees in Urdu were hired as annotators. They independently labeled each segment’s text, audio, and video with sentiment scores on a scale from −3 to 3, where −3 represents extremely negative sentiment, 3 represents extremely positive sentiment, and 0 indicates neutral sentiment. An example of annotated utterances is provided in Table 2. The inter-rater agreement, assessed using Fleiss’ Kappa (Fleiss, 1971), was 0.937, which is considered highly reliable.

Although the overall agreement was very high, a small subset of utterances exhibited disagreement among annotators. These disagreements primarily occurred in segments with subtle or mixed sentiment, especially near the boundaries between neutral and mild positive or negative sentiments. Ambiguity often arose due to implicit emotional cues, sarcasm, or varying contextual interpretations. To finalize labels, majority voting was applied, recognizing that a portion of the dataset inherently contains subjective ambiguity. This analysis highlights the complex and nuanced nature of sentiment annotation in multi-modal Urdu opinion videos, and underscores the importance of multi-modal cues in resolving such ambiguities.

For consistency in analysis, the numeric scores were ultimately consolidated into three categories. Any score greater than 0 was mapped to Positive, any score less than 0 to Negative, and a score of exactly 0 to Neutral. (In cases where intensity was noted, a score of 1 vs. 3 still fell under the same Positive category, but the magnitude indicated the strength of sentiment; however, our classification task treats the labels as categorical.) The annotators also marked each segment as subjective or objective. Objective expressions conveyed factual information about entities, events, or attributes, whereas subjective expressions represented personal opinions, feelings, or attitudes (Liu, 2010). Table 3 shows examples distinguishing a factual statement from a personal opinion. We used majority voting among the three annotators to finalize the sentiment label and subjectivity tag for each segment.

Textual modality processing

For the textual modality, we employed a CNN model to perform sentiment classification on the transcribed utterances. We chose a CNN architecture inspired by Kim & Jeong’s (2019) work on sentence classification due to its effectiveness in extracting local n-gram features and sentiment-bearing phrases. In our implementation, each word in the transcript was first converted into a 300-dimensional vector using pre-trained FastText word embeddings (Joulin et al., 2016) (version 0.9.2) trained on large Urdu corpora. Using pre-trained embeddings helps in handling Urdu’s rich morphology and out-of-vocabulary words. We kept these embeddings fixed during training (fine-tuning them on our dataset did not yield significant improvement, given the dataset size).

The CNN architecture consists of 11 layers in total: four convolutional layers, four max-pooling layers (each following a convolution), and three fully connected layers. We employed multiple filter sizes (e.g., three, four, and five words in width) for the convolutional layers to capture patterns of different lengths in the text. These convolutional filters slide over the embedded word sequence to detect local patterns indicative of sentiment (for example, negations or strongly positive/negative phrases). Max-pooling layers were applied to the feature maps to retain the most salient features from each filter and reduce dimensionality. The fully connected layers at the end then learned higher-level combinations of these features and mapped them to the sentiment classes. Hyperparameters such as the number of filters, learning rate, and regularization were tuned based on validation performance. Overall, this CNN text model effectively learned representations that map input sentences to the Positive, Negative, or Neutral sentiment classes. It captured both local features (through convolution) and global sentiment tendencies (through the dense layers), enabling accurate sentiment predictions from the textual modality alone.

Acoustic modality processing

In the audio modality, we leveraged the Librosa library (version 0.8.1) (McFee et al., 2015) to extract a rich set of acoustic features that could correlate with sentiment. Librosa provides comprehensive tools for analyzing audio signals. We computed features such as Mel-frequency cepstral coefficients (MFCCs), intensity (energy), pitch (fundamental frequency), and loudness from each audio segment (Babu, Nagaraju & Vallabhuni, 2021). An illustrative example of the audio feature set is shown in Fig. 2. MFCCs are particularly valuable for audio-based sentiment analysis because they capture the spectral characteristics of speech, relating to the timbre of the speaker’s voice. By condensing the frequency content of short audio frames into a set of coefficients, MFCCs provide a compact representation that can reflect emotion-related vocal qualities (for example, a tense or excited voice may have distinct spectral patterns) (Luo, Xu & Chen, 2019).

In addition to MFCCs, we included features representing intensity and loudness (which indicate how energetically or loudly something is said) and pitch (which reflects the intonation). These features can signal emotional intensity and tone—e.g., a raised voice with high intensity and pitch might indicate excitement or anger, whereas a low, flat tone might indicate sadness or a neutral mood. The combination of these features gives a comprehensive representation of the audio signal, enabling our model to capture nuances such as sarcasm (often detectable through intonation), excitement, or frustration in the speaker’s voice (García-Ordás et al., 2021). Indeed, MFCCs are among the most frequently used features in speech emotion analysis, as they succinctly represent the audio spectrum.

Leveraging Librosa’s feature extraction capabilities yielded a rich set of descriptors for each audio segment. These features served as inputs to our audio sentiment classification model (a feed-forward neural network) that learns to map acoustic patterns to sentiment labels. By combining spectral features (MFCCs) with prosodic features (pitch, loudness, intensity), we provided the model with multiple cues, improving its ability to detect emotion-related characteristics in the audio. This forms a robust framework for audio-based sentiment analysis within our multi-modal system.

Visual modality processing

For visual feature extraction from the video segments, we employed three-dimensional convolutional neural networks (3D-CNNs). The motivation for using a 3D-CNN was its ability to extract both spatial and temporal features from a sequence of video frames. Unlike a 2D CNN that processes individual images, a 3D-CNN processes a stack of consecutive frames, thereby capturing inter-frame motion cues (e.g., facial muscle movements over time) in addition to intra-frame features (facial expressions in each frame). Our 3D-CNN model implemented a 10-layer architecture, as depicted in Fig. 3. The input to the network is a sequence of video frames rather than a single image.

In our implementation, we standardized each video segment to a sequence of ten frames (converted to grayscale, 128 $\times$ 128 pixels each) for analysis. These ten frames were uniformly sampled over the duration of the segment—covering the beginning, middle, and end—to ensure that we capture any sentiment-relevant expressions that occur at different times in the utterance. Uniform sampling of ten frames ensures a balanced and comprehensive view of the entire video segment, capturing sentiment cues that may occur at any point. This approach avoids the bias and potential oversight of important expressions that can occur with dynamic sampling methods. It provides a simple, consistent, and computationally efficient way to cover the temporal evolution of facial expressions. We found that uniform sampling provided a better overall representation than picking frames from only a specific part of the segment.

We did not perform face cropping or alignment (the entire frame was fed to the network); however, because we focused on videos where one person is speaking and is usually prominent in the frame, the 3D-CNN can learn to attend to the facial region (indeed, visualizing intermediate feature maps showed many filters responding strongly to the face area). Within the 3D-CNN, the first convolutional layer comprised 16 feature maps with a $2\times 2\times 2$ kernel (height, width, depth across frames). The second convolutional layer had 32 feature maps (also $2\times 2\times 2$ kernel), followed by a max-pooling layer of size $2\times 2\times 2$. The third convolutional layer had 64 feature maps ( $2\times 2\times 2$ kernel), followed by another $2\times 2\times 2$ max-pooling. A fourth convolutional layer with 64 feature maps ( $2\times 2\times 2$ kernel) was then applied, followed by a final $2\times 2\times 2$ pooling layer. These convolution and pooling operations progressively reduced the spatial dimensions while increasing the depth of feature maps, thereby encoding facial features and movements.

Importantly, the temporal dimension (frames) is also convolved and pooled, which enables the network to capture dynamics such as the onset of a smile or frown, or quick head movements. Our frame selection strategy (10 uniformly sampled frames) means the model observes the evolution of expressions across the segment. For example, if a speaker delivers a sarcastic remark with a straight face until a slight smirk at the end, some of the ten frames will likely capture that smirk. If we had taken only a single frame or only contiguous frames in a short window, we might miss such cues. This approach does increase computational load, but it provides the model a broader temporal context. The 3D-CNN outputs were flattened and passed through a series of fully connected layers (of sizes 5,000 and 500) to distill the learned features, and finally to an output layer for classification.

The visual model produces one of the three sentiment classes (Positive, Neutral, Negative) for each segment when trained on its own. We found that the visual modality by itself achieved a classification accuracy of about 69% on our test set, outperforming the audio modality but not as high as the text modality. This indicates that facial expressions and visual cues are indeed informative for sentiment (e.g., smiles generally indicating positive sentiment, frowns or head shakes indicating negative sentiment), but they may sometimes be ambiguous without context (for instance, a polite smile while delivering negative feedback can confuse a naive classifier). The strength of the 3D-CNN is in capturing the temporal dynamics of expressions—rapid changes or subtle microexpressions that static image analysis would miss. Although 3D-CNNs are computationally heavier than 2D CNNs, our results show that the temporal information they capture is valuable for sentiment analysis.

Multi-modal fusion strategies

While unimodal sentiment analysis can yield reasonable results, integrating multiple modalities provides a more robust understanding of sentiment. We explored two primary fusion techniques: feature-level (early fusion) and decision-level (late fusion). In feature-level fusion, features extracted from different modalities are concatenated into a single feature vector, which is then input to a classifier (Shan, Gong & McOwan, 2007). In decision-level fusion, each modality is first used to produce an individual sentiment prediction (for example, through separate classifiers), and then those predictions (or confidence scores) are combined—e.g., via weighted voting or a meta-classifier—to produce the final decision (Cambria, Livingstone & Hussain, 2013).

Both approaches leverage complementary information from multiple sources; the choice between them depends on the specific problem and data characteristics. In our framework, we implemented both strategies. For feature-level fusion (early fusion), after obtaining the intermediate features from the text CNN (the last hidden layer before the classification output), the audio model, and the visual 3D-CNN, we concatenated these feature vectors into one long vector representing the segment’s multi-modal characteristics. This combined vector was then fed into a feed-forward network (we used one hidden layer of 128 neurons with Rectified Linear Unit (ReLU) activation) followed by a softmax output layer to predict the sentiment class. Essentially, the model learns to weigh and mix features across modalities—for example, it might learn that when textual cues are ambiguous, certain audio tones or facial expressions can disambiguate the sentiment. For decision-level fusion (late fusion), we allowed each modality-specific classifier to first produce its own prediction for a segment.

We then combined these predictions using a simple voting scheme: if at least two modalities agreed on the sentiment label, that label was chosen as the final output; in the rare case where all three modalities disagreed, we found that trusting the text modality (which was most accurate alone) yielded the best outcome. We also experimented with a weighted voting (assigning slightly higher weight to text predictions and lower to audio/visual), but the results were similar, and we opted for the simpler majority vote for clarity. Figure 4 illustrates the decision-level fusion process in our system. This approach allows each modality-specific model to specialize in its domain and contributes its judgment, and the fusion mechanism then aggregates these judgments. We found that decision-level fusion slightly outperformed feature-level fusion in our experiments, although both improved significantly over any single modality.

Computational complexity and inference times

To evaluate the practicality of deploying our multi-modal sentiment analysis framework in real-world applications, we analyzed the computational complexity and inference times of our models. The experiments were performed using an NVIDIA GeForce RTX 3080 Ti GPU and an Intel Core i7 CPU.

• Training Time: On the RTX 3080 Ti, training the full multi-modal model (with all three modalities) took approximately 2 h per epoch, and we trained for a total of 10 epochs to reach convergence. In contrast, training on the CPU was significantly slower, taking more than 12 h per epoch.

• Inference Time: The average inference time per segment was around 90 ms on the GPU and about 400 ms on the CPU. These speeds indicate that our model can process new data in near-real-time on modern hardware, making it feasible for applications such as live sentiment monitoring (especially when a GPU is available).

• Memory Usage: During training, GPU memory usage peaked at roughly 10 GB, whereas inference required around 3 GB of GPU memory. These requirements are manageable on contemporary high-end hardware. For deployment on devices with limited resources, techniques like model pruning or quantization could be considered to reduce the model’s footprint.

In summary, while our framework is computationally intensive during training (particularly due to the 3D-CNN and the need to handle three modalities), its runtime performance (especially with GPU acceleration) is suitable for practical use. The efficient inference capability means it could be integrated into systems that require on-the-fly sentiment analysis of multimedia content.

Equipped with a robust methodology, we now present the experimental results, demonstrating the effectiveness of our multi-modal sentiment analysis approach on the curated Urdu dataset.

Evaluation on standard benchmark dataset

To further validate the effectiveness and generalizability of our multi-modal sentiment analysis framework, we applied it to the widely-used CMU-MOSI multi-modal sentiment analysis dataset. The CMU-MOSI dataset contains short video clips (monologue utterances) in English annotated with sentiment scores, and it serves as a common benchmark for evaluating multi-modal sentiment analysis models. We retrained our entire framework on the CMU-MOSI data while maintaining the same architecture and fusion strategies used in our original Urdu experiments. Since CMU-MOSI is an English-language dataset, we substituted our text model’s embedding layer with 300-dimensional English FastText embeddings (keeping the CNN architecture identical) to handle the English transcripts. Aside from this embedding change, no modifications to the model were required. We processed the MOSI videos in a similar manner to our own: each video clip (utterance) comes with a transcript, and we extracted audio features and video frames using the same procedures. The original MOSI sentiment annotations range from −3 to +3 (a continuous scale); we mapped these to our three sentiment classes by considering any rating > 0 as Positive, any rating < 0 as Negative, and a rating of 0 as Neutral, to align with our classification setup. We then trained our multi-modal model from scratch on the MOSI training set and evaluated it on the standard MOSI test set.

Error analysis

While our multi-modal model achieves high overall accuracy, examining the cases where it fails can provide insights into its limitations and potential areas for improvement. We performed a detailed error analysis on our Urdu test set, focusing on the segments that were misclassified by the fused model. Out of 903 test segments, the model misclassified approximately 35 instances (about 3.9%, which aligns with the overall error rate given the 91.18% accuracy). We categorized these errors based on the apparent primary cause and identified several common patterns:

• Textual misinterpretation: In some cases, the textual content of the utterance led the model astray. A common scenario was sarcasm or ironic statements where the literal text is positive but the intended sentiment is negative (or vice versa). For example, one segment’s transcript was “Yeh kafi interesting tha, haan bhai, bohat zabardast” (translated: “Yeah, this was quite interesting, oh yeah, really amazing”) said in a clearly sarcastic tone. The ground truth label was Negative, but the model predicted Positive, likely because it took the positive words (“interesting,” “amazing”) at face value and didn’t fully catch the sarcasm through the other modalities. This indicates a limitation in understanding sarcasm or tone purely from text; improving the model’s handling of such cases might involve giving more weight to audio/visual cues (tone, smirk) or using a specialized sarcasm detector.

• Audio misinterpretation: Some errors were primarily due to misleading audio cues. For instance, in one misclassified segment, there was loud upbeat background music and the speaker’s voice had a cheerful tone, but the actual content was a complaint (negative sentiment). The model predicted Positive, seemingly influenced by the “happy” acoustic atmosphere. In such cases, the presence of background music or an unusual tone can confuse the audio model. This suggests that incorporating some form of speaker voice isolation (to ignore background sounds) or training on more data with background noise could help. It also underscores the importance of cross-checking with text content—had the model weighted the negative words more, it might have overridden the misleading audio.

• Visual misinterpretation: In a few instances, the visual cues led to mistakes. For example, one speaker maintained a neutral facial expression while delivering very negative verbal content; the model, seeing a lack of frown or anger on the face, predicted Neutral whereas the correct label was Negative. In another case, a speaker had a habit of smiling or chuckling while criticizing (a form of nervous laughter or social masking), and the model incorrectly leaned towards a positive/neutral prediction because it “saw” a smile. These cases highlight that facial expressions can sometimes be deceiving or person-dependent. The model might benefit from learning more person-specific baselines (e.g., that this particular speaker often smiles even when negative) or from combining modalities more intelligently (realizing “if text is strongly negative but face is smiling, perhaps it’s sarcasm or a polite smile”).

• Modality conflict: There were cases where different modalities gave conflicting signals, and the model chose the wrong one to trust. For instance, consider an utterance where the text is mildly positive, but the tone is dripping with sarcasm (truly negative sentiment). If the model were to believe the text over the audio, it would err. We observed instances of this: the text contained polite or positive words, the audio/visual indicated scorn or negativity, and the model sometimes got confused. Although our fusion scheme generally worked, these edge cases suggest we might improve by using a more sophisticated fusion method—perhaps an attention mechanism that can dynamically decide which modality is more reliable for a given segment (e.g., if it detects a discrepancy, rely more on audio/visual which might carry sarcasm cues).

• Ambiguous or borderline cases: A small number of errors were on segments that were genuinely hard to label even for humans—e.g., very neutral statements that could be interpreted slightly positively or negatively depending on context, or vice versa. In such borderline cases, the model’s prediction might differ from the annotator’s label, but it’s not obviously “wrong” in a practical sense. For example, a segment saying “It was okay, I guess” was labeled Neutral by annotators but predicted as slightly Positive by the model. These are minor and expected discrepancies; handling them might require more nuanced classes or context, but for our three-class setup, a small confusion around the neutral boundary is acceptable.

By analyzing these errors, we gained several insights. First, sarcasm and irony remain challenging for our model; explicitly modeling them (possibly by detecting vocal tone patterns or using textual cues like hyperbolic words) could be a future improvement. Second, background noise or irrelevant audio features can mislead the model, so integrating a noise-robust training strategy or preprocessing step could help. Third, the model could benefit from a more dynamic fusion mechanism to handle conflicting modality inputs—future work could explore trainable fusion with attention or gating that learns to trust one modality over another in certain contexts (for instance, if audio tone strongly contradicts text, it might indicate sarcasm). In light of these error patterns, we propose several refinements: incorporating a sarcasm detection component, augmenting training data with more varied background conditions (to teach the model to focus on the speaker’s voice), and exploring advanced fusion architectures. We discuss these ideas further in the Future Work section. The error analysis ultimately provides a roadmap for how the model can be improved to handle the nuanced and complex nature of human communication.