Design and implementation of the international news commentary Data Intelligent Processing System

Echarts

The implementation of data visualization can be broadly categorized into two primary methodologies: one is user-oriented, such as Dycharts and Tableau. These tools provide intuitive user interfaces that make creating visual charts and graphs easy and quickly. The other type is developer-oriented and is represented by third-party libraries such as Matplotlib in Python and JavaScript-based visualization frameworks such as D3.js, Highcharts, ECharts, and Chart.js (Pei, 2018). With increased adaptability and customization, these frameworks allow developers to add dynamic effects and interactive features to web pages.

ECharts, developed and maintained by Baidu, is a JavaScript library for constructing interactive charts and visualizations. It includes several popular chart types, ranging from line and bar charts to scatter plots and pie charts. In addition, ECharts provides robust support for interactive operations and real-time data updates, contributing to a dynamic and immersive user experience (Li et al., 2018).

Flask

Flask is a lightweight web application framework based on Python (Copperwaite & Leifer, 2015). Its notable features include flexibility, user-friendly interface, and scalability, supported by the foundational components of Werkzeug and Jinja2 (Relan , 2019). As a result, Flask is optimal for rapid prototyping, data visualization, and compact web environments. This inherent ease of use allows developers with limited experience in web development to quickly orchestrate websites with increased proficiency (Niu & Li, 2019). Classified as a “microframework”, Flask is characterized by its concise core codebase and streamlined architectural essence. Unlike its counterparts, such as Django and Pyramid, among Python web frameworks, Flask is distinguished by its lightweight nature and adaptive versatility (Ghimire, 2020). The underlying design ethos of the Flask framework is rooted in a desire to maintain an unadorned elegance while preserving a nuanced degree of adaptability.

Implementation of abstract analysis

This section implemented text summarization for international news commentary using the TextRank algorithm (as shown in Fig. 5). The specific steps were as follows:

The vectorizer. fit(contents) step involved using a TF-IDF vectorizer to transform the text into feature vectors, which is suitable for sentence similarity calculations. TF-IDF (Term Frequency-Inverse Document Frequency) is a statistical method used to evaluate the importance of a word within a corpus. The TF-IDF algorithm consists of term frequency (TF) and inverse document frequency (IDF). TF assesses the prevalence of a term within a document, reflecting its importance. The calculation of TF is as follows: (1)${\displaystyle TF\left(t,d\right)=\frac{\text{Number}\mathrm{of}\text{times}\text{term}\mathrm{t}\text{appears}\mathrm{in}\text{document}\mathrm{d}}{\text{Total}\text{number}\mathrm{of}\text{terms}\mathrm{in}\text{document}\mathrm{d}}.}$

Equation (1) represents the frequency of occurrence of a word within a single document, with the TF value increasing as the frequency of occurrence becomes higher.

Inverse Document Frequency (IDF) measures a term’s rarity across the entire corpus, offering a counterbalance to the TF. The computation of IDF is detailed as follows: (2)${\displaystyle \mathrm{IDF}\left(\mathrm{t},\mathrm{D}\right)=log\left(\frac{\text{Total}\text{number}\mathrm{of}\text{documents}\mathrm{in}\mathrm{the}\text{corpus}\mathrm{D}}{\text{Number}\mathrm{of}\text{documents}\text{containing}\text{term}\mathrm{t}+1}\right).}$

Equation (2) demonstrates that the IDF (Inverse Document Frequency) value is inversely related to the prevalence of a word throughout the corpus. A lower occurrence frequency signifies a higher degree of uniqueness for the word, resulting in an increased IDF value.

The final TF-IDF score is obtained by multiplying TF and IDF, as presented: (3)${\displaystyle TF-IDF\left(\mathrm{t},\mathrm{d},\mathrm{D}\right)=TF\left(\mathrm{t},d\right)\ast IDF\left(t,D\right).}$

Equation (3) represents that a higher TF-IDF score indicates a term’s significant presence in a particular document while being rare across the corpus, thus highlighting its importance for the document’s content.

The preprocess_text(text) function was crucial for text preprocessing, where it removed HTML tags using regular expressions and processed the text through tokenization and cleaning. This step ensured that the sentences were standardized for further processing.

Subsequently, the text_rank_summary(text, num_sentences =2) function generated summaries by preprocessing text into clean sentences, vectorizing them, and constructing a sentence similarity matrix. The application of the PageRank algorithm was vital in determining sentence scores and selecting the top-ranked sentences for the summary.

Finally, the page_rank(similarity_matrix, max_iterations =100, tolerance =1e−4, damping_factor = 0.85) function implemented the PageRank algorithm to iteratively compute sentence scores, continuing until reaching the maximum number or the tolerance level, resulting in the final sentence scores. These steps collectively formed a comprehensive process for analyzing and summarizing text data.

The TextRank algorithm, which worked as a graph-based ranking algorithm, automatically identified the most important sentences in the text (Mihalcea & Tarau, 2004). The adoption of TextRank is predicated on its inherent capabilities to efficiently extract pivotal sentences from voluminous textual datasets, making it particularly apt for the analysis of news commentary. The brief steps are as follows:

Document segmentation: The given document D is divided into n sentences S₁, S₂, …, S_n,  each sentence being a collection of words. Subsequently, a network graph $G=\left(V,E\right)$ is constructed, where V is the set of sentences in the document; E is the set of edges between nodes, in the form $\left(V_i,V_j\right)$, representing an edge between nodes V_i and V_j. The similarity between sentences is calculated using the following formula as the weight of the edges: (4)${\displaystyle W_{ij}=\text{similarity}\left(S_i,S_j\right)=\frac{\left|\left\{w_{\mathrm{k}}\right\},w_k\in S_i,w_k\in S_j\right|}{log\left(\left|S_i\right|\right)+log\left(\left|S_j\right|\right)}}$ where W_ij represents the weight, or similarity, between sentences S_i and S_j.

Weighted graph formation: With the inclusion of weights W, a weighted undirected graph $G^{{}^{\prime }}=\left(V,E,W\right)$ is formed. The cumulative weight of each node is iteratively calculated as follows: (5)${\displaystyle WS\left(V_i\right)=\left(1-d\right)+d\ast \sum _{{}_{V_i\in In\left(V_i\right)}}\frac{W_{ji}}{\sum _{V_k\in Out\left(V_j\right)}{W}_{jk}}WS\left(V_j\right).}$

W_ij denotes the weight between two nodes V_i and V_j, correlating to the similarity between sentences S_i and S_j; The notation $In\left(V_i\right)$ denotes the collection of edges directed towards node V_i, while Out(V_j) denotes the collection of edges emanating from node V_j, with V_i and V_j representing the node under calculation and the node to be shared, respectively. Consequently, WS(V_i) refers to the weight of the node under calculation, whereas WS(V_j) denotes the weight of the node to be shared. The term d serves as the damping factor, signifying the probability of transitioning from a given node to any other node within the network.

Convergence and summary selection: The iterative process continues until the scores stabilize (convergence), after which sentences are ranked based on their final scores. The top k sentences, where is a predefined number, are selected to compose the summary, effectively capturing the document’s key points.

As a result, this feature automatically extracted crucial, representative information from complex international news commentary to form the summary of the article. This automated summarization method enabled users to quickly grasp the core information of the article without having to read the entire content, saving time and providing convenience. It was especially vital in the era of overwhelming information overload and improved the efficiency of information retrieval for users.

Implementation of text sentiment analysis

The system utilized NLTK’s sentiment analysis tool Valence Aware Dictionary for sEntiment Reasoning (VADER), to perform sentiment analysis on the preprocessed text (Fig. 6). VADER relied on a sentiment lexicon that contained a large collection of common English words, associated sentiment polarity, and intensity scores (Elbagir & Yang, 2019). It can recognize negation words and degree adverbs, taking into account the context of words in the text and their relationships to each other. VADER calculated a comprehensive sentiment score that categorized the sentiment as positive, neutral, or negative based on this score.

In this study, a standardized threshold for classifying sentences as positive, neutral, or negative was set as follows (Yin et al., 2022): ${\displaystyle Sfi=\left\{\begin{array}{c}\text{positive vscore}>=0.05\hfill \\ \text{negative vscore}<=-0.05\hfill \\ \text{neutral otherwise}\hfill \end{array}\right.}$ Where vscore was the composite score of the i-th news, Sfi was the final polarity of the tweet. If the composite score was not less than 0.05, the sentence was considered positive. If the score was not greater than −0.05, its polarity was negative. Otherwise, the sentence polarity was neutral.

Implementation of keyword extraction

In Python, keyword extraction can be accomplished by using the Natural Language Toolkit (NLTK) (Loper & Bird, 2002) to calculate word frequencies and store the N most frequent words as keywords in a database (as shown in Fig. 7). The critical steps implemented for keyword extraction were as follows:

The critical steps for keyword extraction were executed as follows: Firstly, the NLTK’s word_tokenize function was employed for tokenization, breaking down the text into individual words. Secondly, the FreqDist class from NLTK was used to calculate the frequency distribution of the words obtained post-tokenization, facilitating the handling of frequency distributions. Lastly, the most_common(N) function retrieved the top N words with the highest frequencies, returning them in descending order. These steps, when combined, enabled the effective extraction of keywords from the text data. The extracted keywords and their frequencies were then ready to be stored in a database for subsequent analysis or reference purposes.

Implementation of similarity recommendation analysis

In this paper, the system implemented a similar article recommendation feature by analyzing the similarity between articles. The system utilized natural language processing techniques such as word segmentation, TF-IDF (Ramos, 2003), and text similarity calculation methods such as cosine similarity (Rahutomo, Kitasuka & Aritsugi, 2012). Cosine similarity quantifies the similarity between two vectors in an n-dimensional space by computing the cosine of the angle between them. This measure is obtained by dividing the dot product of the vectors by the product of their magnitudes. (6)${\displaystyle \mathrm{Sim}\left(\overrightarrow{\mathrm{A}},\overrightarrow{\mathrm{B}}\right)=\frac{\overrightarrow{A}\cdot \overrightarrow{B}}{∥\overrightarrow{A}∥∥\overrightarrow{B}∥}=\frac{\sum _{i=1}^n{A}_i{B}_i}{\sqrt{\sum _{i=1}^n{A}_i^2}\sqrt{\sum _{i=1}^n{B}_i^2}}.}$

The cosine similarity value ranges from −1 to 1, where 1 means $\overrightarrow{A}$ and $\overrightarrow{B}$ are in the same direction, 0 indicates orthogonality (no similarity), and -1 means they are in opposite directions. In addition to its primary applications, the algorithm has found utility in a variety of other domains. The Cosine Similarity-based Image Filtering (CosSIF) algorithm has significantly enhanced medical image analysis models by improving the diversity and quality of training data (Islam, Zunair & Mohammed, 2024). Moreover, the cosine similarity matrix-based video scene segmentation (CSMB-VSS) algorithm has capitalized on the relationships between video scenes and shot similarities to significantly optimize video segmentation (Chen et al., 2024). Furthermore, the integration of cosine similarity with the Alternating Least Squares (ALS) algorithm has substantially enhanced the accuracy and performance of recommendation systems (Hasan & Ferdous, 2024).

By using these techniques, the system efficiently computed the similarity between many articles, which enabled the recommendation of similar articles (as shown in Fig. 8).

Implementation of topic analysis

This system utilized the gensim’s corpora module and LDA algorithm (Yu & Yang, 2001) for topic analysis (as shown in Fig. 9), to efficiently parse and categorize the thematic elements of text data, with the following key steps:

Initially, the text data underwent word segmentation, wherein individual words were identified using a segmentation technique. Subsequently, a corpus and a corresponding dictionary were developed from this segmented data. The corpus served as a bag-of-words representation of the text, while the dictionary mapped words to their unique integer identifiers. The topic model was then trained utilizing the Latent Dirichlet Allocation (LDA) algorithm, a probabilistic method for identifying latent topics in collections of text. The LDA model is characterized by a distinct hierarchical architecture, which comprises three sequential layers: the corpus level, the topic level, and the level of feature words. This architecture is delineated in Fig. 10. Each document is assumed to be generated by a mixture of topics. This mixture is represented by a distribution θ_d over topics for document d, which is drawn from a Dirichlet distribution with hyperparameter α: ${\theta }_{\mathrm{d}}\sim Dir\left(\alpha \right)$. Each topic is characterized by a distribution over words. This distribution β_k for topic k isalso drawn from a Dirichlet distribution with hyperparameter β: ${\beta }_k\sim Dir\left(\beta \right)$. For each word in a document, a topic z_d,n is chosen from a Multinomial distribution parameterized by θ_d, and then a word w_d,n is chosen from a Multinomial distribution parameterized by β_zd,n: Choose topic $z_{d,n}\sim Multinomial\left({\theta }_{\mathrm{d}}\right)$, Choose word $w_{d,n}\sim Multinomial\left({\beta }_{z_{d,n}}\right)$. The joint probability of the model’s variables can be expressed as: (7)${\displaystyle p\left(\theta ,z,w|\alpha ,\beta \right)=p\left(\theta |\alpha \right)\prod _{n=1}^{N}p\left(z_n|\theta \right)p\left(w_n|z_n,\beta \right).}$

Furthermore, the ultimate goal of applying LDA is to infer the posterior distribution: (8)${\displaystyle p\left(z,\theta |w,\alpha ,\beta \right).}$

However, due to its computational complexity, direct computation of this posterior is infeasible for non-trivial corpora. Thus, approximation techniques such as Variational Inference or Gibbs Sampling are employed to estimate the distribution of topics across documents and the distribution of words across topics. Finally, the identified topic keywords were stored in a database, facilitating the generation and output of automatic article summaries. This application of the LDA model directly facilitated the extraction of topic keywords, thereby improving the accessibility and relevance of the content analyzed.

Implementation of comprehensive data analysis and visualization

The comprehensive analysis of international news commentary data (as shown in Fig. 11) mainly included the following aspects:

Proportional distribution of categories: This analysis showed the distribution of different categories in the international news commentary data, indicating the proportions of different themes or topics in the dataset.

International news commentary publication time distribution: This analysis illustrated the distribution of publication times for international news commentary, providing insights into the temporal patterns of content generation.

International news commentary publishing source distribution: This analysis illustrated the distribution of different publishing sources for international news commentary, indicating which platforms or media outlets were prevalent in the dataset.

Distribution of the number of international news commentary: This analysis examined the distribution of international news commentary written by individual critics, providing an overview of their writing frequency and contribution.

Implementation of visualized analysis for article details

The data visualization of the international news commentary content details mainly included the display of the sentiment trend in articles, the display of the abstract, the visualization of the extracted keywords, the display of the topic analysis, and the display of similar article recommendations. This part involved retrieving the article’s unique identifier (ID), which corresponded to the detail page when a user clicked on it. Based on this ID, the system visualized the content of the specific article (as shown in Fig. 12).