Web content topic modeling using LDA and HTML tags

Key concepts

The Latent Semantic Indexing (LSI) model was the first statistical model for grouping co-occurrence terms in documents. Hofmann (1999) proposed a Probabilistic Latent Semantic Indexing (pLSI), an offshoot of LSA, to improve topic modeling (Vayansky & Kumar, 2020). The pLSI combines latent semantics with a probability model to enhance the detection of the co-occurrence of topics. Blei, Ng & Jordan (2003) proposed the Latent Dirichlet Allocation (LDA), which extended the pLSI by adding Dirichlet priors on topic distributions (Alkhodair et al., 2018; Hajjem & Latiri, 2017). Several studies proposed topic models that add constraints on the traditional LDA to generate modified models called LDA variants (Chen, Thomas & Hassan, 2015). In addition to Semantic Indexing and Dirichlet Indexing, researchers use a few other computational methods for topic modeling, such as the independent component analysis (ICA) and principal component analysis (PCA). These methods exhibit efficiency in topic modeling but are still outweighed by the performance of the LDA model.

The LDA and its variants are prominent models in the field of topic modeling due to their outstanding performance. Several LDA variants exist for specific tasks, such as modeling tweets (Wang & Maeda, 2019), web services, authors (Alrabaee, Debbabi & Wang, 2020), news (Yang, Li & Zhao, 2019), social networks (Liu & Hu, 2021), and recommendations (Tang, Zhang & Niu, 2020). Although these models perform well when applied to the tasks for which they are designed, they notably underperform when applied to other tasks, such as modeling web content data.

Generic variant models of the LDA have also been proposed and applied in various tasks and applications. A few studies address the application of the LDA or its variant models on web content data (Gu et al., 2016; Liu & Forss, 2015b). Those use various techniques such as the Correlated Topic Model (CTM), the Dirichlet Multinomial Regression (DMR), Hierarchical LDA (HLDA), the Pseudo-document based Topic Model (PTM), and the Supervised Latent Dirichlet Allocation (sLDA). Our study uses these models for benchmarking due to their solid baselines and wide use. Table S1 summarizes the characteristics and limitations of these benchmark models.

It is worth mentioning that several other models, such as BERTopic and word-embedding models, are somewhat related to the benchmark models of this study. However, several key factors support excluding them from this study. The BERTopic model relies on pre-trained Bidirectional Encoder Representations from Transformers (BERT) and employs a unique strategy for word representation. This study focuses on modelling topic distribution with LDA and its variants, whereas BERTopic captures contextual semantics. The fundamental differences between mathematical foundations and topic representation methods preclude them from our experiments. Moreover, despite their effectiveness in clustering words, word embeddings are not explicitly designed for topic identification. This study seeks to propose a novel topic model for the analysis of coherent web content while highlighting the limitations of established benchmark models. By adhering to our research objectives, we provide valuable insight into the need for innovative topic modelling techniques in the field of web mining.

Related work

Web mining is discovering and extracting information and knowledge from website documents using data mining methods and techniques (Etzioni, 1996). It is an integrated field involving a few research areas, such as informatics, statistics, data mining, and computational linguistics. Although data mining research started more than two decades ago, it became more significant because of the dramatic growth of the availability of information resources on the web (Berardi et al., 2015; Kosala & Blockeel, 2000). The state-of-the-art decomposes web mining into three categories: web content mining, web structure mining, and web usage mining (Anami, Wadawadagi & Pagi, 2014; Kosala & Blockeel, 2000).

Web content mining focuses on the contents of a website itself. Websites contain several data types; text, hypertext, image, video, and audio, which together constitute a topic or more inside the webpage. Literature has utilized topic modeling on web content data for various applications, such as clustering (Alharbi, Hijji & Aljaedi, 2021; Zhao et al., 2018), classifying (Gu et al., 2016), detecting and filtering (Asdaghi, Soleimani & Zahedi, 2020; Liu & Forss, 2015b), and ranking web pages (Lee & Cho, 2021).

Several studies have proven the benefit of topic modeling in classifying and clustering web pages and websites (Gu et al., 2016). The training of a topic model on a large corpus of web pages is a standard method for web classification using topic modelling. This entails treating each web page as a bag of words in which each word is handled as an individual vector. A recent study has proven the impact of topic modeling to cluster web pages using LSA and PLSA (Alghamdi & Selamat, 2015). Other studies have used the LDA topic model to enhance the classification of harmful and inappropriate web pages using their extracted content (Liu & Forss, 2015a; Liu & Forss, 2015b). Similar studies detect spam webpages using the LDA topic models (Asdaghi, Soleimani & Zahedi, 2020; Wan et al., 2015). These studies only utilize the LDA topic model in their applications. Chen and Zhou have modified the LDA model to include user-related tags in order to enhance web clustering (Chen & Zhou, 2014). The authors (Sayadi, Bui & Bui, 2015) combined the LDA topic modeling with the random forest classifier to perform multilayer soft web classification. Other studies address clustering non-English webpages using the LDA (Alharbi, Hijji & Aljaedi, 2021). A recent study has utilized the LDA and word2vec to classify webpages based on their ranking (Lee & Cho, 2021).

Topic modeling is also utilized to discover trending topics in a specific subject area or speciality. Several recent studies have used the LDA topic model to discover the popularity of topics in a specific field, such as the IoT and Industry 4.0 (Shah, Naqvi & Jeong, 2021), terms of service (Sundareswara et al., 2021), and marketing travelling blogs (Shan-shan, Guo-ming & Xi-tong, 2013). Although topic modeling does not directly detect fake news, it has been utilized to identify the content of news articles and, therefore, detect potential sources of fake news (Kim, Park & Mariani, 2023; Xu et al., 2019).

Topic modeling also identifies the most common topics that are being discussed in Question and Answer (Q&A), rating, and review websites. Shah, Naqvi & Jeong (2021) used dynamic topic modeling to identify dominant topics related to COVID-19 health issues. A similar approach was applied using the LDA model to detect and summarize topics of gaming development (Kamienski & Bezemer, 2021), development of security vulnerability (Le et al., 2021), and startup entrepreneurship (Nurhas et al., 2021).

A common drawback of the abovementioned studies is that they have only utilized topic modeling based on textual documents that have been extracted from web pages. However, they have not considered the different structures of web content data. Neglecting the unique structure of web content leads to missing the otherwise topics and, therefore, low topic quality, as the first experiment demonstrates in Section 5.1.

A few recent studies, however, have addressed the unique structure of web content when applying topic modeling. Yang et al., (2020) proposed a distribution topic model, referred to as Named Entity Topic Model (NETM), to extract web content popularity growth factors. Although the NETM has achieved higher accuracy compared to the LDA model, it has not addressed the HTML structure of the webpage content. Another interesting and recent study has been proposed by Zhao et al. (2021), where the authors proposed a topic-graph probabilistic personalization model for web search based on the LDA model. The model includes the relevancy of the webpage based on the previous probabilities of retrieving a relevant or non-relevant webpage. The model has proven its effectiveness compared to the benchmark models; however, it neglects the HTML structure of the webpage. Neglecting the unique structure, ss the first experiment in Section 5.1. indicates, causes to missing produce a low topic quality.

Problem formalization and notations

This section describes the problem formally and addresses the used notations in the following subsections. The definition of the problem is as follows:

Consider a collection of webpages (1)${\displaystyle Dataset\left(D\right)=\left\{W{P}_0,W{P}_1,\dots ,W{P}_{p-1}\right\}}$

where WP_i is the i-th webpage of a dataset collection D, and p is the number of web pages in the collection. Each of these web pages is composed of HTML tags (2)${\displaystyle Webpage\left(WP\right)=\left\{T{G}_0,T{G}_1,\dots ,T{G}_{t-1}\right\}}$

where TG_i is the i-th HTML tag of a webpage WP and t is the number of HTML tags in the webpage.

An HTML tag topic in a given webpage is the distribution of all words relating to this webpage and can be represented as, (3)${\displaystyle {\theta }_{tg}={\left\{{{\theta }_{tg}}_i\right\}}_{i\in 1_{tg}}\sim Dir\left(\cdot |{\alpha }_{wp}\right).}$

Taking a sports news webpage as an example, which includes many HTML tags, each may include different sub-topics. However, some tags in the webpage may include some other recent news and the side of the webpage for users to read. This news can be related to sports as well as other topics such as political news, health news, and many others. In this case, taking the webpage as one piece could give low topic coherence and, therefore, generate low topic quality. The webpage topic modeling problem aims at finding topics that occur on a webpage and ensures that the generated topics are semantically coherent. The final goal of this article is to learn coherence topics in web content data for a given webpage.

Before introducing the generative process and the mathematical explanation of the model, Table 1 tabulates the used notations.

The generative process

As the LDA topic model, the HTM model is based on a generative statistical model, and it uses latent factors to capture the semantic similarities of words and documents. The generative process of the HTM model is as follows. Firstly, we need to specify the optimal number of topics represented by (K). Then randomly choose a distribution over topics (a multinomial of length K). A specific webpage (WP) is modelled as a sequence of words W = (W₁, …, W_ℓ) of length ℓ ∼Poisson(ξ), where ξ is pre-specified. For this webpage WP, a K-dimensional probability vector θ with non-negative coordinates summed to one is used to model the topic mixture. Three probability distributions are assumed to be multinomial distributions: p(z—wp), p(z—tg), and p(w—z). Therefore, the topic distributions in all web pages share the common Dirichlet prior α, and the word distributions of topics share the common Dirichlet prior β. Given α and β as the parameters for webpage WP, parameter θ_wp of a multinomial distribution over K topics is constructed from Dirichlet distribution Dir(θ_wp— α). The alpha is initiated parameter for the webpage (theta_{wp}), and for the simplicity of the distribution, the same parameter is then used for the tags within a specific webpage. Notice that the vector size of alpha will differ within each webpage based on its tags. Therefore, parameter θ_tg of a multinomial distribution over K topics is constructed from Dirichlet distribution Dir(θ_tg— α). For topic t, parameter φ_t of a multinomial distribution over the set of words in the vocabulary (V) is derived from Dirichlet distribution Dir(φ_t— β). The Dirichlet distribution is a convenient choice as a prior and can simplify the statistical inference in the HTM model. The likelihood is multiplied through all the web pages and maximized with the technique of variational inference for the estimation of α and β.

A summary of the generative process for a set of web pages is as follows:

For each topic $t\in \left\{1,\dots ,T\right\}$

Generate ${\varphi }_t={\left\{{\varphi }_{tw}\right\}}_{w=1}^V\sim Dir\left(\cdot |\beta \right)$

For each web page $WP\in \left\{1,\dots ,N\right\}$

Generate ${\theta }_{wp}={\left\{{{\theta }_{wp}}_i\right\}}_{i\in 1_{wp}}\sim Dir\left(\cdot |{\alpha }_{wp}\right)$

For each HTML tag tg in the webpage WP

Generate ${\theta }_{tg}={\left\{{{\theta }_{tg}}_i\right\}}_{i\in 1_{tg}}\sim Dir\left(\cdot |{\alpha }_{wp}\right)$

For each word w in the HTML tag tg

Generate $z_{tgn}\in {\left\{{{\theta }_{tg}}_i\right\}}_{i=1}\sim Multinominal\left(\cdot |{\theta }_{tg}\right)$

Generate $w_{tgn}\in \left\{1,\dots ,V\right\}\sim Multinominal\left(\cdot |{\varphi }_{z_{tgn}}\right)$

The following snapshots elaborate more on how a webpage will transform from metadata to preprocessed data ready to be inserted into the HTM topic model. Figure S3 shows a snapshot from webpage metadata. Notice that the metadata contains various HTML tags, such as <div>, <span>, <svg>, <p>, and <h4>. The HTM assumes that the webpage is a distribution of tags; therefore, each tag will be preprocessed separately. Notice that only visible text tags will be used, and their textual content will be extracted, as explained in Fig. S3. Figure S4 shows a snapshot of the preprocessed textual data, where each tag is represented separately as a list. The following section represents these steps mathematically and elaborates on the role of the tags of a webpage using a plate notation of the HTM topic model.

Mathematical model

The HTM topic model, like the LDA model, is based on the probability distribution model. Figure 1 shows the graphical model of the HTM topic model, referred to as the plate notation graph.

The model infers the distribution of the hidden variables by using the joint probability distribution. This inference aims to approximate the posterior $\rho \left(\beta ,\theta ,\mathrm{Z}\left|W\right.\right)$ with the distribution $q\left(\beta ,\theta ,\mathrm{Z}\right)$ using the variance inference, simplifying the model analysis. Figure S5 illustrates the inner plate representing the probability distribution of words per topic to simplify the model.

In this sub-graph, β acts as a global variable, while $\left.\mathrm{Z}\right|W$ acts as a local variable for each word in the corpus. This part is inherited as it is from the LDA model. The mathematical definition of this plate is as follows: (4)${\displaystyle p\left(\beta ,Z_{1:n},W_{1:n}\right)=p\left(\beta \right)\prod _{i=1}^{n}p\left(Z_i|\beta \right)p{W}_i|Z_i,\beta .}$

This sub-graph is associated with the per-HTML tag topic proportion variable and the word distribution for each topic. Figure S6 illustrates this association of the model.

The parameter α of the Dirichlet distribution models the topic distribution variable θ_tg per HTML tag, while the parameter φ of the multinomial distribution models each specific associated topic Z_i. This association is defined as: (5)${\displaystyle q\left(\beta ,{\theta }_{tg},Z\right)=\prod _{k=1}^{K}q\left({\beta }_k|{\phi }_k\right)\prod _{tg=1}^{TG}q\left({\theta }_{tg}|{\alpha }_{tg}\right)\prod _{n=1}^{N}q\left(Z_{tg,n}|{\phi }_{tg,n}\right).}$ Once the HTM model processes HTML tags, the model then applies a similar step on all the given web pages. The following equation describes the process as follows: (6)${\displaystyle q\left(\beta ,{\theta }_{wp},{\theta }_{tg},Z\right)=\prod _{t=1}^{T}q\left({\beta }_t|{\phi }_t\right)\prod _{wp=1}^{WP}q\left({\theta }_{wp}|{\alpha }_{wp}\right)\prod _{tg=1}^{TG}q\left({\theta }_{wp,tg}|{\alpha }_{wp,tg}\right)\prod _{n=1}^{N}q\left(Z_{tg,n}|{\phi }_{tg,n}\right).}$

Once the HTM model processes all the web pages, the model then updates the parameters of the topics $\left(\phi and\alpha \right)$. The model updates these parameters after each iteration. In each iteration, as the α_tw value increases, the chance of selecting the word W from the HTML tag TG in topic T also increases.

Performance evaluation

Literature in the field of topic modeling uses perplexity measure, topic coherence measure, or both measurements. Several recent studies have argued that perplexity is less correlated to human interpretability and understandability (Li et al., 2016; Röder, Both & Hinneburg, 2015; Zuo et al., 2016) and does not address the goal of exploratory research of topic modeling. Thus perplexity is no longer a general way of evaluating topic models (Zuo et al., 2016). On the contrary, topic coherence metrics have proven to correlate with human judgments and interpretability (Kim, Park & Lee, 2020; Law et al., 2017; Röder, Both & Hinneburg, 2015; Syed & Spruit, 2017). Taken these considerations together, this study evaluates the benchmark topic models and the HTM topic models using topic coherence metrics.

Topic coherence can calculate and measure the consistency and quality of each individual topic with reference to the semantic similarity between the words in the topic or how many words of each individual topic occur within the same set of documents (Li et al., 2016; Mimno et al., 2011). The authors (Mimno et al., 2011) introduced the topic coherence metric, which strongly correlates with human judgements in evaluating topic quality (Chen & Zhou, 2014). Topic coherence indicates the quality of the model and how accurate the terms are. The higher the topic coherence score, the more efficient the model. Topic coherence metrics often rely on the method of sliding window. Sliding refers to the step-by-step movement of a window with a defined size over a text corpus. It permits the metric to recognize local patterns in the text and calculate coherence scores for each window. The window concept is used to determine the scope of the coherence measure for a specific topic. It refers to the subset of documents or terms utilized to calculate the topic’s coherence score. The window size determines the number of words or terms in each window. For instance, a window size of 10 indicates that each window includes 10 consecutive text words. The commonly used coherence calculations in the literature are C_UMass (Mimno et al., 2011), C_UCI (Newman et al., 2010b), C_NPMI (Bouma, 2009b), and C_V (Lau, Newman & Baldwin, 2014; Syed & Spruit, 2017). The following experiments are evaluated by using these metrics.

Dataset

This section describes the datasets used in the experiment. This study investigates differences in the performance of topic models on conventional document/article data and webpage content data. We use two datasets, one for each type of data, to achieve this aim. The following subsections (1 and 2) introduce each dataset and describe its features and sources. Both datasets are then preprocessed for evaluation, as subsection 3 illustrates.

Benchmark models evaluation

We used the Python programming language with the Anaconda distribution platform’s help to evaluate CTM, DMR, HLDA, LDA, PTM, and sLDA based on both CD-based and WC-based datasets. We used the topic coherence metrics to evaluate a topic model’s quality and performance. This section evaluates the topic models using C_UMass, C_UCI, C_NPMI, and C_V metrics. Using these metrics, the performance of the models for each dataset and given the number of topics K ∈{1…,100} will then be compared in the following subsections. Section 5.1.4 then discusses the results of compassion.

First experiment’s results and discussion

The results, as depicted in Figs. 2–5 (where A is the performance on CD-based dataset and B is the performance on WC-based dataset), showed that the overall performance of these topic models on conventional document data outweighs their performance on documents with web content-based data across the number of topics. These results indicate that the benchmark models failed to capture the coherence of topics on webpage content data. The difference in performance was significant for some topic models, such as the PTM and CTM, and it was inconsiderable for other models, such as the LDA. This result answers the first research question of this study. The C_UCI and C_NPMI topic coherence appear to prefer fewer topics on conventional documents data. In contrast, the C_V topic coherence favours a higher number of topics for all models except the HLDA model. The results suggest that the optimal number of topics may vary depending on the evaluation metric used and the type of dataset being analyzed.

An interesting observation was made regarding the LDA model’s performance on both datasets. Although it performed the worst for the CD-based dataset, it performed the best for WC-based datasets, achieving the highest topic coherence score in comparison to other models. This indicates that the LDA model is the most stable model among the benchmark topic models based on the C_UMass, C_UCI, and C_NPMI metrics. This result answers the second research question of this study.

Although the CTM model consistently achieved the highest topic coherence score given different numbers of topics for CD-based datasets, it witnessed the largest failure among the benchmark topic models when evaluated using the CUCI metric on web content data. Similarly, when evaluating the C_NPMI metric, the sLDA and DMR models achieved the highest topic coherence scores given different numbers of topics for CD-based datasets, whilst the PTM model scored the lowest. It is worth mentioning that the sLDA scores are similar to the DMR scores given all metrics and in both datasets, which is likely due to their methodological similarities (Mimno & McCallum, 2008). Although the authors (Mimno & McCallum, 2008) emphasize the difference between their proposed DMR model and the sLDA model, both models perform comparably similarly in our study.

Another interesting observation was that the DMR and sLDA models performed the best on both CD-based and WC-based datasets based on the C_V metric. However, it is worth noting that although all benchmark models perform better on WC-based datasets than CD-based datasets, the increase in topic coherence score is modest, indicating only a slight enhancement.

This finding is somewhat surprising since one might expect models to perform better on web content data, given this data’s relatively free-form and less-structured nature. However, the results indicate that neglecting the web content data structure results in low-coherence topics, thus generating low-quality and semantic topics from web content data.

HTM evaluation

This study presents an analysis of the topic modeling of web content data using the LDA and HTM models. The goal of the HTM topic model is to improve the performance of the LDA model, which is a widely used topic modeling algorithm. The study evaluates both models using four coherence metrics: C_UMass, C_UCI, C_NPMI, and C_V, and compares their performance for different numbers of topics (K ∈ {1…,100}). We used the Python programming language with the help of the Gensim library to perform data preprocessing and implement the LDA model and the HTM topic model.

Second experiment’s results and discussion

The results, as depicted in Fig. 6 (where A is based on C_V metric, B is based on C_UMass metric, C is based on C_NPMI metric, and D is based on C_UCI metric), indicate that the HTM model outperforms the LDA model in terms of overall performance. The C_UMass coherence score shows that the HTM model has a significantly better performance than the LDA model, with a steady value over the number of topics, while the LDA model’s value decreases as the number of topics increases. The improvement of the HTM model was slightly more than 89% of the LDA using the C_UMass metric. This is due to the UMass metric, which considers preceding and succeeding terms in the list and allows the HTM model to relate topics within each HTML tag and among tags on each webpage. Similarly, the C_V coherence score shows that the HTM model outperforms the LDA model for any number of topics, with a C_V value of ≥ 0.9 when the number of topics exceeded 4. The improvement of the HTM model was slightly more than 36% of the LDA using the C_V metric. These results indicate that the HTM model learns better web content topics than the LDA model, which may enhance human interpretability, as some previous studies argued (Mimno et al., 2011; Newman et al., 2010a; Röder, Both & Hinneburg, 2015). These results indicate that considering HTML tags when applying topic modeling on web content data increases the quality of generated topics, which answers the third research question of this study.

The C_UCI and C_NPMI coherence scores show that the HTM model performs slightly better than the LDA model when the number of topics is high. However, when the number of topics is low, the HTM model significantly outperforms the LDA model. Another improvement of the HTM model was recorded using the C_NPMI metric with about more than 26%. This is due to the fact that the HTM model considers the indirect coherence between related terms of each tag, which enhances its performance in learning coherence topics.

The results show that the enhancement made by the HTM model was vast based on C_UMass and CV metrics yet slight based on CUCI and C_NPMI metrics. These phenomena suggest that the HTM model made a significant enhancement of the LDA model in generating topics that are semantically related and coherent in terms of the overall corpus. However, the HTM model was similar to the LDA model in terms of generating a strong association within each topic. This result is due to several reasons related to the nature of each coherence metric.

C_UMass coherence measures the degree of semantic coherence between the words in a topic by comparing the observed co-occurrence of words within the topic to their expected co-occurrence in a reference corpus. C_V coherence measures the degree of coherence based on the exclusivity of the top words in a topic. Both UMass and V coherence metrics often indicate how well the topic model captures global patterns in the corpus (Lau & Baldwin, 2016).

C_UCI coherence and C_NPMI coherence, on the other hand, measure the degree of association between word pairs within a topic. C_UCI coherence calculates the pointwise mutual information (PMI) between the words in the topic, while C_NPMI normalizes PMI by dividing it by the negative logarithm of the probability of the word pair occurring together by chance. Both C_UCI and C_NPMI coherence metrics are based on the PMI, which often indicates how well the topic model captures local patterns in the corpus (Bouma, 2009a).

In conclusion, this study demonstrates that the HTM model outperforms the LDA model in the topic modeling of web content data and provides evidence of the benefits of the HTM model in learning coherent topics. These findings are useful for researchers in the field of web content analysis and topic modeling. These finding also answers the fourth research question of this study. A summary of these coherence metrics results is represented by Fig. S7.