LDAPrototype: a model selection algorithm to improve reliability of latent Dirichlet allocation

Topic models

Propably the most well-known topic model is the classical LDA (Blei, Ng & Jordan, 2003), which we explain in ‘Latent Dirichlet allocation’ in detail. It can be seen as a refinement of probabilistic latent semantic analysis/indexing (Hofmann, 1999), which itself is a refinement of the latent semantic indexing (Deerwester et al., 1990). Consequently, the structural topic model (Roberts et al., 2013) can again be seen as a generalization of LDA. Concretely, it allows correlations between topics as well as the possibility to include metadata in the modeling of topic and word distributions.

Along with the popularity of neural networks, neural topic models or topic models based on word embeddings (Mikolov et al., 2013, cf.) such as the embedded topic model (Dieng, Ruiz & Blei, 2020) emerged. However, due to the rapid innovation cycles, these models have found little popularity and application. Moreover, their architecture prevents them from being directly comparable with traditional probabilistic topic models, meaning that evaluation measures developed for probabilistic topic models are not well suited to determine quality of such models (Hoyle et al., 2022).

In recent years, following the development of the transformer architecture (Vaswani et al., 2017) (not only) for topic models, the focus shifted to enhancements of models based on language models. Examples of such models are GRETEL (Xie et al., 2022), STELLAR (Eklund & Forsman, 2022) and the quite popular BERTopic (Grootendorst, 2022), the latter of which can be used extremely flexibly due to its modular combination of language models, clustering techniques, and scoring functions.

However, these novel transformer-based language models, which can be considered as superposed neural topic models (Lim & Lauw, 2023), allow more than just the possibility of being used directly for modeling. Rather, language models offer possibilities (cf. Sia, Dalmia & Mielke, 2020; Mu et al., 2024) to use them as a complementary evaluation option (Stammbach et al., 2023).

In fact, a major problem in topic model research is the lack of a satisfying task-independent evaluation of topic models. In this regard, instead of using (just) likelihood-based measures (Chang et al., 2009), topic models should always be compared using task-based measures (cf. Grimmer & Stewart, 2013; Doogan & Buntine, 2021). In addition, an automated evaluation often does not provide trustworthy results. Hoyle et al. (2021) show that coherence-based measures sometimes provide incoherent results and they argue that automated measures require ongoing human re-assessment in order to remain reliable. Overall, this shows that there is not a one-fits-it-all solution for all areas where topic models are used (cf. Ethayarajh & Jurafsky, 2020).

In practice, it can be seen that LDA is still very popular and frequently used in state-of-the-art studies (cf. Batool & Byun, 2024; Muthusami et al., 2024), so that further developments of this methodology, which do not change the model assumptions and modeling itself, continue to be highly relevant.

Latent Dirichlet allocation

The method we propose is based on the LDA (Blei, Ng & Jordan, 2003) estimated by a collapsed Gibbs sampler (Griffiths & Steyvers, 2004). The LDA assumes distributions of latent topics for each text. If K denotes the total number of modeled topics, the set of topics is given by T = {T₁, …, T_K}. We define $W_n^{\left(m\right)}$ as a single token at position n in text m. The set of possible tokens is given by the vocabulary W = {W₁, …, W_V} with V = |W|, the vocabulary size. Then, let ${\displaystyle {\mathit{D}}^{\left(m\right)}=\left(W_1^{\left(m\right)},\dots ,W_{N^{\left(m\right)}}^{\left(m\right)}\right),m=1,\dots ,M,W_n^{\left(m\right)}\in \mathit{W},n=1,\dots ,N^{\left(m\right)}}$ be text (or document) m of a corpus consisting of M texts, each text of length N^(m). Topics are referred to as $T_n^{\left(m\right)}$ for the topic assignment of token $W_n^{\left(m\right)}$. Then, analogously the topic assignments of every text m are given by ${\displaystyle {\mathit{T}}^{\left(m\right)}=\left(T_1^{\left(m\right)},\dots ,T_{N^{\left(m\right)}}^{\left(m\right)}\right),m=1,\dots ,M,T_n^{\left(m\right)}\in \mathit{T},n=1,\dots ,N^{\left(m\right)}.}$ when $n_k^{\left(mv\right)},$k =1 , …, K, v =1 , …, V describes the number of assignments of word v in text m to topic k, we can define the cumulative count of word v in topic k over all documents by $n_k^{\left(•v\right)}$ and, analogously, the cumulative count of topic k over all words in document m by $n_k^{\left(m•\right)}$, while $n_k^{\left(••\right)}$ indicates the total count of assignments to topic k. Then, let ${\displaystyle {\mathit{w}}_k={\left(n_k^{\left(•1\right)},\dots ,n_k^{\left(•V\right)}\right)}^T\in {\mathbb{N}}_0^V,k=1,\dots ,K}$ denote the vector of word counts for topic k.

Using these definitions, the underlying probability model (Griffiths & Steyvers, 2004) can be written as ${\displaystyle W_n^{\left(m\right)}\mid T_n^{\left(m\right)},{\varphi }_k\sim \text{Discrete}\left({\varphi }_k\right),{\varphi }_k\sim \text{Dirichlet}\left(\eta \right),T_n^{\left(m\right)}\mid {\theta }_m\sim \text{Discrete}\left({\theta }_m\right),{\theta }_m\sim \text{Dirichlet}\left(\alpha \right).}$ For a given parameter set {K, α, η}, LDA assigns one of the K topics to each token. Here K denotes the number of topics and α, η are parameters of a Dirichlet distribution defining the type of mixture of topics in every text and the type of mixture of words in every topic. Higher values for α lead to a more heterogeneous mixture of topics whereas lower values are more likely to produce less but more dominant topics per text. Analogously, η controls the mixture of words in topics. Although the LDA permits α and η to be vector valued (Blei, Ng & Jordan, 2003), they are usually chosen symmetric because typically the user has no a-priori information about the topic distributions θ and word distributions ϕ.

Topic distributions per text θ_m = (θ_m,1, …, θ_m,K)^T ∈ (0, 1)^K and word distributions per topic ϕ_k = (ϕ_k,1, …, ϕ_k,V)^T ∈ (0, 1)^V can be estimated through the collapsed Gibbs sampler procedure (Griffiths & Steyvers, 2004) by ${\displaystyle {\hat{\theta }}_{m,k}=\frac{n_k^{\left(m•\right)}+\alpha }{N^{\left(m\right)}+K\alpha }\text{and}{\hat{\varphi }}_{k,v}=\frac{n_k^{\left(•v\right)}+\eta }{n_k^{\left(••\right)}+V\eta }.}$

Methods and modifications of LDA to address the random initialization instability

Inferring LDA using Gibbs sampling is sensitive to the initial assignments, that are often chosen as random, and the reassignment is based on the conditional distributions, which leads to different results in multiple LDA runs for fixed parameters. This instability of LDA leads to a lack of reliability of the modeling results. This fact is rarely discussed in applications (Agrawal, Fu & Menzies, 2018), although several approaches have been proposed to encounter this problem as discussed in the following.

LDAPrototype: a new selection algorithm for LDA models

We propose the novel selection algorithm LDAPrototype to improve the reliability of LDA results. For this, we define the reliability score rs for a set of L LDAs based on their mean similarities $\overline{s_1},\dots ,\overline{s_L}$ as (1)${\displaystyle rs\left(\overline{s_1},\dots ,\overline{s_L}\right):=\frac{1}{L}\sum _{l=1}^L\overline{s_l},\overline{s_l}\in \left[0,1\right],}$ where the mean similarities $\overline{s_l},$l =1 , …, L have to be determined by a model similarity measure yet to be defined. Our selection algorithm aims for maximizing this reliability score rs, which measures how reproducible LDA results are in a given setting with fixed parameters. Our algorithm is motivated by an increase in reliability (cf. ‘Introduction’) and selects from a set of LDA models the model that is most similar on average to all other runs. The approach is similar to the choice of the median in the one-dimensional space. In the multidimensional space, this choice is called medoid and differs from a centroid in the sense that it is not obtained by model averaging, but by model selection. There are methods of model averaging for LDA (cf. ‘Methods and modifications of LDA to address the random initialization instability’), but these have the disadvantage that properties of a single run, such as the assignments of individual tokens to topics, are lost. The proposed selection algorithm preserves this information because it does not influence the modeling itself. The LDAPrototype procedure selects one single model from the set of candidate models.

Figure 1 shows schematically the determination of the prototype, and the corresponding procedure is presented in Algorithm 1 . The main idea of the method is to calculate all pairwise similarities of the R LDAs and determine the model that maximizes this similarity. Besides the candidate set of LDAs {LDA₁, …, LDA_R}, a similarity measure for LDA models is needed to obtain the symmetric model similarity scores s_ij = s_ji in Fig. 1 and Algorithm 1 . For this, in ‘S-CLOP: a new similarity measure for LDA models’, we propose a novel tailored model similarity measure called S-CLOP, which in turn requires the choice of a topic similarity measure. In the following section ‘Thresholded version of the Jaccard coefficient: a similarity measure for topics’, we propose a default measure for computing these topic similarities. Through these definitions, we are then able to determine the prototype with respect to Fig. 1. In Appendix A, we discuss other popular measures for computing topic similarities, which we also compare to our proposed Jaccard coefficient from Eq. (3) in ‘Comparison of the implemented similarity measures’.

 
_______________________ 
Algorithm 1 Selecting the medoid of a number of LDA runs______________________ 
Require: A set of R LDA models 
Ensure: The LDA with maximal mean pairwise similarity to all other LDA 
     runs 
  1:  for i = 1 to R − 1 do 
 2:       for j = i + 1 to R do 
 3:            Calculate pairwise similarity of LDAs ldai and ldaj using a similarity 
     measure sim: sij = sim(ldai,ldaj) 
  4:       end for 
 5:  end for 
 6:  Determine ˉ si =   1 __ 
R−1 ∑ 
   j⁄=i sij, i = 1,...,R 
 7:  Determine proto = ldaopt with opt = arg max 
 i∈{1,...,R} ˉ si 
  8:  return proto_________________________________________________________________________________    

Thresholded version of the Jaccard coefficient: a similarity measure for topics

A similarity between two topics can be calculated based on the corresponding vectors of counts or set of words. We build on the well established Jaccard coefficient (Jaccard, 1912) and introduce a more robust thresholded version. Its general form is given by (2)${\displaystyle \text{Jaccard}\left(A,B\right)=\frac{|A\cap B|}{|A\cup B|},}$ where A, B are sets of words.

Suppose we have a text corpus and we estimate a topic model with LDA, with parameters α, η and K topics. This is done R times independently leading to a set of N = RK topics in total. Then, referring to ‘Latent dirichlet allocation’, ${\displaystyle {\mathit{w}}^{\left(r\right)}=\left({\mathit{w}}_1^{\left(r\right)},\dots ,{\mathit{w}}_K^{\left(r\right)}\right)\in {\mathbb{N}}_0^{V\times K},r=1,\dots ,R}$ denotes the matrix of word counts per topic in the r-th replication. Then, for a given lower bound c = (c₁, …, c_N) and for two topics (i, j) represented by their word count vectors ${\displaystyle {\mathit{w}}_i,{\mathit{w}}_j\in \left\{{\mathit{w}}_1^{\left(1\right)},\dots ,{\mathit{w}}_K^{\left(1\right)},{\mathbf{w}}_1^{\left(2\right)},\dots ,{\mathit{w}}_K^{\left(2\right)},\dots ,{\mathit{w}}_1^{\left(R\right)},\dots ,{\mathit{w}}_K^{\left(R\right)}\right\}}$ our thresholded version of the Jaccard coefficient is calculated by (3)${\displaystyle \text{tJacc}\left({\mathit{w}}_i,{\mathbf{w}}_j\right)=\frac{\sum _{v=1}^V{1}_{\left\{n_i^{\left(•v\right)}>c_i\wedge n_j^{\left(•v\right)}>c_j\right\}}}{\sum _{v=1}^V{1}_{\left\{n_i^{\left(•v\right)}>c_i\vee n_j^{\left(•v\right)}>c_j\right\}}}.}$ Reasonable choices for the threshold vector c = (c₁, …, c_N)^T ∈ ℕ^N are an equal absolute lower bound c_abs for all words, or a topic-specific relative lower bound c_rel (see also Table 1). A combination of both can be defined by ${\displaystyle c_l=max\left\{c_{\text{abs}},c_{\text{rel}}{n}_l^{••}\right\},}$ where l = 1, …, K, …, 2K, …, RK = N and c_abs ∈ ℕ₀, c_rel ∈ [0, 1]. To ensure that always enough words per topic are taken, the similarity measure is additionally implemented with a parameter that controls that at least a fixed user-defined number of the most frequent words assigned to the topic are considered.

The interpretation of this tJacc coefficient is the following. It is defined as the ratio of the numbers of the intersection and the union of the words of two topics, but a word is only considered if the number of its occurrences in the texts exceeds the topic-specific threshold. In other words, we first restrict ourselves to the most relevant words per topic with respect to the number of assignments, to get rid of heavy tailed word lists. Then the resulting subsets of words are used to measure similarity of topics using the standard Jaccard coefficient. To be clear: even for a choice of c_rel = 0, the similarities of the topics would not (necessarily) all equal 1. That is because for the calculation not the estimators ${\hat{\varphi }}_{k,v}$, but the numbers of actual assignments $n_i^{\left(•v\right)}$ are used.

We demonstrate how the measure is calculated with a small toy example.

In Table 1, for eleven selected words the counts of assignments over all articles for the two topics w₁ and w₂ are given. We use the relative lower bound with c_rel = 1/500. In the analysis presented in ‘Results’, we mainly also use c_rel = 1/500, which leads to around 100 important words per topic in the setting of the usatoday dataset. The last two columns indicate whether the corresponding word belongs to the thresholded intersection or union, respectively. For example, the word election does not belong to the intersection because its count is below the topic-specific (relative) threshold of at least nine assignments to the topic belonging to w₁. The ratio of the number of entries in the third and the fourth column results in the similarity $\text{tJacc}\left({\mathit{w}}_1,{\mathit{w}}_2\right)=\frac{5}{10}=0.5$ of the two given topics.

S-CLOP: a new similarity measure for LDA models

We introduce the new similarity measure S-CLOP for comparing different LDA runs consisting of topics that are represented by word count vectors. The pairwise distances are calculated with the tJacc coefficient in Eq. (3). However, the measure can be applied using any appropriate distance respectively similarity measure, see Appendix A. We use S-CLOP as pairwise model similarity measure in the LDAPrototype procedure, see also Fig. 1 and Algorithm 1 .

The general idea of the measure S-CLOP is the following. First, join all considered LDA runs to one overall set, then cluster the topics with subsequent local pruning, and then check how many members of the original different LDA runs are contained in the resulting clusters. Then the deviation from the perfect situation of one representative from each LDA run, i.e. that every topic is present in all LDA runs, is quantified. In our selection algorithm always two LDA models are compared. Two models are very similar, if always one topic of the first model is clustered together with one topic from the other model. High S-CLOP similarity values suggest that many topics can be identified that have a representative in each LDA run.

For the initial clustering step, we use hierarchical clustering with complete linkage (Hastie, Tibshirani & Friedman, 2009 pp. 520–525). According to Zhao, Karypis & Fayyad (2005) average linkage or Ward’s method obtain superior results for the case of text clustering. We also tried single linkage, average linkage and Ward’s method in the present case, and in the package the user is also offered, in principle, the possibility of calculation using these link methods. In fact, for the selection algorithm presented here, there is little difference in which of the linkage methods is used due to the characteristic of combining pairwise comparison with local-optimal pruning. We prefer complete linkage over single or average linkage because it uses the maximum distance between topics to identify clusters. This is consistent with our aim of identifying highly homogeneous groups. Of course, topic similarities must first be transformed to distances to apply hierarchical clustering.

 
_______________________________________________________________________________________________________ 
Algorithm 2 Determining the minimal sum of disparities U∗(g) of a cluster g 
Require: A node of a dendrogram 
Ensure: The minimal possible sum of disparities for this node 
  1:  if is.leaf(node) then 
 2:       return (R − 1)/R 
 3:  else 
 4:       return min{U(node),Recall(node.left) + Recall(node.right)} 
 5:  end if______________________________________________________________________________    

 
__________________________________________________________________________________________ 
Algorithm 3 Finding the optimal set of clusters G∗___________________ 
Require: A dendrogram with a root 
Ensure: A list correponding to the optimal set of clusters G∗, obtained by local 
     pruning of the dendrogram node = root 
  1:  if U(node) == U∗(node) then 
 2:       Add all objects belonging to the cluster corresponding to node as one 
     cluster to list 
  3:  else 
 4:       Recall(node.left) 
  5:       Recall(node.right) 
  6:  end if 
 7:  return list____________________________________________________________________________________    

Using S-CLOP for LDAPrototype

As described in ‘LDAPrototype: a new selection algorithm for LDA models’, the LDAPrototype algorithm (cf. Algorithm 1 ) relies on finding the medoid using a suitable similarity measure for LDA models. For this purpose, we use the S-CLOP measure from ‘S-CLOP: a new similarity measure for LDA models’. In terms of the LDAPrototype selection procedure, to compute the S-CLOP similarities we always compare only two LDA runs, so that the clustering and the measuring of disparities aims at matching pairs of topics from the two different runs. Note that in this special case of comparing just two LDA runs with the same number of topics K, the normalization factor is $U_{\Sigma ,\mathbf{max}}=2\cdot K\cdot \frac{1}{2}=K$. With respect to the definition of U we can simplify Eq. (7) to get (8)${\displaystyle \text{S-CLOP}\left(G\right)=1-\frac{1}{2K}\sum _{g\in G}\left|g\right|\left(\left|\right|g_{|1}|-1|+\left|\right|g_{|2}|-1|\right)\in \left[0,1\right],}$ where g_|1 and g_|2 denote groups of g restricted to topics of the corresponding LDA run. By using the described pruning algorithm (cf. ‘S-CLOP: a new similarity measure for LDA models’) in the two-LDA case, we obtain a set consisting of groups of size two or one. This means that a topic either has a directly similar counterpart topic in the other LDA, or that it forms a cluster itself. A topic always forms a cluster itself if it is either most similar to another topic from the same LDA, or if the complete linkage distance to one of the already found clusters of matched topics is smaller than to all remaining single topics of the other LDA run. Then, the medoid is determined by the run that maximizes the average pairwise S-CLOP value to all other LDA runs from the same set, namely $\overline{s_i},$i =1 , …, R in Fig. 1 and in Algorithm 1 .

Cluster analysis and similarity calculation

In this section, we present an example analysis of a clustering result of four independent LDA runs based on newspaper articles from USA Today. We run the basic LDA four times, such that the number of runs to be compared is R = 4 and the total number of topics to be clustered is N = R⋅K = 4⋅50 = 200. To demonstrate how dissimilar replicated LDA runs can be, we cluster the N = 200 topics from the R = 4 independent runs with K = 50 topics each using the tJacc coefficient from Eq. (3), complete linkage and the new introduced algorithm for pruning. The four runs were selected from 10,000 total runs. In fact, the runs Run1 and Run2 were chosen as the top two models in mean similarity of the 100 prototypes, which means their points lie at the top of the very right (orange) curve in the plot from usatoday in Fig. 4. Their similarity values are 0.902 and 0.898 in the set of prototypes or 0.877 and 0.871 in the original sets, respectively. The model Run3 was chosen as the worst of the 100 prototype models with a similarity value of 0.872 in the set of prototypes and 0.863 in its original set. Run4 was chosen randomly as one of the worst models realizing a mean similarity to all other models in its original set of 0.807.

We apply hierarchical clustering with complete linkage to the 200 topics. The topics are labeled with meaningful titles (words or phrases). These labels were obtained by hand, based on the ranked list of the 20 most important words per topic. For this, the importance of a word v = 1, …, V in topic k = 1, …, K (Chang, 2015) is calculated by (9)${\displaystyle I\left(v,k\right)=\frac{n_k^{\left(•v\right)}}{n_k^{\left(••\right)}}\left[log\left(\frac{n_k^{\left(•v\right)}}{n_k^{\left(••\right)}}+\varepsilon \right)-\frac{1}{K}\sum _{l=1}^{K}log\left(\frac{n_l^{\left(•v\right)}}{n_l^{\left(••\right)}}+\varepsilon \right)\right],}$ where ɛ is a small constant value which ensures numerical computability, which we choose as ɛ = 10⁻⁵. The importance measure is intuitive, because it gives high scores to words which occur often in the present topic, but less often in average in all other topics.

Figure 5 shows a snapshot of the two dendrograms visualizing the result of clustering the 200 topics and of Algorithm 3 for clustering with local pruning. The vertical axes describe the complete linkage distance based on our tJacc coefficient with limit.rel = 0.002. In the lower dendrogram, the topic labels are colored with respect to the LDA run (Run1: grey, Run2: green, Run3: orange, Run4: purple). In the upper dendrogram, topic labels are colored according to the clusters obtained with our proposed pruning algorithm. In addition, every topic label is prefixed by its run number.

Looking at the upper dendrogram, we see that there are a number of clusters with identical labels for topics. This demonstrates that the four LDA runs produce a number of similar topics that are represented by similar word distributions. Examples for such stable topics are Trump vs Clinton Campaign colored yellow and Olympics Medals colored green.

However, there are also considerable differences visible. In the lower dendrogram in Fig. 5, strikingly, there are several topics from Run4, highlighted in red, where no other topic is within a small distance. It is remarkable that Run4 creates such a great number of individual topics, e.g., Video Games, Gender Debate, TV Sports (which includes words for describing television schedules of sport events) and Terrorism. Also, Run4 leads to six explicit stopword topics, which is the maximum number compared to the other runs with four to six stopword topics.

In the upper dendrogram, the color depends on cluster membership. We measure combined stability of these four LDAs by applying the proposed pruning algorithm ( Algorithm 3 ) to the dendrogram. This leads to 61 clusters and a S-CLOP score of 0.83. The normalization factor is given by U_Σ,max = K⋅(R − 1) = 50⋅3 = 150, and the minimization of the sum of disparities yields U^∗ = U_Σ(G^∗) = 25, resulting in S-CLOP = 1 − 25/150 = 0.83. There are seven single topics, one from each of the first three runs and four from Run4. The eleven clusters, which consist of exactly three topics, contain ten times a topic from Run1. Topics from Run2 and Run3 are represented nine times each, whereas only five of the mentioned clusters contain a topic from Run4. This shows that LDA run Run4 strongly differs from the others. There are a lot of cases, where only one topic from this run is missing to obtain perfect topic clusters with exactly one topic from each run.

For comparison to the proposed local pruning algorithm, we once applied an established global criterion. Since 50 topics were originally modeled for each run, it is not reasonable to determine less than 50 clusters. Therefore, we try the global criterion with 50, …, 70 as the target cluster count. The largest similarity value according to S-CLOP based on the resulting clusters is obtained as 0.3 for 59 clusters. However, considering the dendrogram, we do not observe such large differences in the topic structures between the runs to justify such a low similarity. This shows the necessity of the presented local pruning method.

In addition, the dendrograms illustrate that random selection can lead to a poor model regarding interpretability and especially to some kind of an outlier model as Run4. This means that random selection can indeed lead to low reliability.

Increase of reliability

As an improvement for LDA result selection, we recommend to use the introduced approach based on prototyping of replications. It increases mean similarity, which comes along with an increase in reliability. We demonstrate how to determine a prototype LDA run as the most representative run out of a set of runs, based on our novel pruning algorithm. We show that this technique leads to systematically higher LDA similarities, which suggests a higher reliability of LDA findings from such a prototype run. This is indeed the case in comparison to basic LDA replications, but also in comparison to the well known selection criteria perplexity and NPMI. We calculate perplexity (Grün & Hornik, 2011) by (10)${\displaystyle exp\left\{-\frac{\sum _{m=1}^M\sum _{v=1}^V{n}_{•}^{\left(mv\right)}log\left(\sum _{k=1}^K{\hat{\theta }}_{m,k}{\hat{\varphi }}_{k,v}\right)}{\sum _{m=1}^M{N}^{\left(m\right)}}\right\},n_{•}^{\left(mv\right)}=\sum _{k=1}^K{n}_k^{\left(mv\right)}}$ and NPMI with respect to the definition from Röder, Both & Hinneburg (2015). The authors offer a web service (see https://github.com/dice-group/Palmetto) to retrieve NPMI scores using the English Wikipedia as reference corpus. The score indicates a normalized pointwise mutual information for one topic using co-occurrences of the first 10 top words of the topic, determined using the score in Eq. (9). The NPMI score for one LDA model is then defined as the mean NPMI scores of all topics from the model.

The introduced similarity measure S-CLOP Eq. (7) quantifies pairwise similarity of two LDA runs by ${\displaystyle 1-\frac{1}{50}\sum _{g\in G^{\ast }}U\left(g\right)=1-\frac{1}{100}\sum _{g\in G^{\ast }}\left|g\right|\left(\left|\right|g_{|1}|-1|+\left|\right|g_{|2}|-1|\right),}$ where K = 50 is the number of topics per model and G^∗ an optimized set of topic clusters identified by our proposed pruning algorithm. We investigate the mean S-CLOP scores per LDA on the corpus from USA Today newspaper articles.

We propose to select the LDA run with highest mean pairwise similarity to all other runs. The following study shows that this is a suitable way to identify a stable prototype LDA, thus leading to improved reliability of LDA findings based on this particular run. We fit R = 100 LDA models and select the model with highest mean similarity as prototype. This procedure is repeated H = 100 times, which results in 100 prototype models. Then, for the 100 prototypes, also mean pairwise similarities to the other prototypes are calculated according to the schema in Fig. 2. The results are visualized in Fig. 6.

The very right curve describes the empirical cumulative distribution function of the mean similarities obtained for the 100 prototypes. At the very left there are the H = 100 curves of the 100 × 100 original runs. In addition, we also determine 100 prototypes from subsamples. For this, only 10, 20, 30, 40 or 50 runs from each original set of R = 100 runs are randomly selected and are used for the following calculation steps. The resulting curves are plotted and labeled in Fig. 6, the aggregated reliability scores are given in Table 4.

The minimum of the mean similarities from the original 100 sets of 100 models is 0.796, while the maximum is 0.877. It turns out that NPMI does not show a good ability to increase the reliability of LDA runs. For the reliability score of NPMI-selected runs, it makes little difference whether 10 or 100 models are available to the selection procedure. This suggests that the measure is better suited for other optimization goals than for increasing reliability. The perplexity does a relatively good job in the task of improving the reliability. For R = 100 candidate models, the relative reduction in the remaining improvement to the maximum reliability of 1 in comparison to random selection is 38% for our method LDAPrototype, namely (1 − 0.8425)/(1 − 0.8858) = 0.1575/0.1142 = 1.38, and 19% (0.1575/0.1326) for selection by perplexity. Therefore, the improvement in reliability for R = 100 is half that achieved by LDAPrototype. Overall, it can be seen that our method outperforms the other two popular and well-known selection methods in terms of increasing reliability.

Based on the findings from Fig. 6 and Table 4, we recommend to fit at least 50 replications because this leads to an increase of similarity to 0.862 at the minimum and 0.895 at the maximum, or a reliability score (mean) of 0.8816, respectively. This corresponds to a relative reduction of the remaining possible improvement of 33%. Higher values for the number of repetitions are desirable. In general, the choice depends on the complexity of the corpus. Encapsulated topics or certain complicated dependency structures make the modeling procedure more prone to a larger span of possible fits and therefore to smaller mean similarity values. However, if computational power is limited, already taking the prototype model from 10 candidates considerably increases the reliability. Here, the minimum and maximum of mean similarity are 0.842 and 0.880 (rs = 0.8622), considerably higher values than these associated with random selection and 14% relative reduction of remaining improvement.

Comparison of the implemented similarity measures

Up to this point, we have performed all calculations with our tJacc Eq. (3) in ‘Thresholded version of the Jaccard coefficient: a similarity measure for topics’. Now we study differences in reliability with different choices of measures of topic similarity. For this we compare cosine similarity Eq. (15), Jensen–Shannon similarity Eq. (14) and Rank Biased Overlap (RBO) Eq. (16) with the tJacc coefficient. We consider, where applicable, effects of different parameter constellations on the correlation of the similarities. We show that the tJacc coefficient is a good choice for the topic similarity measure, because it improves reliability while not having a disproportionate runtime.

We again consider the usatoday dataset for the comparison. For the tJacc coefficient we set limit.abs = 10 and atLeast = 0 and vary limit.rel ∈{0, 0.001, 0.002, 0.005}, for the RBO we choose eight combinations of k ∈ {10, 20, 50} and p ∈ {0.1, 0.5, 0.8, 0.9}. We compute all pairwise topic similarities of the total R⋅K = 5000 topics using the given measures and first consider the correlation of the pairwise LDA similarities based on these. In Fig. 7 all pairwise correlations of the similarity values are given.

Thereby, clear patterns can be identified. The similarities obtained with cosine and Jensen–Shannon are correlated with a value of 0.50. The similarity values based on the tJacc coefficient correlate with the cosine similarity depending on the parameter in the range from 0.40 to 0.47. It is noticeable that the correlation initially increases from 0.41 to 0.47 when considering a longer tail of words belonging to a topic, but drops to 0.40 when considering the complete tail. In contrast, the correlation with Jensen–Shannon LDA similarities increases steadily from 0.38 for limit.rel = 0.005 to 0.66 when considering the complete tail, or these words that were assigned to that topic at least 11 times (limit.abs = 10). It is interesting to note that even the different versions of the tJacc coefficient are mostly less correlated with each other than the tJacc LDA similarities are with the Jensen–Shannon LDA similarities. This illustrates that there can be significant differences between different choices for the threshold based on limit.rel.

As mentioned in ‘Thresholded version of the Jaccard coefficient: a similarity measure for topics’, choosing the parameter limit.rel as 0.002 determines around 100 words per topic as relevant. Thus, one could assume that the corresponding similarities should correlate strongly with those of the RBO with k = 100. In fact, we see that the RBO is rather weakly correlated with the other three measures overall; in particular, for low values of k ≤ 50 and p ≤ 0.8. The corresponding correlations are all below 0.30. However, it is also noticeable that for increasing k and p, the RBO similarities appear to be increasingly correlated with the other three similarity measures. This is due to the fact that for larger values of k and p the RBO converges to a measure very similar to the Jaccard coefficient or for p ≠ 1 to the AverageJaccard defined in Eq. (11) in Appendix A. This means that RBO similarities calculated with parameters k = 100 and p = 1 should be very close to tJacc similarities with limit.rel = 0.002. In the present case, however, even with 0.9, p is still much smaller than 1. Note that 0.9¹⁰⁰ ≈ 0 and thus the 100th word in the ranked list gets practically no weight, which is exactly the idea of the RBO. Therefore it is not recommended to choose the parameter p close to 1. Alternatively, a better approach would be to choose an implementation based on Jaccard, e.g., AverageJaccard, because it is faster to implement. These findings show that k and p should always be chosen dependent on each other. One can also see in Fig. 7 that words from rank 50 onwards no longer have a significant influence with a choice of p = 0.9. The correlation between the LDA similarities based on RBO with k = 50, p = 0.9 and k = 100, p = 0.9 is 1.

The four measures considered have different complexities in terms of their implementations. While the cosine similarity can be computed very quickly, the tJacc coefficient and the Jensen–Shannon similarity require computationally intensive precomputations, which increases the runtime. Cosine similarity and Jensen–Shannon similarity have no parameters to be set. For the tJacc coefficient, the calculations do not depend on the chosen parameters. The calculation of the RBO is generally time-consuming, since values must be calculated individually for all considered depths up to the maximum rank, which also means that the runtime for the calculation of the RBO strongly depends on the parameter k.

The runtimes are documented in Table 5. The times here are in hours, the calculations were run on four cores in parallel and are based on at least 100 different values. For each measure, a serial calculation was performed once in each case in order to be able to give a value for estimating the parallelizability of the calculations or implementations. It can be seen that the cosine similarity is the fastest to calculate with well under one hour. The thresholded Jaccard coefficient requires slightly more than twice as much time, while the Jensen–Shannon similarity computes with 83 and 275 min, respectively, again almost twice as long as the tJacc coefficient. The Rank Biased Overlap with k = 100 takes even over 2 days in parallel computation. This is also due to the fact that the implementation is only parallelized with a score of 0.6, i.e., one achieves only 60% of the maximum possible time saving through parallelization. With a maximum score of 1, the calculation would only take 30.97 hours instead of 51.62. The cosine similarity is also not implemented maximally parallelized. This is due to the overall low runtime. The tJacc coefficient, instead, is implemented in parallel with an approximate maximum time saving of 0.93.

After comparing the measures in terms of runtime and correlation of the resulting pairwise S-CLOP values, we restrict the analysis to one parameter combination per measure and compare the increase in reliability in dependence of the measure.

In Fig. 8, these are plotted against each other. We compare the cosine similarity, Jensen–Shannon similarity, the tJacc coefficient with limit.rel = 0.002 and the RBO with k = 20, p = 0.9. On the diagonal of the plot matrix, the known ecdfs of the LDA similarities are shown in black and the ecdf of the LDAPrototypes in red, respectively. In addition, we calculated—using the same measure—the similarities of the LDAPrototypes, that were determined based on the other three topic similarities. The colors of the ecdfs correspond to the color of the box of similarity measures. This allows the different measures to be analyzed in terms of their selection behavior taking into account the level differences of the similarity values. Table 6 gives the corresponding reliability scores of the ecdfs on the diagonal. The lower triangular plot matrix provides the LDA similarities as pairwise correlation plots, the upper matrix the corresponding correlations themselves. Determining the pairwise LDA similarities based on 100 experiments with R = 100 replications each yields 100⋅(R − 1)R/2 = 495000 values per similarity measure, on which the heatmaps and correlations are both based.

It can be seen that the LDA similarities according to the tJacc coefficient and according to the Jensen–Shannon similarity are most highly correlated with each other. The point cloud is least circular, but rather narrow and elliptical in shape. It can also be seen that the RBO LDA similarities are very different from the others. In the lower right plot (outlined in blue), the three ecdfs of different colors show significant differences from the red curve, with the cosine curve showing even slightly more similarity than the curves of the other two measures. Similarly, the blue curve is in the Jensen–Shannon (purple) and tJacc (orange) windows far behind the other colored ecdfs. The cosine LDA similarities (top left, green) differ barely for all the foreign-determined LDA prototypes, but the red curve sets itself apart from the others. Thus, the selection by cosine similarity obviously also differs significantly from the others.

The plots from Fig. 8 suggest that all four measures differ with regard to their selection criteria. However, a qualitative ranking which of the measures increases the reliability of the results the most is difficult because for this question the true evaluation measure has to be identified first, which is a vicious circle. For this reason, all these measures have their justification to be used within the procedure. We prefer to use the thresholded Jaccard (tJacc) coefficient because it has plausible heuristics and reasonable runtime. Its selection of the LDAPrototype strongly correlates with that of the Jensen–Shannon similarity, for which, according to Appendix A, besides the tJacc coefficient itself, the best results in terms of correlations with human perceptions could be obtained.

Comparison of different values for the parameters R and K

In addition to the choice of the similarity measure, the choice of the parameters K, number of topics to be modeled, and R, number of replications, also has a large influence on the runtime of the method. In ‘Increase of reliability’ it has already been shown that larger values for R result in a larger increase in reliability. We will confirm these findings based on the reuters dataset on the one hand and on the other hand bring them into a combined comparison with the number of modeled topics.

In Table 7 the runtimes for the determination of the LDAPrototypes based on the calculations of the topic similarities by the tJacc coefficient are given. Obviously, the runtime increases more than linear in both in the number of topics K to be modeled and in the number of replications R. This is due to the fact that the computation of the matrices of topic similarities have a quadratic complexity, since pairwise similarities are computed. The runtime for modeling the LDAs is linear in the parameters R and K.

Figure 9 shows the corresponding plots for the increase in reliability for all combinations of R = 50, 100, 200, 500 and K = 5, …, 15 and Table B.1 in the Appendix the corresponding reliability scores. Consistent with expectations, for fixed K, increasing the replication number from 50 to 500 does not change the location parameter for the ecdfs of the mean pairwise LDA similarities. However, the variance of the similarities decreases due to the higher replication number. It can be seen that for fixed R and increasing topic number K the level of similarities decreases. From an average similarity of 0.90 for K = 5 for the LDA replications, the value decreases to 0.75 for K = 10 and 0.65 for K = 15. The increase in reliability is marked by the orange ecdfs and is clearly pronounced for all parameter combinations. The gain is larger for higher topic numbers, so the level of LDAPrototype similarity does not decrease quite as much with increasing parameter K. The gain is also larger for increasing R. This is an expected behavior because then each prototype is determined from a larger set of individual LDAs and thus each LDAPrototype becomes even more reliable. For K = 5, the similarities of the prototypes thus increase from 0.97 for R = 50 to close to 1 for R = 500 (with LDA similarities around 0.90). For K = 9 they increase from 0.91 to 0.95 (0.78) and for K = 14 from 0.84 to 0.90 (0.67).

The findings from Fig. 6 are confirmed here. With increasing R the reliability gain increases. However, already for small values of R a clear increase is recognizable. Accordingly, R should be chosen as large as possible depending on the available computing power.

Comparison of the introduced datasets

For the corpus of 7,453 newspaper articles from the USA Today from 01/06 to 11/30/2016 and the reuters dataset with M = 91, an increase in reliability could clearly be shown. Lastly, we show that this increase results independently of the dataset, so that the LDAPrototype method can be reasonably applied to many different types of text data.

Figure 10 and Table 8 in particular show the increase in reliability for the datasets tweets, politics, and economy that have not yet been considered so far. For comparison, the corresponding plots for the usatoday dataset are shown, for reuters with K = 10 and the four different values for R = 50, 100, 200, 500 (cf. ‘Comparison of different values for the parameters R and K’). Figure 4 already showed the computability for large datasets on the example of the nyt dataset. We did not repeat the computation of the LDAPrototype 100 times for this dataset due to the comparatively higher runtime. The tweets dataset, with 3706740, does consist of a larger number of individual documents than the nyt dataset (1,993,182, cf. Table 2). However, due to the more specific topic structure, we chose a smaller topic number for this dataset. Together with the fact that the modeled tweets are much shorter than journalistic texts from the New York Times, this leads to a significantly lower runtime of just over one day per LDAPrototype than for the nyt dataset with about 130 days (cf. Table 3).

The plots show a lower gain in reliability for tweets than for the dataset economy, for example. This might surprise because the similarities of the basic LDA repelications are already somewhat higher for the latter. To be concrete, the similarities increase from about 0.88 to 0.94 for economy and from 0.87 to 0.91 for the dataset consisting of tweets. This reduced gain could be due to the shorter texts and the resulting higher uncertainty in the modeling. As a result, the instability of the LDA on this dataset is more pronounced, so that higher reliability increases are only possible with larger values for R. On the right side of Fig. 10 the results from the ‘Increase of reliability’ and ‘Comparison of different values for the parameters R and K’ are recapitulated. The increase of reliability increases steadily with an increasing number of LDA replications R.