All reviews of published articles are made public. This includes manuscript files, peer review comments, author rebuttals and revised materials. Note: This was optional for articles submitted before 13 February 2023.
Peer reviewers are encouraged (but not required) to provide their names to the authors when submitting their peer review. If they agree to provide their name, then their personal profile page will reflect a public acknowledgment that they performed a review (even if the article is rejected). If the article is accepted, then reviewers who provided their name will be associated with the article itself.
Following the comments of reviewers, the authors have successfully addressed all comments.
[# PeerJ Staff Note - this decision was reviewed and approved by Stefan Wagner, a PeerJ Section Editor covering this Section #]
The authors have made their work suitable for publication by considering our suggested corrections and improvements. The study was re-examined and discussed by the authors in various aspects and brought to a maturity suitable for publication.
The authors considered four different common topic similarity measures, which led to 14 different calculations in the size of the similarity measures with different parameter choices.
The authors chose hierarchical clustering when designing the method. However, in the first experiments, they tried different connection methods to demonstrate the effectiveness of the current method.
It is noted by the authors that perplexity is the most widely used performance metric for topic models.
The authors have already conducted their current analysis on six different datasets of different sizes (from 91 texts to 3.7 million texts), on different topics, and in German and English, testing the validity of the results in different experimental settings. This variety of datasets considered contributes to assessing the general potential of their method.
Therefore, the consistency and accuracy of the results obtained with different methods and datasets have been analyzed and evaluated.
Conclusion:
By expanding the scope of the study and considering the suggested improvements, the authors have conducted a more in-depth investigation. This comprehensive approach significantly enhances the study's reliability, validity, and generalizability. For these reasons, it is concluded that the study is suitable for publication and will make a significant contribution to the field.
Recommendation:
It is recommended that the study be accepted for publication in the relevant journal.
No comment
No comment
No comment
I have reviewed the revised manuscript Manuscript ID: 100312 titled “LDAPrototype: A model selection algorithm to improve reliability of latent Dirichlet allocation”. The authors have successfully addressed all the comments and suggestions provided during the initial review. The revisions made have significantly improved the quality and clarity of the manuscript. Therefore, I find the paper to be satisfactory and acceptable for publication.
Dear Authors,
The Reviewer 2 raises different comments on the experimental evaluations and on the discussion related to the achieved results. Please follow the suggestion and address the comments of this reviewer in order to improve the discussions related to your approach
1. Basic reporting
In order to solve the instability and reproducibility problems with the popular Latent Dirichlet Allocation (LDA) method in text mining, a novel strategy named LDAPrototype is presented in this work. Because LDA uses random initializations, it might produce inconsistent results on the same dataset, which can be problematic for reproducibility. By identifying the most representative model from several LDA runs on the same dataset, this study seeks to provide more consistent and trustworthy results.
By offering more replication similarity than typical LDA results or models chosen based on metrics like perplexity and Normalized Pointwise Mutual Information (NPMI), LDAPrototype increases the dependability of LDA results. S-CLOP, a novel model similarity metric, is employed to do this. Rank Biased Overlap, cosine similarity, Jensen-Shannon divergence, thresholded Jaccard coefficient, and other metrics are used by S-CLOP to compare topic similarities. The approach's efficacy is showcased on six distinct real-world datasets, encompassing tweets and newspaper articles.
2. Experimental design
In this work, the novel selection algorithm LDAPrototype is proposed for LDA models. LDAPrototype is based on a tailored similarity measure called S-CLOP. The experimental design of the study addresses the following processes:
1. Two-fold Contribution:
• The S-CLOP measure can assess the stability of LDA with clustering techniques applied to replicated LDA runs.
• High stability increases the reliability and reproducibility of findings.
2. Automated Clustering Method:
• A new automated method for clustering topics and a pruning algorithm are introduced.
• This method is based on the optimality criterion that each cluster should ideally contain exactly one topic from each replication.
3. New Similarity Measure S-CLOP:
• This measure, specifically designed for LDA runs, has the potential to improve the reliability of results.
4. Repetition Strategy and Selection Criterion:
• Based on S-CLOP, a combination of a repetition strategy and a selection criterion is proposed to increase the reliability of findings from LDA models through the LDAPrototype algorithm.
5. Performance Comparison:
• It is demonstrated that LDAPrototype outperforms perplexity and NPMI in terms of a reliability measure based on LDA similarities.
3. Validity of the findings
The results show that LDAPrototype enhances the repeatability and dependability of topic modeling outcomes. LDAPrototype is also commended for its interpretability, use, computational efficiency, and practicality. Because its concept may be applied to other topic modeling procedures that use word distributions to characterize themes, this technique is a versatile tool for text data analysis.
LDAPrototype provides more consistent and reliable results across repeated runs on the same dataset. This increases the reliability of the results obtained in research and applications and ensures the reproducibility of scientific studies. Therefore, the method needs to be tested on more experimental data sets to ensure full reliability of the findings. There may be an obvious application area for comparison with the findings of future studies.
Additional Comments:
This work is expected to contribute severally to the methodological landscape of topic modeling and natural language processing research.
My suggestions regarding this study are as follows:
Examining Various Similarity Measures: To find the best metrics for various data sets and application domains, it is possible to experiment with similarity metrics than S-CLOP. Alternative measurements of similarity, including the Mahalanobis and Hamming distances, can be investigated.
Comparison of Clustering Algorithms: Assessing whether clustering techniques best fit S-CLOP requires comparing various clustering algorithms. This may contribute to more precise and effective clustering outcomes.
Performance Comparison: Other performance metrics can be added to the fact that LDAPrototype appears to outperform perplexity and NPMI in terms of a reliability metric based on LDA similarities.
Experimental parameters: The experiment can be extended using a larger number of topics. How the results change according to LDA and other parameters such as alpha and beta can be discussed?
Comparison with Other Topic Modeling Methods: In addition to LDA, the impact of S-CLOP and LDAPrototype on these approaches can be investigated by contrasting them with other topic modeling approaches like NMF (Non-negative Matrix Factorization) and HDP (Hierarchical Dirichlet Process). It is also important to highlight the rationale behind the selection of LDA.
Testing Using Challenging Datasets: To evaluation the LDAPrototype's effectiveness using datasets with several languages or subjects. To understand how the algorithm performs on bigger and more complicated data sets, this is crucial.
• The entire manuscript has been produced by the authors in professional, precise English. Their attention to punctuation, spelling and grammars appears to have been significant during the manuscript preparation process. The writing style is professional, clear, and concise.
• The related work section demonstrates a thorough understanding of the existing methodology. The review successfully identifies specific gaps in the existing literature. However, authors could have included some relevant work, if any, by other researchers.
• Figures and tables are appropriately captioned and labelled, relevant and well-constructed. Figures quality have been maintained across the paper.
• The experimental work is designed and executed with a high degree of methodological rigor. The research provides detailed protocols for the experimental procedures, including precise measurements.
• Datasets have been explained and presented well. However, I would suggest the dataset description should be presented in tabular format.
• Experimental outcomes have been validated by a variety of runs. The experiments have been performed well across the parameter settings. Also, different datasets have been considered for numerous experiments. Overall, the experimental works have been carried out in a 360-degree view.
Authors are suggested to explain ‘the reliability of topic model’ in more detail. Give some appropriate example for better understanding of the article.
The S-CLOP is an abbreviated term. Please mention full-form of it in the introduction section. Therefore, reader can interpret it better.
Authors should cite recent research articles in the domain under study. They should discuss the limitations and issues of the recent papers.
Your related work needs more detail. I suggest that you include similar research work.
Authors should have explained the term ‘replication’ with respect to LDA topic inference for better understanding of the readers.
I suggest authors should explain how the medoid is chosen.
Page No. 6 – It seems that first line is not part of text. Please check and remove if not needed. I suggest to remove if it is not needed.
Line No. 224 – “Then the deviation from the perfect situation of one representative from each LDA run is quantified.” Please explain in detail how it is done.
Line No. 225, authors have mentioned ‘representative’ of each LDA run, Please elaborate this termilogy.
Line No. 237, authors have explained LDA disparity. I suggest, if possible, authors to take an example for the readers. Why is disparity calculated? Please describe.
Line No. 248 and 249 needs to be revised, as it seems less interpretable.
As you have removed the words with total count less than some limit (5 in this study) to reduce vocabulary size, have you considered upper bounds to remove the words that appear very frequently?
Relative limit is 0.002, and it results in 100 words per topic as relevant. Generally, tip 10 or 20 words are chosen for the topic representation. Could you please explain the reason behind considering 100 words per topic?
All text and materials provided via this peer-review history page are made available under a Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.