Mobile app review analysis for crowdsourcing of software requirements: a mapping study of automated and semi-automated tools

Mapping study’s structure

Our article is structured in a clear and organized manner. “Background and Related Work” delves into an in-depth analysis of existing systematic literature reviews that have examined the use of app reviews for software features and requirements. This background provides a solid foundation for understanding this domain’s current state of research. “Study Methodology” then outlines the meticulous mapping review methodology, including the research questions, literature search strategy, inclusion and exclusion criteria, study selection process, data extraction and synthesis, and quality assessment process. This detailed explanation of the research approach ensures transparency and replicability. The mapping study results are presented in “Mapping Study Results”, offering a comprehensive overview of the identified tools and their capabilities. “Discussion and Future Research” discusses the implications of these findings, highlighting key trends, strengths, and limitations of the existing approaches. This discussion also provides directions for future research in this rapidly evolving field. “Threats to the Validity” outlines the threats to the study’s validity, addressing potential biases and limitations. Finally, “Conclusion” summarizes the overarching conclusions and valuable insights from this systematic mapping study. Figure 1 visually represents the structure and fundamental concepts of this mapping study. This comprehensive overview diagram visually represents the article’s structure, key concepts, and their interrelationships. The main sections of the article are represented as primary nodes, with subsections as secondary nodes. The solid lines indicate the flow of the article’s structure, while the dotted lines show conceptual relationships and influences between different parts of the study.

Research motivation and significance

This mapping study addresses the critical need for efficient analysis of user feedback in mobile app development. Developers struggle to leverage user reviews for software improvements as the app ecosystem expands. Automated and semi-automated tools offer a solution, but their landscape remains fragmented. This study aims to comprehensively analyze existing research on these tools, synthesizing findings from diverse primary studies. It will highlight current trends, common approaches, and areas for future development. The study’s significance lies in its potential to inform researchers and practitioners, guiding tool selection and optimization within development workflows. Moreover, the insights gained can have broader implications for the software engineering community, potentially driving advancements in requirements engineering across various domains. By synthesizing the current state of the art, identifying key trends and limitations, and highlighting future research directions, this work aims to catalyze further innovation and improve the effectiveness of requirements engineering processes in the dynamic and user-driven mobile app ecosystem.

Mobile app ecosystems and user feedback

Mobile app ecosystems and user feedback have become integral components of the software development lifecycle, shaping how applications evolve and adapt to user needs. These ecosystems, primarily dominated by Apple’s App Store and Google Play Store, serve as platforms where developers can distribute their applications and users can download, use, and review them. Within this framework, user feedback emerges as a crucial element, providing developers with direct insights into user experiences, preferences, and pain points (Motger et al., 2024c). The significance of user feedback in mobile app ecosystems cannot be overstated. As Pagano & Maalej (2013) point out, app reviews serve as a rich source of information for requirements elicitation and prioritization. According to Malgaonkar, Licorish & Savarimuthu (2022), these reviews often contain feature requests, bug reports, and user experiences that can directly inform the development process.

Moreover, the public nature of these reviews creates a unique dynamic where user opinions can significantly influence an app’s reputation and, consequently, its success in the marketplace. However, the sheer volume of user feedback presents both opportunities and challenges. On the one hand, developers have access to an unprecedented amount of user-generated data to guide their decision-making processes (Oh et al., 2013). On the other hand, manually processing and analyzing this vast amount of information is often impractical and resource-intensive (Araujo, Gôlo & Marcacini, 2022; Wang et al., 2022a). This challenge has led to the development of various automated and semi-automated tools aimed at efficiently extracting actionable insights from user reviews (Guzman & Maalej, 2014).

The evolution of mobile app ecosystems has also led to changes in user behavior and expectations. Users now expect rapid responses to their feedback and quick iterations in app development. This shift has necessitated more agile and responsive development practices, further emphasizing the need for efficient feedback analysis tools (Khalid, Asif & Shehzaib, 2015). While user feedback in mobile app ecosystems offers valuable insights, it is not without limitations. The voluntary nature of app reviews can lead to sampling bias, where only users with intense positive or negative experiences may choose to leave feedback. Additionally, the unstructured format of reviews can make it challenging to extract precise requirements or prioritize user needs effectively. These limitations underscore the importance of developing sophisticated analysis tools to account for such biases and extract meaningful patterns from unstructured data.

Crowdsourcing in software engineering

Crowdsourcing has emerged as a powerful paradigm in various domains, and its application in software engineering has gained significant traction in recent years. At its core, crowdsourcing in software engineering involves leveraging a large’s collective intelligence and efforts (Satzger et al., 2014), often diverse groups of individuals, to address software development challenges. In software requirements engineering, crowdsourcing offers a novel approach to gathering, refining, and prioritizing user needs (van Vliet et al., 2020; Khan et al., 2019). As Hosseini et al. (2015) argue, crowdsourcing can lead to more comprehensive and user-centric requirements by tapping into a wider pool of perspectives and experiences. This approach is particularly relevant in the mobile app domain, where user bases are often large and diverse, and user needs can vary significantly across different demographics and usage contexts.

One of the key advantages of crowdsourcing in software engineering is its ability to scale. Traditional requirements-gathering methods, such as focus groups or surveys, are often limited in their reach and can be time-consuming and costly to implement at scale (Courage & Baxter, 2005). In contrast, crowdsourcing through app reviews allows developers to gather feedback from thousands or even millions of users continuously (Palomba et al., 2018) and at relatively low cost (Maalej et al., 2016). However, the application of crowdsourcing in software engineering is not without challenges. Quality control remains a significant concern, as the open nature of crowdsourcing can lead to noise, irrelevant contributions, or even malicious inputs. Moreover, managing and coordinating large crowds of contributors can be complex, requiring sophisticated platforms and incentive structures to ensure effective participation (Stol & Fitzgerald, 2014). Another critical aspect of crowdsourcing in software engineering is the need for effective aggregation and synthesis of diverse inputs.

As Khan et al. (2022) note, transforming raw crowd input into actionable software requires sophisticated analysis techniques. This need has driven the development of various automated and semi-automated tools designed to process and analyze crowdsourced feedback, particularly in the context of mobile app reviews. The intersection of mobile app ecosystems, user feedback, and crowdsourcing in software engineering presents exciting opportunities and significant challenges. As we explore automated and semi-automated tools for app review analysis, it is crucial to keep in mind the complex ecosystem within which these tools operate. The effectiveness of these tools will ultimately be judged by their ability to harness the power of crowdsourced feedback while addressing the inherent challenges of scale, quality, and synthesis in the mobile app development context.

Automated and semi-automated analysis tools

As mobile app ecosystems continue to evolve and user feedback becomes increasingly valuable for software requirements engineering, a range of automated and semi-automated tools have emerged to address the challenges of analyzing large volumes of app reviews. These tools leverage various techniques, including natural language processing, machine learning, and sentiment analysis, to streamline the extraction and categorization of user feedback (Guzman & Maalej, 2014).

One of the primary advantages of these tools is their ability to scale and process large datasets efficiently. Traditional manual review of app reviews is often time-consuming and resource-intensive, particularly as the number of reviews grows exponentially. Automated tools can rapidly sift through thousands or even millions of reviews, identifying patterns, extracting features, and classifying user feedback such as bug reports, feature requests, and user experiences (Iacob & Harrison, 2013; Guzman & Maalej, 2014; Malgaonkar, Licorish & Savarimuthu, 2022).

The degree of automation in these tools varies, with some employing fully automated approaches and others relying on a combination of automated and human-in-the-loop processes. Fully automated tools typically leverage sophisticated natural language processing algorithms and machine learning models to analyze review content with minimal human intervention (Pagano & Maalej, 2013). These tools offer the benefit of speed and consistency, but they may struggle with complex or nuanced language, requiring careful tuning and validation to ensure reliable performance. In contrast, semi-automated tools incorporate a degree of human oversight and involvement, often using automated techniques as a starting point and then relying on human experts to validate, refine, or override the system’s outputs (Maalej et al., 2016). This approach can help address the limitations of fully automated systems, particularly in cases where user feedback is ambiguous, context-dependent, or requires deeper understanding. However, the involvement of human analysts introduces additional time and resource requirements, potentially limiting the scalability of these semi-automated approaches.

One key area of focus for automated and semi-automated tools is the extraction and categorization of user feedback into meaningful and actionable insights. By automating this process, developers can quickly prioritize and address the most critical user needs, potentially leading to more user-centric and successful applications. Additionally, some tools have been explored using more advanced techniques, such as topic modeling and sentiment analysis, to uncover hidden patterns and trends within user feedback (Noei et al., 2019). These approaches can help developers better understand user sentiment, preferences, and emerging requirements, informing both short-term iterations and long-term product roadmaps.

While developing these automated and semi-automated tools has been a significant area of research, the field has limitations and challenges. One key concern is the accuracy and reliability of these tools, as even minor errors in classification or sentiment analysis can have cascading effects on downstream requirements engineering processes (Maalej et al., 2016). Developers must carefully evaluate the performance of these tools and ensure that their outputs are consistently reliable and trustworthy. Another challenge is the adaptability and generalizability of these tools across different app ecosystems and user populations. Many existing studies have focused on English-language reviews, raising questions about the applicability of these techniques to multilingual or culturally diverse app markets (Tavakoli et al., 2018; Yin et al., 2024b). Addressing these challenges will be crucial for the widespread adoption and effective implementation of automated and semi-automated app review analysis tools.

Previous related studies and surveys

Analyzing user feedback in mobile application marketplaces is an active research area, providing valuable insights into user experiences, requirements, and requests (Jacek et al., 2022). Numerous studies have focused on extracting information from app reviews, with some targeting detailed features and specifications. For instance, Chen et al. (2014) reviewed general techniques for app review analysis, including information retrieval, topic modeling, and natural language processing (NLP). Their research highlighted the rich information content of user reviews and proposed a taxonomy for categorizing this feedback. This study was instrumental in demonstrating the value of app reviews as a source of user requirements and set the stage for subsequent research on automated analysis techniques. Building on this foundation, Maalej et al. (2016) presented a comparative study of manual vs. automated classification of app reviews. Their research evaluated different machine learning classifiers for categorizing reviews into bug reports, feature requests, and user experiences. While their results showed promise for automated classification, they also highlighted the continuing need for human oversight in interpreting and acting on the classified feedback. Martin et al. (2017) conducted a systematic mapping study categorizing various app review analyses, highlighting techniques like topic modeling, sentiment analysis, and NLP for extracting requirements-related information. However, their study was high-level and lacked in-depth comparative analysis of feature extraction techniques.

A comprehensive survey by Tavakoli et al. (2018) provided a systematic mapping of user feedback analysis techniques in app stores to assist developers in extracting insights from user reviews, analyzing and categorizing 34 studies based on techniques, everyday topics, and challenges in feedback mining. The research emphasizes domain-specific influences on user reviews when selecting mining techniques. Recent studies have emphasized extraction techniques, understanding domain influences, and emerging themes in user feedback. Genc-Nayebi & Abran (2017) explored automated systems for identifying, classifying, and summarizing opinions from app store reviews. These addressed challenges like data sparsity in short reviews and domain barriers in opinion extraction. They proposed methodologies like domain adaptation and grammar rules for identifying opinion-bearing words, aiming to provide evidence-based guidelines for app store practitioners and future research directions.

Dąbrowski et al. (2022a) presented a systematic literature review of app review analysis for software engineering (SE), categorizing app review analyses and data mining techniques. They provided insights for researchers and practitioners on extracting valuable information from reviews, emphasizing the need for deeper stakeholder understanding to enhance tool applicability. The study advocates for improved evaluation methods, reproducibility, scalability, and efficiency in future research to advance app review analysis in SE.

A significant contribution to the field came from Lin et al. (2022), who conducted a systematic literature review on app store analysis for software engineering. This comprehensive survey synthesized findings from 185 articles, providing a holistic view of the state of research in app store analysis, including techniques for review analysis. Their work highlighted the rapid growth of this research area and identified key challenges and opportunities for future work. Santos, Groen & Villela (2019) reviewed automated classification techniques in RE, focusing on Crowd-based Requirements Engineering (CrowdRE) and NLP. CrowdRE adapts NLP techniques to analyze large amounts of user feedback in RE. However, the suitability of specific NLP techniques for CrowdRE is poorly understood, making it challenging to choose the proper technique. ML is commonly used in CrowdRE research, with naïve Bayes with Bag of Words-Term Frequency (BOW-TF) and support vector machines (SVM) with BOW-TF being popular algorithm-feature combinations. Initial assessments show that precision and recall in RE classifications need improvement, urging researchers to explore new strategies and ML models to advance the field.

Dąbrowski et al. (2022b) analyzed how mining app reviews can benefit SE activities by examining 182 articles published between 2012 and 2020. They provided an overview of various use cases to improve SE processes like requirements gathering design, maintenance, and testing. The study highlights the benefits of app review analysis for software engineers and unifies existing research efforts into a reference architecture for future tool development and evaluation. It also addresses the practicality of 29 existing app review analysis tools. It suggests areas for further research and improvement in academia and industry, acknowledging limitations regarding interpretation, validation, and completeness of use cases.

Dąbrowski et al. (2023) presented two empirical studies on opinion mining and text summarization for software requirements. The first study evaluated three opinion mining approaches: SAFE (Johann, Stanik & Maalej, 2017), GuMa (Guzman & Maalej, 2014), and ReUS (Dragoni, Federici & Rexha, 2019), using review extraction and sentiment analysis techniques. The second study compared three approaches for capturing requirements reviews: Lucene, MARAM (Iacob, Faily & Harrison, 2016), and SAFE (Johann, Stanik & Maalej, 2017), with Lucene performing better. The findings suggest the potential of using these text summarization and sentiment analysis techniques to enhance requirements extraction from app reviews.

Martin et al. (2017) extensively examined reviews from app stores like Google Play, Apple App Store, and BlackBerry Store, discussing the evolution of review-centered literature since 2012. They addressed the “App Sampling Problem” and suggested future research directions, such as tools for extracting requirements from reviews and comparing review cultures across platforms. The study also investigated app security trends, noting a lower likelihood of malware in popular apps. They emphasized sentiment analysis, tools like WisCom for review summarization, and the challenges of large review samples and accurate data labeling.

Al-Subaihin et al. (2019) explored techniques for measuring the similarity of mobile applications based on textual descriptions to enhance clustering solutions. Through an empirical study of 12,664 apps from the Google Play Store, they compared different methods, including topic modeling and keyword feature extraction, using hierarchical clustering algorithms. The results showed that similarity-based techniques perform well in detecting app-feature similarity, while dependency-based techniques struggle. The study highlights the need for continued research and effectiveness across different app stores, suggesting their potential for improving app review analysis techniques.

Despite these advances, several limitations and challenges persist in the field. Many studies have focused on English-language reviews, raising questions about the applicability of these techniques to multilingual app ecosystems. Additionally, the dynamic nature of app stores and rapidly evolving user expectations pose ongoing challenges for maintaining the relevance and accuracy of analysis tools. Furthermore, while much progress has been made in automating the extraction and classification of user feedback, translating this information into actionable software requirements remains a complex task.

Building upon the work of Maalej et al. (2024) on automated user feedback processing, this systematic review and mapping study addresses a critical gap in the literature. Despite recognizing app reviews’ value in requirements engineering, a lack of systematic evaluation of automated feature extraction methods remains. This study aims to fill this void by comprehensively analyzing and comparing various techniques, focusing on performance metrics. Furthermore, it explores the potential of integrating visualization and recommendation systems to enhance analyst interaction with processed feedback, facilitating more effective information retrieval (Wang et al., 2024; Huang et al., 2023; Zhang et al., 2024). By synthesizing existing research and identifying areas for improvement, this work contributes to the advancement of automated analysis tools in requirements engineering, paving the way for more efficient and accurate feature extraction from user reviews.

Research questions

The primary objective of this mapping study is to conduct a meticulous investigation into the diverse array of automated and semi-automated techniques employed for extracting software requirements from reviews of mobile applications. To accomplish this aim, we will seek guidance from the following formulated research inquiries (RQs):

• RQ1: What feature extraction techniques are employed for analyzing mobile app reviews to extract software requirements? The main objective of RQ1 is to examine and classify feature extraction techniques or methods and tools, considering their fundamental approaches (such as natural language processing, rule-based systems, topic modeling, and hybrid techniques).

• RQ2: What automated and semi-automated tools are available to support the implementation of these feature extraction techniques? This question will comprehensively overview the automated and semi-automated tools implemented in this domain. It seeks to identify the solutions proposed in the literature by employing these feature extraction techniques to extract relevant features, requirements, or user feedback from mobile app reviews.

• RQ3: How do the app review analysis tools compare performance, scalability, and user-friendliness? Evaluating the performance of the identified techniques is essential for assessing their effectiveness and practical applicability. This research question investigates the metrics and methodologies used to measure the performance of the methods, such as accuracy, precision, recall, or F-score. Furthermore, it aims to collate and compare the reported performance results, providing valuable insights into the most promising approaches.

• RQ4: What are the significant strengths and limitations observed in current techniques based on their methodology or evaluation results? Understanding their respective strengths and limitations is crucial for RQ4. This RQ investigates the advantages and drawbacks of each approach, enabling a comparative analysis and facilitating the selection of appropriate techniques based on specific needs or scenarios. Moreover, this analysis may reveal potential areas for improvement or opportunities for new technique development.

• RQ5: What future research directions could address current gaps in capabilities for efficient and precise analysis of app reviews for requirements? This RQ seeks to identify emerging trends and potential future research directions in automated and semi-automated feature extraction from mobile app reviews. This question may uncover gaps, challenges, or unexplored avenues that could guide future research efforts and drive innovation in this domain by analyzing the existing literature.

The RQs evaluate and compare various feature extraction techniques and tools for analyzing mobile app reviews, aiming to understand their effectiveness in deriving software requirements from user feedback. The questions explore the practical implications and potential applications of the extracted features in the software RE process. The findings will provide insights into available techniques and tools while laying the groundwork for future research and development in this rapidly evolving domain.

Literature search strategy

A rigorous literature search strategy was employed to conduct a comprehensive and systematic mapping study on feature extraction techniques and tools for mobile app review analysis. This paragraph explains the search process’s key aspects, such as data sources, search queries, and criteria for selecting relevant studies, as shown in Fig. 3.

As Fig. 2 reflects, the literature search began by identifying appropriate digital libraries and databases. Specifically, we leveraged well-established sources: IEEE Xplore, Scopus, ScienceDirect, ACM Digital Library, and SpringerLink. These databases extensively cover peer-reviewed literature in software engineering, requirements engineering, and mining software repositories. Both keyword searches and backward snowballing will be used to retrieve candidate articles. Keyword searches will involve combinations of terms related to the domain (e.g., “app”, “mobile application”, “app store”) and interventions of interest (e.g., “mobile app review analysis”, “user review analysis”, “feature extraction”, “automatic feature extraction”, “mining user reviews”, “feature extraction tools”, “feature requests”, “identifying key features”), adapted appropriately for each database.

Additionally, we employed Boolean operators (e.g., AND, OR) and wildcards to broaden the search and capture relevant variations of the terms. Backward snowballing will entail scanning reference lists of highly relevant articles to find additional studies. The search process will be refined through trial searches to maximize coverage of pertinent literature. Search results will be collated in a reference management tool, and duplicate items will be removed. The overall search and screening process is outlined as shown in Table 1. The searches will be limited to literature published in the past 10 years to focus on current techniques.

During title and abstract screening, articles will be assessed for relevance based on mentions of apps, reviews, requirements, and features. The full article will be thoroughly retrieved and scrutinized in the text review stage. Only peer-reviewed literature published in English with the whole article accessible will be retained. Grey literature, pre-prints, extended abstracts, and unavailable documents will be excluded. Finally, the methodology rigor and alignment to the comparative research questions will be evaluated for the final selection. Any articles with flawed, weak, or insufficient methodology will be excluded to ensure the review is based only on evidence from robust studies. Following this systematic process, the literature pool can be refined to contain only high-quality studies focused on feature extraction from app reviews to inform a comparative analysis. At each stage, inclusion and exclusion criteria will be applied to evaluate each article’s relevance and methodology rigor. A rigorous and multi-stage search strategy will aim to identify an exhaustive corpus of literature for answering the comparative research questions.

The literature search results in Table 2 offer valuable insights into the mapping study’s systematic approach and comprehensive scope. A broad search across five major academic databases initially yielded 1,179 potentially relevant studies. IEEE Xplore and Scopus emerged as the most prolific sources, contributing 286 and 324 initial results, respectively. This breadth of initial results underscores the extensive automated app review analysis research activity. The screening process, conducted in two stages, significantly refined the pool of studies. After title and abstract screening, the number of relevant studies was reduced to 534, less than half of the initial count. This substantial reduction highlights the importance of precise search terms and the challenge of identifying relevant studies in a rapidly evolving field. The full-text screening further narrowed the selection to 351 studies, demonstrating the rigorous criteria applied to ensure the quality and relevance of the included research.

Notably, the study incorporated additional sources through backward and forward searches, yielding 21 more studies in the final selection. This approach enhances the comprehensiveness of the review by capturing relevant work that might have been missed in the initial database searches. The final count of 180 included studies represents a carefully curated subset of the available literature, balanced across various databases and supplemented by targeted searches. IEEE Xplore contributed the largest number of final included studies (48), followed by Scopus (36), highlighting these databases’ significance in the field. Including studies from diverse sources, including ACM Digital Library, ScienceDirect, and SpringerLink, ensures a broad representation of research perspectives and methodologies in the mapping study.

By employing this systematic and rigorous literature search strategy, we aimed to ensure the comprehensiveness and quality of the included studies. This, in turn, provided a solid foundation for the mapping study and enabled robust analyses and insights into the state-of-the-art feature extraction techniques and tools for mobile app review analysis.

Inclusion and exclusion criteria

Establishing explicit inclusion and exclusion criteria in systematic literature reviews is crucial to outlining the study range. Precise criteria enable reviewers to objectively evaluate each study’s relevance and methodological soundness during the selection phase. This review applies both criteria across four dimensions: domain, language, interventions, and methodology, as shown in Table 3. It considers explicitly studies from peer-reviewed journals, conference proceedings, and book chapters, as these sources typically uphold high academic standards through rigorous peer-review processes. Furthermore, to facilitate a comprehensive understanding and analysis of the findings, only studies published in English were included. While this decision may have excluded potentially relevant studies published in other languages, it was necessary to ensure consistency and avoid potential misinterpretations due to language barriers. Only studies on analyzing user reviews from mobile apps will be included regarding the domain. Any literature centered on other domains like desktop or web applications will be excluded as out of scope.

Only studies addressing feature extraction methods or tools for analyzing mobile app reviews with software requirements elicitation were considered to meet the mapping study’s goals. Studies focusing solely on traditional software requirements elicitation techniques without considering user reviews or mobile app contexts were excluded, as they did not directly contribute to the mapping study’s specific research questions and goals. Moreover, the included studies were expected to provide substantive details and empirical evaluations of the proposed feature extraction techniques or tools. Studies that merely mentioned user reviews or feature extraction without providing in-depth descriptions, implementation details, or empirical evaluations were deemed insufficient and were consequently excluded from further analysis. Finally, studies that deviated significantly from the scope and objectives of the mapping study were excluded to maintain a focused and coherent analysis. This criterion ensured that the included studies were directly relevant to the research questions and contributed valuable insights to the mapping study. The mapping study aimed to create a thorough and excellent collection of studies by following specific inclusion and exclusion guidelines. This comprehensive resource enables robust analyses and syntheses of cutting-edge feature extraction methods and tools for mobile app review analysis, particularly in software requirements elicitation contexts.

Data extraction and synthesis process

Upon defining the inclusion and exclusion standards and selecting pertinent studies, a systematic data extraction and synthesis method was applied to analyze and integrate the results thoroughly. This section outlines the details of this process, ensuring transparency and reproducibility of the mapping study. The data extraction process involved carefully examining each included study and recording relevant information in a standardized data extraction form. This form was designed to capture essential details such as publication metadata (e.g., authors, year, publication venue), study characteristics (e.g., research methodology, dataset details), and key findings related to the research questions. As shown in Table 4, the data extraction form collects essential information from articles, such as authors, year, techniques, evaluation method, datasets, performance metrics, limitations, and key findings. The form is adaptable, allowing for additional parameters during the review process. After completing the data extraction, a thorough analysis was conducted to identify patterns, trends, and insights related to the research questions mentioned in the introduction. The results were systematically organized, making it easy to comprehend the current state of feature extraction methods and tools for mobile app review analysis. This understanding helps with software requirements elicitation. We held frequent team meetings and discussions to maintain the synthesis’s validity and reliability. These meetings served as a forum for reviewing and refining the synthesized findings, resolving any ambiguities or disagreements, and ensuring that the conclusions drawn were supported by the extracted data and aligned with the objectives of the mapping study. Through a thorough and organized data extraction and synthesis process, the mapping study aimed to deliver a comprehensive and reliable analysis of advanced feature extraction methods and tools for mobile app review assessment. This contributes to the progress of software requirements elicitation and guides future research in this area.

Study selection and characteristics

The mapping study involved a rigorous study selection process to ensure the inclusion of relevant and high-quality research works. This section discusses the characteristics of the selected studies and presents a detailed overview in Table 5. Through a thorough literature search and applying selection criteria, 180 studies were chosen for the mapping study (Supplemental Information). These studies, from various sources like top journals, conferences, and books, showcase the field’s multidisciplinary nature. Table 5 offers an overview of the studies’ characteristics, including publication patterns, methodologies, and evaluation datasets. As shown in Fig. 4, the publication timeline shows an upward trajectory, with notable growth from 2016 to 2021. While the early years (2014–2015) saw modest output, research activity peaked during 2019–2021, accounting for a third of all studies. A slight decline is observed in 2022–2023, though the partial data for 2024 suggests sustained interest in the field.

Publication types are evenly distributed between conference proceedings (48.9%) and journal articles (47.8%), with a small representation of book chapters (3.33%). This balance indicates the topic’s relevance in academic and practical spheres, fostering discussions across various platforms. Empirical studies dominate the research methodology, comprising 67.2% of the selected works, underscoring a strong focus on data-driven approaches. Theoretical and comparative studies, while less prevalent, contribute to the field’s conceptual development and evaluation of different techniques. Regarding data sources, mobile app store reviews are overwhelmingly favored and used in 73.3% of studies. This preference aligns with the research area’s practical orientation, focusing on real-world user feedback for software requirements engineering. Repository datasets and online store reviews are utilized less frequently, potentially due to accessibility or relevance constraints.

The consistent publication rate over the years, with recent peaks, reflects the ongoing relevance and evolution of automated app review analysis. This trend suggests a maturing field that continues to address the challenges of efficiently processing user feedback in mobile app development. The predominance of empirical studies using app store data highlights the field’s commitment to practical, real-world applications. At the same time, the balance between conferences and journals indicates active discourse across immediate and in-depth research contexts.

Quality assessment

We conducted a rigorous literature search and screening to identify relevant primary studies. To ensure the quality and reliability of the findings, we further evaluated the selected primary studies based on well-established quality assessment criteria, as recommended by Kitchenham’s (2004) guidelines for systematic literature reviews in software engineering. Several quality assessment questions were considered to assess the primary studies’ rigor, validity, and potential for bias. This step is crucial to ensure the robustness and trustworthiness of the review’s findings. The following quality assessment (QA) questions can be adapted for this mapping study:

• QA1: Are the research objectives and questions clearly stated?

• QA2: Is the study context (e.g., application domain, type of review data) adequately described?

• QA3: Is the feature extraction technique or tool thoroughly explained, including its underlying principles, algorithms, and implementation details?

• QA4: Are the evaluation datasets representative of real-world app reviews and adequately described (size, source, pre-processing steps)?

• QA5: Are the evaluation metrics and performance measures clearly defined and appropriate for assessing the technique’s effectiveness?

• QA6: Are the evaluation methods and experimental design sound and well-documented?

• QA7: Are the study’s limitations and threats to validity acknowledged and discussed?

• QA8: Does the presented evidence and analysis support the study’s findings and conclusions?

• QA9: Is the study’s contribution to app review analysis and software requirements elicitation clearly stated and justified?

• QA10: Is the study well-written, organized, and understandable for the intended audience?

As captured in Appendix A, we provided hypothetical quality assessment scores for all 180 studies included in the mapping study based on the predefined criteria. The quality assessment questions (Q1–Q10) are listed in the column headers, and a score of 1 (Yes), 0.5 (Partially), or 0 (No) is assigned for each question based on the assessment of the respective study. The total score for each survey is calculated by summing up the individual scores for all questions. This total score can provide an overall measure of the study’s quality, with higher scores indicating better quality and lower scores suggesting potential issues or limitations.

Specifically, a significant portion of the studies (approximately 30%) can be categorized as good quality, with total scores ranging from 8 to 9.5. These studies have effectively addressed most, if not all, of the quality criteria, including clearly stated research objectives, thorough descriptions of the feature extraction techniques and tools, well-documented evaluation methods, and comprehensive discussions of limitations and future research directions. Moreover, most studies (approximately 55%) fall into the average quality category, with scores ranging from 6.5 to 7.5. For instance, some studies may have provided insufficient details regarding the evaluation datasets or the specific implementation of the feature extraction techniques. In contrast, others may have lacked a comprehensive discussion of the study’s limitations or potential threats to validity.

Additionally, a smaller subset of the studies (approximately 15%) can be classified as poor quality, with scores below 6.5. These studies may have significant shortcomings in one or more quality criteria, such as a lack of clear research objectives, inadequate descriptions of the feature extraction techniques or tools, poorly designed or executed evaluation methods, or absence of a critical discussion of the study’s limitations and implications.

However, this assessment highlights potential areas for improvement. It can guide future research efforts in developing more robust and reliable feature extraction techniques and tools for mobile app review analysis in the context of software requirements elicitation.

RQ1: What feature extraction techniques are employed for analyzing mobile app reviews to extract software requirements?

Data was gathered on techniques explored in each article to study feature extraction methods for mobile app analysis (RQ1). Figure 5 presents diverse extraction techniques used in software requirements elicitation. This variety of techniques, such as Topic Modeling Techniques, Collocation Finding Techniques, Association Rule-Based Techniques, Aspect-Based Sentiment Analysis, Frequency-Based Techniques, Word Vector-Based Techniques, Hybrid Techniques, and Large Language Models, offers researchers and practitioners a broad toolkit for addressing the multifaceted challenges of extracting meaningful features from user-generated app reviews. Each category represents a unique approach with specific strengths, limitations, and applications. Out of the 180 included studies, Fig. 6 highlights that hybrid techniques emerge as the most prevalent approach, accounting for 21.6% of the identified methods. This preference for combining multiple techniques suggests recognizing the complex nature of app review analysis and the potential benefits of leveraging complementary approaches. Following closely are word vector-based techniques at 17.0%, indicating a solid reliance on advanced natural language processing methods that capture semantic relationships between words. Topic modeling techniques and large language models represent the next tier of popularity at 14.5% and 12.6%, respectively. Despite their relative novelty, the significant presence of large language models underscores the rapid adoption of state-of-the-art artificial intelligence (AI) technologies in this field. Aspect-based sentiment analysis (11.6%) and frequency-based techniques (9.9%) also maintain a notable presence, highlighting the continued relevance of both sentiment-aware and statistical approaches. Less frequently employed methods include collocation-finding techniques (7.9%) and association rule-based techniques (5.0%). While these approaches appear less dominant, their inclusion in the taxonomy reflects the multifaceted nature of feature extraction in app review analysis.

The diversity of techniques identified in this study points to a field actively exploring various methodological avenues. This breadth of approaches suggests that researchers are addressing the challenges of feature extraction from multiple angles, likely in response to the varied nature of app reviews and the specific requirements of different analysis tasks. Moreover, the prominence of hybrid techniques and the adoption of advanced methods like large language models indicate a trend towards more sophisticated, integrated approaches to feature extraction. This evolution may reflect the growing complexity of app ecosystems and the increasing expectations for nuanced, context-aware analysis of user feedback. The mapping study reveals a rich and evolving landscape of feature extraction techniques in mobile app review analysis. The field is characterized by methodological diversity, with a trend towards hybrid and advanced model approaches. By offering an overview, it helps researchers and practitioners understand current trends and select suitable techniques based on specific needs and limitations. The following subsections detail each technique.

RQ2: What app review analysis tools have been developed to support the implementation of these feature extraction techniques?

Software requirements analysis has seen the development of various tools and frameworks that leverage techniques like natural language processing (NLP), machine learning, and information retrieval to facilitate accurate and efficient extraction and analysis of user requirements from diverse data sources, including user reviews. This study identified 48 tools used in the selected research for extracting features and requirements from user reviews, as illustrated in Table 6. The following section provides an overview of these prominent tools and approaches employed in this domain.

Palomba et al.’s (2015) CRISTAL approach traces mobile app user reviews to code changes to enhance app success. Using the AR-MINER tool extracts informative feedback from app stores, allowing developers to prioritize significant concerns and make informed code modifications.

Johann, Stanik & Maalej (2017) introduced the SAFE technique, which extracts features from app descriptions and user reviews to better understand app functionalities. It preprocesses text data using POS analysis and sentence patterns, matching features with descriptions through term and semantic similarity matching, aiding app store analytics.

SURF (Summarizer of User Reviews Feedback), proposed by Di Sorbo et al. (2017), summarizes app reviews to help developers handle large volumes of feedback. It evaluates sentences based on relevance and frequency, generating structured summaries that categorize topics and intentions, thus providing insights into user needs.

Phong et al. (2015) developed the MARK framework, which uses tf-idf and the Vector Space Model to analyze keywords from app reviews. MARK ranks, clusters, and expands keywords, employing Word2Vec for effective grouping and k-means clustering, allowing analysts to identify significant user opinions and trends.

Jiang et al. (2019) presented the SAFER technique, which extracts features from app descriptions and recommends new features to developers. Combining data cleaning, linguistic rule filtering, cosine similarity, and classification, SAFER accurately identifies feature-describing sentences and uses LDA for similar app identification, outperforming other algorithms in feature recommendations.

The CLAP technique by Villarroel et al. (2016) aids in mobile app release planning by analyzing user reviews. It categorizes reviews into bug reports, feature suggestions, and more using Random Forest and prioritization techniques. CLAP enhances accuracy through n-gram extraction, stop word removal, stemming, and synonym unification. It helps developers make informed decisions for app release planning, validated by studies from Scalabrino et al. (2019), Liu et al. (2018), and Gomaa et al. (2020).

SUR-Miner, introduced by Gu & Kim (2015), assists developers in analyzing user reviews and understanding user preferences. It employs the Max Entropy algorithm to categorize review sentences into evaluations, bug reports, feature requests, and praise. Using part-of-speech (POS) tagging and pattern-based parsing with cascading finite state machines, SUR-Miner extracts aspect-opinion pairs based on semantic templates and Semantic Dependence Graphs (SDGs).

The User Request Referencer (URR) by Ciurumelea et al. (2017) categorizes mobile app reviews and suggests code enhancements using machine learning and information retrieval techniques. It employs feature extraction methods like n-grams and term frequency, as well as preprocessing steps such as stop word removal, punctuation elimination, and stemming, enabling the system to learn from reviews and classify them accurately.

Motger et al. (2024a) introduced T-FREX, a Transformer-based approach that uses large language models (LLMs) to extract features from app reviews. T-FREX fine-tunes LLMs on a named entity recognition task and implements a voting-based feature extraction mechanism, enhancing software engineering tasks related to mobile app development.

SRR-Miner, by Tao, Guo & Huang (2020), summarizes security-related user reviews in mobile apps. It uses a keyword-based method to extract security-focused sentences, identifying app misbehaviors and user opinions. SRR-Miner surpasses logistic regression with Bag of Words features, providing comprehensive insights into security concerns and user sentiments.

PUMA, developed by Vu et al. (2016), extracts user opinions from mobile app reviews using POS templates to identify phrases, cluster similar opinions, and track negative sentiments over time. By analyzing word sequences that convey specific meanings and feelings, PUMA offers detailed insights into user feedback.

RE-BERT, proposed by de Araújo & Marcacini (2021), combines local context word embeddings and deep neural networks to extract software requirements from reviews. It enhances rule-based methods and uses a multi-domain training strategy, outperforming methods like SAFE and ReUS by leveraging pre-trained models for semantic text representation.

Alves de Lima, de Araújo & Marcondes Marcacini (2022) developed MAPP-Reviews to analyze the temporal dynamics of software requirements from app reviews. Using contextual word embeddings based on RE-BERT, it captures word meanings in context, extracting relevant software requirements and identifying crucial trends in user feedback.

SIRA, introduced by Wang et al. (2022a), is a semantic-aware app review analysis method that extracts and visualizes problematic features from user feedback. SIRA combines a BERT+Attr-CRF model for detailed feature extraction with a graph-based clustering technique for summarizing common issues. Leveraging BERT’s contextual understanding and CRF’s sequence labeling, SIRA improves semantic capture compared to models like KEFE, Caspar, SAFE, and BILSTM-CRF. By incorporating review descriptions and attributes, SIRA accurately identifies and clusters problematic features, helping developers address concerns and enhance user experience efficiently.

DIVERSE, aids software developers in evolution tasks by analyzing sentiment and categorizing reviews based on feature-sentiment scores. It extracts verbs, nouns, and adjectives through POS tagging, identifies features with a collocation algorithm, and uses SentiStrength for sentiment analysis. A greedy algorithm retrieves diverse feature specimens, enabling developers to prioritize tasks and analyze varying opinions based on user feedback.

Chen et al. (2014) introduced AR-Miner, a tool designed to analyze user feedback and provide insights for app improvement. It employs LDA for feature extraction and group reviews based on rating, temporal fluctuations, and volume. The tool prepares data using text normalization, tokenization, and sentiment analysis. The EMNB algorithm classifies reviews as informative or uninformative, with EMNB-LDA outperforming EMNB-ASUM.

Panichella et al. (2016) developed ARdoc, combining NLP, text analysis (TA), and sentiment analysis (SA) to categorize valuable feedback in app reviews. ARdoc breaks down review text into sentences and extracts linguistic, structural, and sentiment features to classify reviews using the J48 algorithm. This tool supports developers in software maintenance and evolution tasks.

Gao et al. (2018b) introduced the IDEA framework, which analyzes app reviews to detect emerging issues. It includes preprocessing, LDA topic modeling, interpretation, and visualization. IDEA identifies abnormal topics as app issues, labeled by representative reviews. PMI extracts meaningful phrases from reviews, aiding in topic labeling and improving issue identification.

Iacob, Faily & Harrison (2016) developed MARAM, a tool for managing online reviews of mobile apps. It builds upon MARA by automatically extracting feature requests and bugs from reviews. MARAM allows developers to search through reviews and export selected items to platforms like GitHub, JIRA, or Bugzilla. It uses a linguistic rule-based feature extraction method to identify and abstract feature requests and bugs from review contexts.

The MApp-IDEA method, proposed by de Lima, Barbosa & Marcacini (2023b), enhances app review issue detection and prioritization. It incorporates sentiment analysis, expands issue detection scope, and efficiently manages smaller datasets. Using word embedding techniques to create app-related problem graphs, MApp-IDEA identifies and ranks issues from reviews, even with limited data. The method’s performance was assessed using various classifiers and F1-Score as the metric. The MApp-IDEA dashboard provides real-time reports on issue trends, updates, and identified problems.

The SOLAR tool, developed by Gao et al. (2023), automates the summarization of valuable user reviews for developers. It uses a review helpfulness prediction model to filter unhelpful reviews, group topics with sentiment analysis, and prioritize reviews through a multi-factor ranking system. Techniques like POS tagging, readability assessment, sentiment scoring, and content analysis using n-gram and TF-IDF are employed for feature extraction. Compared to AR-Miner (Chen et al., 2014) and IDEA (Gao et al., 2018b), SOLAR performs better in summarizing helpful user reviews.

CASPAR, a tool by Guo & Singh (2020), extracts and consolidates user stories from app reviews using NLP. It identifies and isolates user issues and suggestions by focusing on verbs and employing POS tagging. CASPAR distinguishes event phrases, enabling the extraction of crucial user actions and app issues. Integrating NLP techniques, CASPAR simplifies app review analysis, offering developers valuable insights to enhance app performance.

GuMa, introduced by Guzman & Maalej (2014), automatically extracts app features and analyzes sentiments from user reviews. It combines collocation finding, lexical sentiment analysis, and topic modeling to generate summaries. GuMa identifies key app attributes by examining user reviews, such as functionalities and technical aspects, helping developers understand user opinions and plan future app releases. Evaluation results show high precision and recall rates.

Dragoni, Federici & Rexha (2019) introduced ReUS, an unsupervised aspect extraction method for monitoring real-time review streams without manual annotation. Using linguistic rules and Open Information Extraction (OpenIE), ReUS focuses on identifying significant aspects like product features for further analysis. This method automatically detects aspects and associated opinion words to infer polarities based on context.

The RE-SWOT method, by Dalpiaz & Parente (2019), efficiently gathers app requirements from reviews by analyzing competitor data. It leverages NLP algorithms to automatically extract features from reviews, enhancing the efficiency of requirements elicitation.

FENL, a semi-automated approach by Bakar et al. (2016), uses NLP to extract software features from online reviews. Techniques like WordNet lemmatization, POS tagging, and TF-IDF are employed to automate phrase extraction, aiding domain analysts in reusing requirements by generating valuable feature lists.

Wu et al. (2021) introduced KEFE, a method for identifying crucial features in mobile apps that impact ratings. By analyzing app descriptions and user feedback from the Chinese Apple App Store, key feature descriptions are extracted and assessed for their impact. A feature classifier is initially trained on a subset of apps, followed by preprocessing and phrase extraction to identify key phrases from descriptions. These phrases are manually reviewed to ensure accurate feature extraction, offering developers insights into essential functionalities that boost app ratings and outperform methods like SAFE and SAFER.

Xiao et al. (2020) introduced DSISP, an automated framework that detects user intentions by analyzing sentiment-preference correlations. DSISP extracts sentiment scores and preferences from app reviews over time by combining sentiment analysis and NLP techniques. It monitors users’ evolving intentions and identifies significant sentiment shifts, demonstrating a high precision of 0.962 in real-world data evaluations, allowing developers to enhance their apps effectively.

Palomba et al. (2017) presented ChangeAdvisor, which uses user reviews to suggest and localize change requests in mobile apps. By connecting user feedback clusters to code components through feature extraction and clustering techniques like LDA, LDA-GA, and HDP, ChangeAdvisor identifies and links relevant feature requests to code components, prioritizing essential terms and phrases in user reviews.

Zhou et al. (2020) introduced RISING, an automated system incorporating user feedback into software development. Using domain-specific constraints and ARDOC, RISING groups reviews based on shared user demands. It employs the Vector Space Model and principal component analysis to analyze word vectors, identifying files for user requests with COP-Kmeans and BoW, demonstrating higher accuracy than ChangeAdvisor.

Gao et al. (2019a) introduced DIVER, a tool that identifies emerging app issues from user feedback. DIVER processes review extracts relevant word collocations, categorizes issues by topics, and visually presents them for developers. It enhances context understanding and has significantly improved over IDEA in evaluation experiments.

Gao et al. (2021) introduced MERIT, combining topic modeling and sentiment analysis to identify critical app review concerns. Using features like Biterm Sentiment-Topic (BST) and Joint Sentiment/Topic Model (JST), MERIT analyzes user feedback, integrating PMI to pinpoint significant phrases based on co-occurrence frequencies. MERIT has outperformed baseline methods like IDEA and OLDA in issue detection metrics.

Wang et al. (2022b) proposed UISMiner, a technique to extract UI suggestion information from user reviews to improve app GUIs. UISMiner trains a classifier using review attributes, linguistic information, semantic relations, and sentence patterns. Key contributions include summarizing classification factors, establishing extraction rules, and evaluating UISMiner’s effectiveness through Google Play experiments. Results show UISMiner’s superiority over SAFE in extracting UI suggestions, with developers confirming its usefulness for updating app interfaces.

Gao et al. (2015b) developed the PAID framework to automatically prioritize app issues based on user reviews across different versions. The framework extracts key phrases, creates a Phrase Bank, and uses topic modeling to rank phrases with True Mutual Information (TMI). Extraction rules include Length Limit, Informative Assurance, Part of Speech Limit, and Quality Assurance. The ranked Phrase Bank is presented to developers for prioritization, and results are visualized using ThemeRiver. Evaluation with app changelogs showed high precision in matching prioritized issues.

Wei, Liu & Cheung (2017) introduced OASIS, a method for prioritizing Lint warnings in Android apps by integrating user reviews. OASIS connects static analysis results with user feedback, improving the identification of critical issues affecting user experience. The method uses a semantics-aware similarity calculation to associate Lint warnings with user reviews, capturing relationships between warnings and feedback to enhance prioritization performance.

Gao et al. (2015a) introduced AR-Tracker, which gives developers insights into user feedback by extracting and visualizing main themes from app reviews. The tool compares topic modeling algorithms like LSI, LDA, RP, and NMF to mine user reviews effectively. By tracking feedback dynamics, AR-Tracker helps developers prioritize areas for improvement. A large-scale experiment involving 500 k reviews demonstrated its effectiveness in understanding user demands and visualizing results for straightforward interpretation.

Gao et al. (2018a) introduced INFAR, which helps developers make informed decisions by extracting insights from app reviews. INFAR identifies key topics, abnormal subjects, topic correlations, and factors influencing ratings. It preprocesses and categorizes reviews based on predefined topics, presenting insights visually and in natural language. Evaluated with six popular apps and 12 experts, 92% found INFAR’s insights valuable.

Man et al. (2016) introduced CrossMiner, a framework that analyzes app issues from user reviews across platforms like Google Play, App Store, and Windows Store. It extracts keywords using the Word2Vec model to identify user perceptions. By representing words with 300-dimensional vectors, CrossMiner reflects user concerns and identifies platform-specific issues, offering developers insights to enhance app testing and user satisfaction.

CIRA, a causal impact analysis tool for Google Play app releases, was introduced by Martin et al. (2017). This tool identifies key factors in app launches and evaluates the effectiveness of causal impact analysis for developers. By analyzing prices, ratings, descriptions, and app updates, CIRA reveals patterns between app features and user ratings. It addresses imprecise ratings on Google Play by calculating precise average ratings based on the rating distribution, offering a more accurate representation for analysis. This method sheds light on the effects of app releases on user ratings and their underlying characteristics through feature extraction and analysis.

Zahoor & Bawany (2023) introduced CEAR, a framework for automating the analysis of educational Android app reviews. They collected and categorized 13,000 user reviews from 25 apps, examining the effects of data preprocessing and class balancing on various machine-learning techniques. Utilizing TF-IDF for feature extraction, they efficiently trained models by identifying crucial words from reviews and tested eight machine learning methods, employing LIME to explain model decisions.

Panichella & Ruiz (2020) developed Requirements-Collector, an automated tool that classifies software requirements using machine learning and deep learning methods. It includes Requirement-Collector-DL-Component and Requirement-Collector-ML-Component. The authors preprocess text data with feature extraction techniques, converting raw text into numerical formats like Bag of Words, TF-IDF, Word Embeddings, and N-grams, enhancing model accuracy and streamlining manual tasks to improve software quality in early development stages.

Bakiu & Guzman (2017) introduced a method to gauge user satisfaction with software features through user experience (UUX) analysis. This approach involves sentiment analysis, identifying review features, and categorizing UUX dimensions. It filters essential nouns, verbs, and adjectives, removes irrelevant terms, and uses a collocation algorithm to pinpoint feature descriptions. A preliminary evaluation of video game and software reviews identified UUX issues and strengths, aiding developers, UUX designers, and researchers in enhancing software applications based on user feedback.

Tan et al. (2023) introduced STRE, an automated tool that analyzes historical data to suggest when app developers should stop reading reviews. It preprocesses reviews, identifies topics, and extracts the longest TSTs using LDA to pinpoint significant textual similarities among reviews. This helps developers focus on common themes, sentiments, and issues to prioritize feedback, make informed decisions, and improve apps efficiently.

Hadi & Fard (2023) introduced the Adaptive Online Biterm Topic Model (AOBTM) for short text analysis in mobile app reviews and Twitter data. AOBTM improves traditional topic modeling by adapting to short texts and utilizing historical statistical data. Using Pointwise Mutual Information (PMI) prioritizes phrases with higher co-occurrence frequencies. Evaluated using Precision, Recall, and F-hybrid metrics, AOBTM outperforms existing methods like IDEA in identifying coherent and distinct topics.

de Lima, Barbosa & Marcacini (2023a) introduced an OPT-based method to analyze mobile app reviews for prioritizing software maintenance tasks. The approach uses dynamic prompt generation and open pre-trained language models to automate feature extraction and evaluate risk impact. Utilizing the OPT-6.7b model, it extracts features and generates prompts based on cross-domain information, identifying key aspects mentioned by users and prioritizing maintenance actions. Experimental results show that the OPT-based approach outperforms traditional methods (GuMa, SAFE, ReUS) and large language models (RE-BERT) in producing risk matrices, highlighting the potential of OPT models for software maintenance, especially in resource-constrained environments and regarding user data privacy.

Assi et al. (2021) developed FeatCompare, a tool for analyzing high-level features in mobile apps by mining user reviews. It includes a data preprocessor, GLFE neural network-based feature extractor, and review aggregator. GLFE, based on the ABAE model, distinguishes global and local features in reviews to effectively compare apps. Analyzing 14,043,999 reviews from 196 Android apps, FeatCompare demonstrated high precision and recall rates. A user study with 107 developers verified its usefulness for competitor analysis.

BECLoMA, introduced by Pelloni et al. (2018), enhances stack traces with user reviews for better bug diagnosis and fixing in mobile apps. It combines a machine learning classifier, testing tools, and Google Play Store user feedback to link stack traces with relevant reviews, streamlining testing and bug resolution.

IETI, proposed by Zhou et al. (2022), introduces Enhanced Emerging Topic Identification (EETI) to detect emerging topics in app reviews. Using natural language processing and the adaptive online biterm topic model (AOBTM), it minimizes noisy data and extracts relevant phrases. Employing pointwise mutual information (PMI), EETI evaluates word pairs to derive meaningful phrases. The study shows that EETI’s accuracy is higher than IDEA and OLDA on six popular apps.

Gallego Marfa et al. (2023) introduce TransFeatEx, an innovative tool that integrates established syntactic and semantic feature extraction techniques with a RoBERTa-based pre-trained model. This combination allows for efficient automatic extraction of app features from diverse textual sources. TransFeatEx is designed as a flexible, customizable pipeline, enabling researchers and developers to fine-tune the tool for specific domains.

Each approach leverages various techniques to support different aspects of software requirements analysis, including summarization, information retrieval, visualization, recommendation, and information extraction. The selection and adoption of these tools depend on factors such as the characteristics of the input data, project requirements, and available resources. Software requirements analysis has witnessed the development of many software tools and frameworks, each offering unique capabilities and leveraging different approaches to support the implementation of feature extraction techniques. As the field continues to evolve, there is a need for more robust, scalable, and context-aware tools that can effectively extract relevant requirements from diverse data sources while incorporating human expertise and domain knowledge.

RQ3: How do the app review analysis tools compare their performance, scalability, and user-friendliness?

Analyzing and comparing the performance, scalability, and user-friendliness of feature extraction techniques and tools is essential for guiding researchers and practitioners in choosing the best approach for their needs. RQ3 examines 12 identified methods for extracting features from mobile app reviews, assessing their accuracy through metrics like precision, recall, F1-score, and overall accuracy reported in the studies.

Comparison of developed tools in terms of their performance

Researchers employ various validation methods to assess the reliability and effectiveness of their proposed tools or approaches for feature extraction and sentiment analysis. Quantitative metrics commonly include precision, recall, F1-score, accuracy, MAPE, MCC, Hit Ratio, NDCG, PMI-Score, and confusion matrices. These metrics measure how well a tool identifies features or sentiments in text data. Researchers also employ topic coherence evaluation metrics like PMI-Score and Discreteness Score (Dis Score). PMI-Score assesses topic coherence using point-wise mutual information from large text datasets like Wikipedia, calculated from the top 10 terms of each topic. This Score evaluates the distinctiveness of topics based on semantic similarity mapping. Higher PMI-Score and Dis-Score values indicate more coherent and distinctive topics. By analyzing performance metrics and conducting comparative analyses, researchers demonstrate the effectiveness of their proposed methods and tools, ensuring reliability and accuracy in feature extraction and sentiment analysis tasks. The equations of these metrics are illustrated in Table 7. In addition to quantitative evaluations, comparative analyses with state-of-the-art baselines are conducted for functions such as feature extraction, issue detection, classification, feature recommendation, and summarizing user reviews, as shown in Fig. 7. A total of 12 studies have compared their tools or approaches with baselines like SAFE (used in six studies) and IDEA (used in five studies) as shown in Fig. 8. Meanwhile, some studies compared their approach to three to four state-of-the-art baselines. OPT-based approach (de Lima, Barbosa & Marcacini, 2023a) has been compared to ReUS, RE-BERT, SAFE, and GuMa. Also, RE-BERT (de Araújo & Marcacini, 2021) has been compared to SAFE, ReUS, and GuMa.

Table 7:

Evaluation metrics.

Formula	Evaluation metrics
Hit Ratio	Hit Ratio = (# of hit features/# of features) × 100%
NDCG	NDCG = G/Ideal G; G = Σ (2 scorei/log2 (i + 1))
PMI-Scores	PMI-Score(k) = (1/T(T − 1)) Σ1 ≤ i < j ≤ T log (P(wi, wj)/(P(wi) * P(wj))) Here, P(wi), P(wj), and P(wi, wj) represent the probabilities of word wi, wj, and the co-occurring word-pair (wi, wj) respectively.
Dis score	Dis_Score = (Σk = 1K ((Σj = 1k Djs(ϕk∥ϕj*))/k))/K;
	Djs(ϕk∥ϕj*) = 1/2 DKL(ϕk∥M) + 1/2 DKL(ϕj∥M);
	DKL(P∥Q) = Σi P(i) log (P(i)/Q(i)); M = 1/2 (ϕk + ϕj);
Precision, Recall, F-Measure, and Accuracy	Precision = TP/(TP + FP)
	Recall = TP/(TP + FN)
	F-Measure = (2 × Precision × Recall)/(Precision + Recall)
	Accuracy = (TP + TN)/(TP + TN + FP + FN)

DOI: 10.7717/peerj-cs.2401/table-7

Several studies have evaluated the performance of various tools and approaches for feature extraction, issue detection, and topic modeling from user feedback and app reviews. MERIT outperforms AOLDA and IDEA in precision, recall, and F1-score for identifying emerging app issues, with improvements ranging from 20.9% to 22.3%. It achieves an average precision, recall, and F-score of 81.4%, 81.2%, and 80.9%, respectively, showing better balance than baselines. Additionally, MERIT outperforms IDEA and OLDA in the F-hybrid score. RISING (Zhou et al., 2020) demonstrates superior Likert scale values for feature requests and problem discovery categories compared to CHANGEDVISOR. On average, ChangeAdvisor scores 2.07 and 1.94 in these categories, while RISING scores 4.20 and 4.26. RISING also shows better clustering and localization accuracy in Top-k hitting. Accuracy for feature requests improves from 44.64% to 76.74% (Top-1), 70.54% to 91.77% (Top-3), and 76.65% to 98.00% (Top-5). In the problem discovery category, accuracy improves from 48.50% to 76.04% (Top-1), 65.08% to 93.84% (Top-3), and 76.00% to 98.04% (Top-5) on average. RISING also outperforms RISING, highlighting the significance of commit messages in enhancing localization performance.

DIVER outperformed IDEA significantly, showing an average improvement of 29.4% in precision and 32.5% in recall for detecting emerging app issues (Gao et al., 2019a). This suggests DIVER’s effectiveness in analyzing user feedback. Similarly, SAFER demonstrated advantages over CLAP in all categories, achieving a 78.27% Hit Ratio in the Business Category compared to CLAP’s 60.00%. On average, SAFER surpassed CLAP by 23.54% in Hit Ratio and 0.1522 in NDCG. Statistical analysis using the Wilcoxon test confirmed the significant difference, with p-values of 0.003 for Hit Ratio and 0.009 for NDCG, indicating SAFER’s superior performance in recommending features from mobile app descriptions.

Furthermore, AOBTM (Hadi & Fard, 2023) exhibited superior performance compared to IDEA in metrics like precision, recall, and F-hybrid, with higher PMI scores indicating more comprehensive and coherent topics. AOBTM demonstrated improved topic quality and coherence, with an accuracy of 0.593, recall of 0.619, and F-hybrid score of 0.608, outperforming IDEA. These comparative evaluations highlight the effectiveness of tools like DIVER, SAFER, and AOBTM in extracting features, detecting issues, and modeling topics from user feedback and app reviews, surpassing the performance of baselines like IDEA and CLAP.

KEFE, as presented by Wu et al. (2021), efficiently extracts features from Chinese app descriptions and user reviews, surpassing SAFE and SAFER. In a test with 1,108,148 reviews from 200 apps, KEFE achieved 82.49% precision and 74.83% recall. Although SAFER had the highest recall at 80.27%, KEFE demonstrated superiority in the Tool, Travel, and News app categories, averaging 78.13% for feature extraction and 62.02% for review matching. SAFE exhibited lower accuracy at 74.83% and 43.40% compared to SIRA, outperforming other tools like KEFE, SAFE, and CASPAR with 84.27% precision and 85.06% recall. Although SAFE’s high recall rate of 73.94% using PoS patterns for feature-related phrases, its precision rate of 15.51% falls short. CASPAR identifies events in reviews with temporal conjunctions, while KEFE utilizes a BERT classifier, but its accuracy is affected by pattern-based methods.

SOLAR outperforms competitors, prioritizing 85% of 11,659 reviews across five apps. It excels in semantic relevance to topics compared to AR-Miner, IDEA, and itself. IDEA’s focus on online reviews and requirement for multiple historical versions hinder its performance. SOLAR enhances precision, recall, and F1 score by 10.41%, 12.75%, and 28.49%, respectively, surpassing baseline methods. This demonstrates that SOLAR-prioritized reviews are more informative and compelling for filtering out irrelevant data in downstream tasks. UisMiner excels at extracting UI-related suggestions with 77.50% precision and 76.50% recall, but it focuses solely on user interfaces.

In contrast, SAFE handles a broader spectrum of functionalities in reviews. The comparison between UisMiner and SAFE does not definitively declare one superior. Furthermore, SIRA, developed by Wang et al. (2022a), demonstrates remarkable efficiency in identifying problematic app features from reviews.

The OPT-based approach by de Lima, Barbosa & Marcacini (2023a) demonstrates promising feature extraction results, achieving an F1 score of 45.5% and competing with rule-based methods like GuMa, SAFE, and ReUS. Compared to RE-BERT, with an F1 score of 63.5%, the OPT-based proposal yields inferior results from 363,843 user reviews across eight Android apps. RE-BERT excels at generating semantic textual representations by focusing on software requirement tokens’ local context rather than open-access language models. In constructing risk matrices, the OPT model exhibits lower error in the impact dimension than the proprietary GPT model, suggesting its potential for accurate risk matrix generation. T-FREX, a Transformer-based model (Motger et al., 2024a), consistently surpasses SAFE in all metrics. It significantly enhances performance when upgrading from BERT to XLNet. While BERTbase recall (0.300) is slightly lower than SAFE’s by 0.021, XLNetbase remarkably improves performance over BERTbase, particularly for recall (+0.117 for out-of-domain, +0.153 for in-domain). In both in- and out-of-domain evaluations, T-FREX showcases adaptability, generalization, and accurate prediction of new features within specific domains for which it has been fine-tuned. Its effectiveness and adaptability stem from Transformer-based models, actual user annotations, and iterative refinement processes, enabling it to extract features from mobile app reviews efficiently (Zhang et al., 2023; Huang et al., 2024). The study by Zhou et al. (2022) reveals that IETI effectively identifies emerging app review topics, outperforming baseline methods IDEA and OLDA. IETI exhibits enhanced precision, Recall, and F1 scores, demonstrating its superiority in phrase and sentence label evaluations. Compared to IDEA, IETI improves precision by 0.094, recall by 0.107, and F1 score by 0.126 in phrase labels. In sentence labels, IETI shows advancements with a 0.068 increase in Precision, 0.025 in recall, and 0.061 in F1 score.

Comparison of developed tools in terms of their scalability

Scalability is crucial when evaluating feature extraction techniques for analyzing large volumes of user reviews from popular app stores. This mapping study revealed that different tools and techniques vary in their scalability characteristics, impacting their effectiveness for large-scale or real-time applications. Figure 9 shows that only two tools, UisMiner and KEFE, have demonstrated the ability to process more than 500,000 user reviews efficiently. KEFE, for example, successfully handled a dataset of 200 Chinese app descriptions and over 1 million user reviews, highlighting its scalability. Similarly, UisMiner showed promise in efficiently handling and analyzing UI suggestions from 651,981 reviews across 5,467 apps.

Additionally, six tools were found to handle less than 500,000 user reviews, making them ideal for managing data from popular apps. For instance, SIRA’s method (Wang et al., 2022a) uses a BERT+Attr-CRF model for feature extraction and a graph-based clustering method for summarization, showing its ability to handle 318,534 reviews from 18 apps. The OPT-based approach (de Lima, Barbosa & Marcacini, 2023a) employs open-access language models and dynamic prompt generation to analyze 363,843 reviews from nine apps efficiently. Its few-shot learning feature reduces the need for extensive labeled data, enhancing scalability in scenarios with limited data or dynamic reviews. The DIVER tool (Gao et al., 2019b) is more scalable than the IDEA approach (Gao et al., 2018b). DIVER can process thousands of user reviews in seconds, making it suitable for managing large data volumes from popular apps. Despite increasing data sizes, DIVER’s performance remains efficient, ensuring consistent results even with growing data volumes.

SOLAR, T-FREX, RISING, and AOBTM have all shown scalability in analyzing small datasets with fewer than 100,000 user reviews. AOBTM’s scalability proves beneficial in managing growing data volumes and complexities, enabling efficient topic modeling. T-FREX’s iterative refinement process allows adaptation to new domains and continuous performance improvement by integrating crowdsourced features and recent app reviews, enhancing scalability. RISING utilizes domain-specific constraints and semi-supervised learning to create fine-grained user review clusters, resulting in more stable and deterministic clustering than ChangeDvisor for large-scale analysis tasks.

RQ4: What are the significant strengths and limitations observed in current techniques based on their methodology or evaluation results?

To investigate RQ4, we thoroughly analyzed both automated and semi-automated tools illustrated in Table 7. Figure 10 compares these various feature extraction tools based on their scalability, performance, and user-friendliness. KEFE and UisMiner demonstrate superior scalability, efficiently processing over 500,000 user reviews. These tools excel in managing large volumes of data, making them suitable for analyzing popular apps with extensive user feedback. Performance varies among the tools, with SIRA, T-FREX, and RE-BERT showing high effectiveness in extracting features and requirements. These tools often outperform baselines in precision, recall, and F1-score metrics. For instance, T-FREX consistently surpasses SAFE across all metrics, showcasing its adaptability and accurate prediction of new features within specific domains. MERIT’s visualization interface for emerging app issues has garnered positive feedback from developers, while DIVER aids in prioritizing issues through word clouds and line charts. The OPT-based approach enhances user-friendliness by providing customized instructions for review analysis through dynamic prompt generation.

Despite their progress in extracting features and requirements from app reviews, these tools face several limitations and challenges. These include the need for manual intervention or labeled data, difficulties with complex linguistic structures and context-dependent information, scalability issues, limitations in evaluation methods, interpretability challenges, generalizability concerns, and effectively incorporating user feedback and domain knowledge. Addressing these limitations is essential to enhance the practical application of these techniques in software requirements analysis and related fields. Many tools rely on manual intervention or labeled data for training and evaluation. For instance, the KEFE approach requires manually reviewing and labeling key features in app descriptions, which is time-consuming and subjective. Similarly, the RE-BERT method and other supervised learning techniques need high-quality annotated datasets, which can be challenging and resource-intensive in rapidly evolving or niche areas. Additionally, some methods struggle with complex linguistic structures, context-dependent information, or domain-specific terms. For example, the SAFE approach primarily uses part-of-speech patterns and predefined rules, potentially missing the nuances of natural language in user reviews. This leads to inaccurate feature or requirement extraction in scenarios with ambiguous, colloquial, or domain-specific language. Specific tools face scalability issues when handling large volumes of user reviews or real-time data streams. While tools like SOLAR and RISING have effectively managed extensive datasets, other methods may struggle with massive data volumes or real-time processing requirements.

The evaluation methodologies and metrics used to assess these techniques may not capture all aspects of their effectiveness or practical applicability. Standard metrics like precision, recall, and F1-score may not fully reflect the usefulness or relevance of the extracted features or requirements in real-world scenarios. Additionally, comparative evaluations against baseline approaches or state-of-the-art techniques may not comprehensively understand a technique’s limitations or challenges in specific contexts or application domains.

Some methods face interpretability or transparency challenges, particularly those relying on complex machine-learning models or black-box approaches. This lack of interpretability can hinder developers’ or analysts’ ability to understand the reasoning behind the extracted features or requirements, limiting trust in the results and hindering the adoption of these techniques in critical decision-making processes.

The generalizability of specific techniques across different domains, platforms, or languages may be limited. Techniques tailored for particular domains or languages may not perform well with user reviews from other domains or in different languages, necessitating additional adaptation. Moreover, incorporating user feedback and domain knowledge during feature extraction remains challenging. While some methods, like SAFER and RISING, integrate domain-specific constraints or semi-supervised learning, further exploration and improvement are needed to combine human expertise and contextual information effectively.

RQ5: What future research directions could address current gaps in capabilities for efficient and precise analysis of app reviews for requirements?

Emerging research directions aim to enhance app review analysis for requirements (RQ5). Future studies should focus on developing resilient and scalable methods to manage diverse and evolving user data while integrating human insights and domain knowledge. As mobile applications expand, analysis techniques must adapt to new domains, platforms, and languages without extensive retraining or manual effort, ensuring effective and accurate user requirement capture across various apps.

Improving the interpretability and transparency of feature extraction models is essential for fostering trust and facilitating adoption in software development. Developers and stakeholders must understand how requirements are extracted from user reviews and how they inform decision-making. Explainable AI techniques, such as model visualization and feature importance ranking, can make these models more accessible to non-experts (Chatzimparmpas et al., 2020).

Human-in-the-loop approaches could enhance model interpretability and transparency. By involving domain experts in the analysis process, researchers can ensure that the extracted requirements are meaningful and relevant to software development. These approaches can also help identify and correct model or underlying data biases (Mosqueira-Rey et al., 2023).

Additionally, research should focus on developing comprehensive evaluation frameworks that assess feature extraction’s accuracy and practical relevance. Techniques must be technically accurate and provide actionable insights that improve software development quality. Evaluation frameworks should include relevance, usefulness, and impact metrics to ensure the extracted requirements are valuable to developers and stakeholders.

Overall, future research in app review analysis should bridge the gap between technical accuracy and practical relevance. By developing robust, scalable, interpretable, and relevant techniques, researchers can address current gaps and drive the adoption of user review analysis in real-world settings.