DiscourseDash: designing and evaluating a tool for analysing social media discourses to support decision-making

Providing a visual overview

The overview provides users with a high-level summary of trending discourse topics and their implications. It includes the following components:

• Trending Topics: We use clustering techniques to identify the most actively discussed topics within a selected time window. This surface-level aggregation enables users to quickly understand which issues are dominating public discourse and assess their alignment or conflict with organisational goals.

• Harmfulness: Each topic from the Trending Topics is assessed for its potential level of harm, based on the proportion of posts containing toxic, hateful, or otherwise harmful language. This classification helps users allocate priorities by identifying those that may require immediate attention or intervention.

• Stance: We display the distribution of stances (favour, neutral, against) within each topic. We hypothesise that understanding the stance distribution on topics previously categorised as harmful can help gauge the urgency of a topic. For example, if a highly harmful topic receives predominantly favourable stances, it can inform the prioritisation of further investigation.

• Sentiment: Sentiment (positive, neutral, negative) provides an additional emotional context to each topic. This helps users understand public emotions and make informed decisions about countermeasures.

• Popularity: Engagement metrics over time reveal the relevance and impact of topics. Visual graphs showing the popularity of each topic help users understand the dynamics of discourses and identify emerging trends or sudden spikes that may indicate concerning impacts.

Further detailing the analyses

Building on the overview, users can select specific topics to investigate in detail, exploring analyses accompanied by discourse through the following features:

• Summary An in-depth view of the selected topic, including visual graphs of stances and sentiments. This provides clarity and supports users in unfolding the progression and influence of different viewpoints within the discourse.

• Discourses We present all the discourses on a selected social media platform that reflect the topic under investigation. The goal is to support effective countermeasures by providing a detailed view of the stances and sentiments labelled within each text in the discourse. It also facilitates the extraction of specific contexts and dynamics revealed through the analysis. Presenting these discourses in detail, accompanied by visuals and interactive elements, is a key design goal aimed at enhancing transparency and explainability.

Recruitment and participants

Participants were recruited from the university research groups, including professors, researchers, and students. No further screening was applied. Participants were assigned to interact with the conditions in a counterbalanced order. To respect participant privacy and because demographic details were not relevant to the study’s aims, detailed demographic data were not collected. This aligns with prior evidence suggesting that participants may be uncomfortable sharing such information (Frederick, 2021). Participant codes (e.g., G1) serve as anonymised identifiers, where “G” denotes a general user. A complete list of participant codes, their assigned counterbalanced condition orders, interview dates, and anonymised excerpts from the transcripts is available in the dataset.

Study design

We employed a within-subjects design where participants completed three predefined tasks using two experimental conditions in counterbalanced order: (1) Reddit as a baseline, and (2) DiscourseDash. After each task, semi-structured interviews were employed to gather qualitative feedback, followed by questionnaires to quantify user experience. Following the completion of both conditions, participants engaged in a final interview to compare their experiences and provide feedback for improvements. This approach enabled an evaluation of DiscourseDash’s effectiveness in supporting decision-making, the usefulness of its features, and opportunities for user-centred enhancements.

Baseline justification: Baseline selection should reflect realistic, representative user practices (Purchase, 2012). Manual exploration of unstructured social media is common among experts such as journalists (Humayun & Ferrucci, 2022; Hellmueller et al., 2024). Thus, an unstructured social media platform serves as a realistic analogue. Reddit was chosen for its recognised open discourse structure: threaded comments, subReddits, upvotes, and downvotes (Hollender et al., 2010). Compared to Twitter/X or Facebook, Reddit’s longer discussions, clear hierarchical threads, and topic-focused communities make it suitable as a baseline for this study. We acknowledge Reddit may not capture all interaction types of other platforms. Yet, its open, community-driven nature represents unstructured social media well. For clarity, we define a root-level comment as equivalent to a Reddit post, and a child-level comment as equivalent to a Reddit reply (see Page 7).

Procedure

Participants were provided with an initial briefing on the research topic and the study’s objectives, which focused on evaluating the effectiveness of DiscourseDash in analysing online discourse. To minimise bias, the comparison between conditions was not disclosed. Each participant experienced two counterbalanced conditions: (1) baseline Reddit and (2) DiscourseDash. Participants began by familiarising themselves with the assigned condition. Drawing from virtual experience methodologies, a realistic narrative was provided to enhance ecological validity (Gorini et al., 2011). Participants assumed the role of a fact checker tasked with identifying newsworthy topics and crafting impactful articles, inspired by journalistic practices (Humayun & Ferrucci, 2022; Hellmueller et al., 2024). This scenario helped link DiscourseDash’s metrics with its intended impact (Lam et al., 2011). Then, participants completed three separate, predefined tasks specifically designed to reflect these realistic workflows. These tasks are detailed in File S1, Set A. During these tasks, participants used the think-aloud protocol to verbalise their thought processes (Jaspers et al., 2004). After each task, a semi-structured interview and questionnaire assessed their experience. This procedure was repeated for the second condition. The study concluded with a final semi-structured interview comparing both conditions to identify preferences and gather feedback. Participants received a £10 Amazon voucher for approximately 60 min of participation. Figure 9 illustrates the study conditions and procedure.

Data collection and analysis

We used a mixed-method approach (Creswell, 2021) to gather both quantitative and qualitative data. Quantitative data provided measurable insights into participants’ interactions and perceptions, while qualitative data helped explore underlying reasons. All audio recordings were transcribed via Microsoft Teams, then manually reviewed to ensure accuracy. The transcriptions were then extracted and organised by questions, as shown in the provided dataset, allowing for a structured mapping of participant responses to facilitate the analysis process.

Recruitment and participants

Participants were recruited through existing networks, direct outreach on LinkedIn, and targeted recruitment via Prolific (See File S5). The final sample includes 11 participants with diverse roles, responsibilities, and sector experiences. While most were UK-based, some had experience in Japan, Nigeria, and Europe. Initial expert selection was based on predefined criteria (see File S4, SET A), excluding years of experience as a sole indicator of expertise, since prior research shows it does not reliably reflect skill or judgment (Ericsson et al., 2018). Instead, we prioritised domain knowledge, contextual understanding, and reflective insights. Participants lacking these qualities during the interview were excluded from the final sample. Participant codes (e.g., E1) serve as anonymised identifiers, where ‘E’ denotes an expert. A complete list of participant codes, their profession, interview dates, and anonymised excerpts from the transcripts is available in the dataset.

Study deign

Our interview study employed an inductive approach to explore how experts interpret social media discourses and their needs for supportive tools. This allowed us to gather insights through pre-defined questions while allowing flexibility for emergent themes. The interview questions were divided into three parts. The first explored participants’ professional roles and workflows, including decision types, methods for analysing social media discourses, and workflow challenges. The second focused on their perceptions of the conceptual dashboard prototype, asking which features stood out and what was missing. The third focuses on practicality within experts’ workflows, inviting use case scenarios, potential barriers, and suggestions. Full question sets are provided in (see File S4, set B). All questions were open-ended, supplemented with prompts for clarity and coverage of key themes.

Procedure

The interview began with a brief introduction outlining the study goals and assuring participant confidentiality. Given the small sample size and focus on high-quality insights, a short demographic questionnaire was included to ensure diversity and contextualise perspectives. The interview comprised three stages. First, to establish a baseline, participants answered questions about their engagement with social media discourses in decision-making, current tools and methods, and their limitations (see File S4, set B: B1). Next, participants interacted with a working prototype of the DiscourseDash tool via a shared screen. Participants first explored the Overview page, followed by perception-focused questions regarding the design’s relevance and any missing elements relative to their workflow (see File S4, set B: B2). This process was repeated for the Detailed Analysis page to gather perception and relevance on DiscourseDash’s feature in providing in-depth discourse insights. Finally, participants reflected on their overall experience with DiscourseDash through examples of real-world use cases, adoption barriers, and integration considerations (see File S4, set B: B3). Participants received £20 Amazon vouchers for approximately 45 min of participation. Figure 10 illustrates the study design and procedure.

Data collection and analysis

All interviews were transcribed using Microsoft’s automated transcription service immediately after each session and manually reviewed for accuracy against the original audio. Relevant responses were extracted and organised according to interview questions, as reflected in the provided dataset. We applied the same thematic analysis approach as described above (See 14, Qualitative analysis), using inductive coding to structure the data by interview sections. Then, broader themes were developed through iterative team discussions to illustrate patterns aligned with our research objectives.

Comparative analysis of the ease of prioritisation

We conducted a Wilcoxon Signed-Rank test to compare the ease of identifying a topic between the Reddit baseline and DiscourseDash. The analysis revealed a Z statistic of $Z=-3.928$ and a p-value of $p<0.001$. These results were statistically significant, suggesting that users perceive DiscourseDash as making it easier to prioritise topics for creating impact compared to the Reddit baseline.

Evaluating the perceptions of the usefulness of dashboard features

To illustrate the Likert scale agreement by participants, we present our results in a stacked bar chart (see Fig. 11). Observing the proportions of “Agree” and “Strongly Agree”, we can see that these categories occupy the majority, indicating the relevance of our proposed features in achieving decision-making tasks.

Comparative analysis of the perceptions of the usefulness of dashboard features amongst participants

Additionally, a Friedman test was conducted to compare the participants’ perceptions of the usefulness of features. The features evaluated are the harmfulness label (M = 2.56), stance detection (M = 2.90), sentiment analysis (M = 3.40), popularity analysis (M = 3.25), and discourse investigation (M = 2.88). Although sentiment analysis received the highest mean rank, the test did not reveal a statistically significant difference in perceived usefulness among the features, ${\chi }^2(4)=7.295$, $p=0.121$. This suggests that participants rated the usefulness of these features similarly, with no feature standing out as significantly more or less useful than the others.

Preliminary usability testing outcomes

DiscourseDash achieved a SUS score of 78.9, which is well above the commonly accepted usability benchmark. This suggests that participants found DiscourseDash to be usable and accessible. This result reflects positive user perceptions of the system’s ease of use and overall design suitability for achieving the tasks provided. These initial usability scores indicate that the design flow was sufficiently intuitive, demonstrating its appropriateness for evaluation in the next phase of the study.

General user perceptions of DiscourseDash

Participants consistently preferred DiscourseDash over the baseline social media platform Reddit for the given tasks, citing its efficiency in consolidating extensive data into actionable insights. G26 highlighted that “This would make it easy to justify your thoughts… then you can tell him [your superior] how many thousands of people are interested in this [topic] right off the bat.” Similarly, G15 noted, “I’d say that was much more informative and easier to navigate through.” Another frequently mentioned reason for preferring DiscourseDash was its visual clarity and ease of use. G19 commented, “DiscourseDash gave us a clear overview of where you should focus on.”

The ability of DiscourseDash to quickly access and interpret large volumes of data allowed participants to make more informed decisions, which is especially valuable in contexts where decisions need justification. G25 explained, “The graphical representation and all of these graphs were telling me exactly the information required to [achieve tasks] and finding out what’s going on [in the discourse].” Another participant, G15, added, “[Reddit] gave me comments that I can read through, but I would see the statistics and graphs [from DiscourseDash] to make sure I make an informed decision. By doing that passively, I not only get information from all the comment sections, but I also get to read through the comments if needed.” These suggest that DiscourseDash is perceived as an effective means of providing objective analysis for informed decision-making.

Theme 1: Effectiveness in guiding a decision-making process. Participants found that DiscourseDash facilitated a more structured approach to decision-making by providing a “good overview” (G16) that “helps to boil down the overall topics that are being discussed” (G6). For example, G1 compared their experiences and mentioned that “it gave a better overview… whereas [Reddit] is not conducive for me to find out the most important viewpoints and topics that I [as fact checker] should be addressing.” Participants perceived this structured approach as beneficial in professional settings when decisions require concrete reasoning. Another advantage mentioned is time efficiency. As G10 mentioned “It’s much faster and easier to identify the for and against positions that are already categorised for me.” Correspondingly, G13 found the stance and sentiment summary provided in DiscourseDash “very useful”, allowing participants to promptly grasp public opinion.

Contrary to Reddit, participants highlighted the challenges they faced with the same tasks. G3 explained, “[In Reddit] it takes ages to find what I want to do, and if I want to determine sentiments, I need to check these replies manually.” G19 shared a similar thought “ It’s like I need to read, for example, thousands of comments […] to get an idea [from Reddit] while the analysis here [dashboard] saves a lot of time […].” G23 added, “In Reddit, I have to like maybe read the comment twice to understand the sentiments and stances.” G6 pointed out that “if you want to do this work properly, you need a tool to help you […]. Maybe a ready dashboard […] there’s too much information [on Reddit] that is too hard to see the big picture.” Overall, DiscourseDash was perceived as more efficient and effective than manually navigating through large volumes of unstructured data.

Theme 2: Guidance towards articulation of ideas. Participants highlighted how DiscourseDash facilitates understanding of discourse by revealing ideas, correlations, and trends. For example, G15 articulated their view that “The discussion is high on the first topic, which states that there is some confusion or statements made without proper facts being spread […]. Here [stance and sentiment analysis] clearly states that they are contradictory opinions, and that’s not being resolved.” Similarly, popularity was useful in revealing correlations between different topics of discourse. For example, G16 articulated their understanding of the interconnected trends across various topics, noting that “whenever there’s a spike in topic one, there’s a spike in topic three as well. This [particular topic] is also interesting because it follows a correlation with these [two other topics].”

Another example illustrates the interpretation of some ideas around the discourse by G2, “[…] at the same time they are talking about the second dose, they are also talking about side effects, with [negative sentiments towards it] […] they don’t like vaccines and very few of them are thankful for all the work done [for vaccines] so far.” Another key observation, G7 illustrated how the data trends in DiscourseDash stimulated curiosity “I tend to [want to find out] what happened around that time [a decrease in popularity]. What kind of event might have occurred at that point?” These analyses demonstrated how DiscourseDash supports and guides a user’s thought process.

An example of how stance and sentiment analysis shape participants’ ideas, G18 concluded their thoughts that “What I can make out is that people are happier in taking a vaccine rather than not taking it […] because from both the graphs [stance and sentiment], […] the number of positive is more and here the favour is more than against.” While the harmfulness label was interpreted as “high rating of harmfulness is probably more important” (G10). Adding to that, G17 “feel harmfulness also determines newsworthiness […] So I’m leaning [my decision] towards the first two [topics labelled as high harmfulness].” Instead of the term harmfulness, G18 “would rather it be a saying impact, but I guess it denotes the negative sense of these topics.” These observations highlight how the harmfulness label was used and interpreted throughout the decision-making process.

Finally, the Detailed Analysis facilitated participants in formulating ideas about the factors influencing different stances. For example, G15 articulated their understanding that “there are some people who want to take the vaccine, they show negative sentiment […] because of limited supply and they feel insecure as they are the essential workers, […] people who have a neutral stance and sentiment, they are not exactly sure how the vaccination works and how it helps.” Another informed idea articulated by G25 is “a lot of people are asking how the government is going to [take action towards the vaccine], […] there’s a lot of confusion.” Consequently, most participants expressed similar ideas when concluding their investigation.

Theme 3: Supporting trust through detailed discourse-level analysis. Several participants expressed initial scepticism toward the overview page due to limited transparency. For instance, G6 remarked, “I have to blindly trust this classification […] I have no idea what data this has been generated on.” Similarly, G26 asked, “Where are the sources from?” However, after exploring the Detailed analysis page, G6 noted, “I do have access to the data […] this addresses what I criticised [previously]”. This shift illustrates how access to underlying discourse-level data can improve perceived explainability and trust.

Another key observation highlighting how Detailed analysis facilitated the investigative tasks is evident in G26’s experience. They initially described having difficulty distinguishing between stance and sentiment. However, they noted that examining examples in the discourses helped them better understand these concepts, highlighting the value of this feature in clarifying their meaning. They expressed that their understanding of stance and sentiment, and their correlation, was “not very clear until I toggled around.” After reviewing several “different combinations and permutations [of stance and sentiment pairs]” in the discourse, G26 understood the correlation better. This suggests that while the overview page provides a high-level summary, examining the detailed data in the discourses is necessary to understand the complexities behind these summaries.

Theme 4: Concerns about analysis measures. A common concern raised during the study was the disagreement participants experienced between their manual inspection of the data and the labels provided on DiscourseDash. For example, G16 pointed out that DiscourseDash “incorrectly marked some sentiments” indicating perceived inaccuracies in the sentiment classification. Similarly, G6 stated that “when I dig into the data, which I think is important that I have the option to, I very quickly identify several [labels] for me don’t match up [with own judgement], […] which reduces the reliability of what I saw in overview.” These observations suggest that improving transparency in the labelling process is necessary to enhance trust and confidence in the system’s outputs.

Usefulness of features in influencing decisions

Here, we grouped the features mentioned by participants during their interactions to show how they evaluated each feature’s usefulness when performing tasks on DiscourseDash.

Theme 1: Effectiveness of harmfulness labels. Harmfulness labels were indicated as helpful in guiding the prioritisation of topics that were both widely discussed and deemed significant due to their potential impact. For example, G16 noted that “the harmfulness is quite high. So I’m leaning towards like the first two.” Similarly, G17 articulated that “harmfulness also determines newsworthiness, so it might be important to [prioritise] more harmful news.” They recognised that higher harmfulness signifies more pressing issues that deserve greater attention in decision-making. However, some participants expressed confusion about the interpretation of harmfulness levels. For example, G22 pointed out that “I don’t understand what the high harmfulness means […]” though they still interpreted that “if the harmfulness is high, then it is important.”

Theme 2: Roles of stance and sentiment analysis. The analysis of varying stances and sentiments reveals how these factors can influence the perceived importance of a topic. For example, G19 mentioned that “although people talked about it [the topic] a lot [from the stance detection], not a lot of people have a clear opinion on that [topic]. So, that might [warrant] a discussion in the article [as a fact checker to address it].” This example shows that the presence of diverse opinions signals to users to explore the discourse more thoroughly. Additionally, G26 observed that stance detection helps to correlate public sentiment, stating “we can tell that not many people are just singing to the choir; the public seems to agree on this [topic] but may have more questions about it.” This indicates areas of both agreement and contention, suggesting topics that may benefit from further investigation. G22 mentioned, “I think for me the stances and sentiments help shape which [topic of discourse] is important […] because if you have [a contrasting opinion like against a topic] it enriches the discussion and it’s more exciting [to investigate].”

Theme 3: Use of popularity over time. Participants leveraged popularity over time to connect shifts in interest with specific events. For example, G19 observed, “Especially here [spike in the graph] you can see some different changes, which might correspond to specific events like news that happened at that moment.” The potential application of popularity for future content planning by fact-checkers was also noted. G26 noted “comparing these [topics], although there is a decreasing amount of interest in the general vaccine, […] maybe a follow-up article like what might inform my work [as a fact checker] next would be about the second dose, which is the next trending topic [observed from the trends].” This shows how tracking popularity trends prepares decisions for interventions as a fact-checker. Besides, G07 mentioned the longitudinal aspect of popularity data commenting, “[during] the investigation, focusing on specific points like maybe this part [decrease in engagement trend] would be interesting to see what kind of events happened.” Overall, “although [a topic] is less popular than [another] topic” the popularity graph helped participants determine that “it’s rising in importance” based on the emerging relevance.

Understanding existing workflows before we introduce DiscourseDash

This section demonstrates how experts engage with social media discourses—their roles, workflows, tools, and challenges to set the scene for the next results section.

Theme 1: Usage of social media discourses in decision-making contexts. This theme contextualises how experts engage with social media discourses in their day-to-day decision-making. Practices from their domains show how understanding discourses is interconnected within their organisational workflows and priorities. This means that they influence both high-level strategies and immediate responses (see Table 3).

Theme 2: Guiding factors in decision-making. Experts described workflows from data aggregation and analysis to planning and management, highlighting factors from social media discourses that guide decisions. We identified three consistent patterns: (1) data-driven indicators, including measurable metrics and observable patterns; (2) interpretive insights, involving subjective assessments and qualitative readings; and (3) situational/contextual dependence, where decisions are shaped by organisational goals, stakeholder expectations, or broader events (see Table 4).

Theme 3: Navigating social media discourses. Experts tasked with interpreting social media discourses face challenges not only in data collection but also in making sense of this rapid, extensive, and often ambiguous information. We summarise how they navigate this space, extracting transferable insights relevant to their decision-making workflows. Our findings identify two approaches: (1) manual exploration and (2) automated analysis. These approaches are used independently or in combination, depending on the needs (see Table 5).

Theme 4: Limitations of current approaches in workflows. This section summarises key limitations experts identified in their current methods for navigating and analysing social media discourse, informing how the design of DisocurseDash addresses these challenges and highlighting the need for improved tools across workflows (see Table 6).

Perceptions during experts’ interactions with DiscourseDash

Building on the analysis of experts’ workflows, this section explores their perceptions of DiscourseDash. We examine how DiscourseDash aligns with their practices, supports their analysis and decision-making, and identifies opportunities to enhance social media discourse interpretation.

Theme 1: Supports the balancing of time and effort to allocate priorities. The overview page was described as helping experts make split-second decisions about which issues required immediate attention (E6). Experts emphasised prioritising social media topics based on their potential harm, especially regarding brand damage or public response. The ability to identify and act on harmful content was important for timely and effective decision-making. E5 illustrated how a harmfulness indicator can support resource allocation across issues of varying severity:

“The more harmful ones might bring down your brand very fast […] so when making decisions, we prioritise those with high impact first, and the ones with low to medium effect come after. We can’t prioritise [topics] that are less likely to have a harmful effect on the brand.”

This illustrates how the harmfulness indicator actively supports operational decision-making for risk management, prioritisation, and balancing resources. Even when harmful topics lack immediate solutions, experts noted they can still be prioritised due to their potential to escalate quickly. This allows organisations to balance resources between urgent threats and less harmful, manageable topics. They also mentioned the importance of an overview of a topic’s popularity and temporal dynamics to help determine relevancy and response strategies. These features could support experts in distinguishing short-term attention spikes from longer discourses that may signal more serious or lasting public concerns. E4 described how understanding a topic’s virality can guide prioritisation by revealing its current and potential reach:

“It’s important to know what is in virality at the moment […] to know the range OK, is it just a temporary thing? Is it something that has been going on, getting more audience over time, or is it just […] a momentary buzz? After knowing that, whatever is trending becomes very important.”

This showed how tracking the momentum of virality helps experts decide if a topic needs immediate action or further monitoring. Similarly, E3 highlighted topic longevity as an indicator of importance and engagement:

“If the topic is well spoken about, definitely we’ll have to check the time at which people have been talking about it. Some topics just come, and in 10 min they’re out […] but topics that last for days, those are the ones we check out.”

Experts saw this feature as useful for filtering noise and focusing on persistent topics that may signal long-term risks. These features were perceived as valuable for helping experts strategically allocate attention and resources to high-priority issues.

Theme 2: Monitoring public discourses supports targeted interventions. Experts noted that DiscourseDash helps monitor social media discourses for timely decisions and adjusting interventions. E1 highlighted the importance of public stance over personal opinion: “It’s people’s opinion that matters, not mine.” Similarly, E4 described how sentiment analysis supports decision-making by revealing public preference: “[It] helps to know how people are leaning towards whatever is going on. What decisions are they making?” Building on this, E7 explained how employee discussions, such as debates on remote work, can shape internal strategies and workplace policies: “If there’s a heated argument about work hours in the corporate setting, we’ll align workplace strategies and policies accordingly, redesign workflows, and adjust benefits.”

DiscourseDash was also valued for tracking sentiment changes and evolving narratives for timely interventions. E2, a public health expert, explained how DiscourseDash helps anticipate responses to vaccination campaigns by analysing discourse across key demographic groups before and after announcements: “Before the second dose information was public, I would search the media for narratives and sentiment, then trace the same keywords to see how they change over time.” They further explained how this process would unfold through different stages of the campaign:

“From the time before the second dose information was released, I had already monitored public discussions. Once the information became public, I observed how anti-vaccine groups introduced their narratives and how various age groups or community segments reacted. I continued tracking this over time, during the rollout and afterwards, to understand how narratives and sentiments evolved within specific demographic groups.”

This demonstrates how the expert’s workflow aligns with our design concept of monitoring narratives and sentiment, and informs effective public health messaging strategies that address emerging concerns or misconceptions. They noted that this type of analysis supports both immediate responses and long-term planning:

“Based on [monitoring trends], it is possible to create interventions for current goals. But it’s also important for the next time, for future doses or other vaccines. How do we prepare communities, especially vulnerable ones, to accept the doses? Or maintain a positive response? If one community responds well, there’s an opportunity to study what makes them different. Is the sentiment in their media bubble more supportive?”

By tracing the success of initial interventions, experts can better prepare for future events and tailor responses to specific demographic groups. Beyond public health, some noted how harm assessment could support strategic communication. For instance, E6, a marketing expert, explained: “I’ll be able to determine how to mitigate the harmfulness of the content and make sure I’m marketing to the right people. Because if you’re marketing to the wrong people, it can increase the harmfulness of the post.”

Theme 3: Re-emphasis on the need for tools to provide quantitative aspects beyond statistical visuals. A key finding was the experts’ consistent need to go beyond summary metrics and access the actual discourses driving public responses. This occurred even before participants interacted with the Detailed analysis page in DiscourseDash, which supports deeper exploration of filtered discourses. This emphasis on qualitative analysis validates our design decision in DiscourseDash that underscores a broader gap in existing analytic tools. As E7 put it, while visual data distributions were expected, the real challenge lies in bridging qualitative insights with quantitative measures: “there’s a need for us to translate qualitative data into measurable terms.”

We observed a similar pattern when experts described the limitations of their current workflows. They often noted that high-level statistical indicators are only useful to a point. To truly understand public sentiment, they need to trace these metrics back to the underlying conversations. As E1 explained: “You have to understand if you’re actually doing better this week or you’re not doing so well. You have to know the cause. This would help you strategise your next move.” Similarly, E5 stressed that identifying general trends is not enough: “We need something specific […] we need to put ourselves in their shoes, so we need to get exactly what the problem is.” E8 further explained how qualitative analysis fits into their workflow: “After the employer has […] looked through the charts, looked through the engagement and all that, the person wants to read why people don’t like this particular service.”

These insights reveal the complexity of how experts interpret social media discourses, not only through metrics but by understanding narrative shifts and contextual factors. Current tools prioritise computational scale over human interpretation, which leaves a gap in addressing the need to encourage human collaboration in questioning and refining automated outputs rather than relying on them. DiscourseDash was designed to investigate whether it helps bridge this gap by supporting qualitative insights. In the next theme, we examine how experts perceived and aligned the detailed analysis of discourses with their existing workflows. We investigate whether features like topic selection, discourse filtering, and labelled stances and sentiments helped them contextualise proposed metrics and better understand public responses.

Theme 4: The conceptual idea to provide a detailed analysis of discourses aligns with expert practices. The ability to investigate not just what is happening in public discourse but also why remains a key limitation in many current analytical tools. Before introducing DiscourseDash’s discourse-layered features, experts consistently highlighted the need for deeper qualitative insights to help contextualise the patterns shown in visual analytics. This need directly informed our design and reinforces its relevance. After being introduced to the Detailed analysis page, experts noted that it aligned with their existing practices and addressed previous limitations. For example, E1 described how access to actual social media conversations, combined with structured sentiment and stance labels, supported their interpretation efforts:

“Having to know words, months, people talk more about the product and the reason why it’s trending and also why people talk negatively about it […] kind of gives you a heads up to know what’s coming […] and I can choose to know why people go against the product and how many percent of the people going against the product.”

Here, unfolding analysis on a discourse level becomes a means of rationalising decisions, allowing experts to stay ahead of emerging risks. E2 expanded on this by explaining how the combination of stance and sentiment helps them better understand public responses:

“[It] would really work well because […] it gives you a first-hand degree of information about these topics. You know, involving the intensity of these emotions and understanding our public opinion and sentiments.”

These granular details, reflecting the fast-moving and emotional nature of discourses, were perceived as highly relevant for staying aware of the depth of feeling in different situations. Additionally, E5 directly highlighted how this kind of discourse analysis is missing from their current workflow:

“I was talking about the discourse earlier because I haven’t even seen it before […] we need to get to the root of the problem. We need to read what people are saying, their experiences, their concerns.”

For these experts, it’s important to unfold public voices to make sense of trends, controversies, or misinformation. Similarly, E6 discussed how replies alongside stance and sentiment indicators enabled a more elaborate examination:

“If something is very harmful, I want to understand what people meant by that, I want to see the replies, I want to see their stance […] analysing those replies helps us understand if more [on why] people are ‘for’ or ‘against’ the product. That knowledge helps us get better at delivering to customers.”

These signals were valued for providing evidence of real-world impact and public investment on a topic. Beyond reactive monitoring, experts picture using it for strategic applications. E4 reflected on how these discourse-level analyses could inform long-term planning:

“We can [use this feature to] shape a general view of how people are thinking […] and we can redefine our products and services to better align with their readiness.”

Here, DiscourseDash was perceived as supporting experts in forecasting what might come next and preparing for it. They gave examples from trade policy and climate initiatives to show how DiscourseDash could help anticipate future developments and inform strategic planning. In healthcare, experts noted that understanding public beliefs, hesitations, and motivations is key to designing effective interventions. As E8 reflected on the importance of presenting these justifications through detailed analysis:

“These are people’s opinions. These are people’s thoughts. It’s important for those providing services to know what people think about what they want to present […] What are the fears, the sentiments, reasons why they do or don’t engage.”

Rather than relying only on abstract metrics, experts valued tools that help reveal public concerns and experiences. They also noted that this deeper understanding supports shaping responsive actions. As E11 explained that sentiment analysis within discourses can directly support timely decision-making:

“Having seen this, this is how we will strategise […] If the sentiment distribution is highly negative, we restart the system […] we can bring in some interventions to get people to accept [vaccines].”

In these demonstrations, qualitative insights functioned as inputs to human efforts in investigation and planning. Across domains, experts valued the distinct concept of DiscourseDash for supporting their reasoning and investigative judgment.

Theme 5: Misalignments with existing analytical practices. Experts consistently highlighted missing key features essential to their daily workflows, revealing gaps in our initial design that are crucial for aligning with established practices. Addressing these misalignments will guide iterative refinement to better fit real-world organisational needs and sociotechnical contexts.

A frequently raised gap was the ability to filter demographics of analysed discourses. Targeted demographic details—such as age, location, language, and behaviour are central for tailoring strategies and contextualising sentiment. For example, healthcare experts emphasised segmentation to design interventions for specific audiences. E10 explained: “It would be nice to filter by, say, 30–45-year-old mothers in [country] […] so I could create specific reports and track changes over time for preparedness and response.” Experts also noted the absence of hashtag analysis, which is important for monitoring trends since hashtags often serve as entry points to discourses.

Experts described current sentiment analysis as too simplistic for decision-making. Instead of just positive, negative, or neutral categories, they needed richer insights into emotional intensity. E2 used an analogy: “Imagine giving a child a biscuit. If the child doesn’t want the biscuit and is forced to take it, that’s not the same as the child eagerly accepting the biscuit.” This analogy illustrates the importance of understanding whether someone’s emotion or reaction is genuine. To measure this, the expert suggests using a graph to visualise the intensity of emotions. For example, a graph could show that 70 percent of people strongly support an idea, while 20 percent only somewhat support it, allowing for a more accurate understanding of how deeply people feel about the topic.

Similarly, E7 noted that they: “need to understand the degree of how much people are engaged, how strong the negativity is. That helps us dig into the intensity of how people feel about certain things.” For example, adding a “confused” category to capture people who are undecided or unsure, which is crucial in public health contexts like vaccine uptake (E8). Tracking sentiment over time was also emphasised as crucial for responsive interventions. E10 noted: “Without that information, you can’t build any future response or interventions because you don’t know the changes [of sentiments] over time.”

Experts also called for better contextual and cultural relevance. Large data volumes are less useful without alignment to specific decision contexts. Filters and indicators need to be context-aware, especially for regions with diverse dialects or sensitive cultural environments. Some research experts preferred manual, qualitative interpretation over automated tools, arguing that qualitative categories must emerge through human evaluation. E9 explained: “ Every research is different and depends on the cultural context that is very specific […] We have had to make the categories [analysis aspects] ourselves […] these kinds of tools cannot give you that flexibility.” This reflects a broader gap in the current analytical tools’ ability to support experts conducting targeted or high-stakes social media analysis.

Reflecting on the real-world applicability of DiscourseDash

We summarise experts’ overall reflections after using DiscourseDash, offering insights to guide its integration into real-world workflows.

Theme 1: Practical applications identified by experts. Experts noted how DiscourseDash could support their or their organisation’s decision-making by fitting into existing workflows for monitoring discourses, informing strategies, and responding to emerging issues.

Sub-theme 1: Monitoring real-time discussions. Experts emphasised DiscourseDash’s value in unfolding emerging discourses and reducing manual effort across platforms. This supports early detection of sentiment shifts and discourse attention for preventative actions before narratives spread. For brand or campaign monitoring, DiscourseDash was viewed as a diagnostic tool providing signals for timely, effective responses. As E6 described:

“[I] need to know why people are talking [negatively] about [my brand], so if we realise it’s the price, that way we can try to [fix it quickly]. If you don’t know exactly the problem, it’s gonna take a longer time before it comes to a solution. So the [dashboard] is gonna help us get to the root of the problem, help us find the starting point to solve these problems.”

These reflections showed experts viewed DiscourseDash as supporting strategic analysis, moving beyond statistical monitoring to understanding drivers behind public discourses. Assessing whether trends are relevant or spreading across platforms was fundamental to gauging response scale and urgency. In journalism and media planning, experts envisioned using DiscourseDash to align content strategies with real-time public attention. E4 described:

“So I put up the date [to compare] pre-Grammy to after Grammy […] it gives me the summary and brings out the keywords […] the sentiment if it’s +1 the charts will show this particular reading […] I get to know the number of people engaging with that particular topic at a particular set time, it just makes life easy for me.”

Rather than relying on assumptions about audience interest, experts perceived DiscourseDash as guidance for navigating discourses with clearer context and understanding. Across domains, they noted how it identifies when discourses start, gain momentum, and why public attention shifts—insights valuable for staying informed and making rapid decisions in fast-paced settings.

Sub-theme 2: Informed decision-making for planning and evaluation. Experts highlighted that DiscourseDash supports ongoing planning and evaluation throughout an intervention, not just initial decisions. It enables a flexible process to refine actions as public narratives shift, especially important when sentiment impacts outcomes. For example, E10 described using DiscourseDash across different health intervention phases:

“It’s helpful for monitoring and evaluation of [health] interventions, which allows scaling up or scaling down depending on the needs as you go. After the intervention, it helps to reflect and create preparedness plans for future interventions in the same context, for vaccines or flooding.”

Discourse insights were valued not only for real-time response but also for learning and preparing for future scenarios. Rather than relying on traditional reports or surveys, DiscourseDash offers immediate indicators of public reactions. Similarly, E6 emphasised how analysing public discourse can help refine product development:

“When a new product was launched […] we wanted to know, read their feedback […] [with this] we’re able to determine how to improve the product.”

These reflections suggest experts view DiscourseDash as a method to monitor real-time impacts, guide adjustments, and inform future strategies. Rather than waiting for formal feedback like surveys, sales data, or delayed reports, DiscourseDash provides timely insights into public signals. This supports evidence-based decisions grounded in ongoing discourse, enhancing workflows for both immediate action and post-implementation reflection.

Sub-theme 3: Managing crises and reputation risks. Experts highlighted DiscourseDash’s value during crises such as public opinion shifts, misinformation, or reputational damage. Quickly identifying the core issue was crucial for timely responses. DiscourseDash provides a structured and efficient alternative to current workflows. As E4 explained: “in crisis management […] when you know what the issue is specifically, it should be easy to turn things around.” They further explained how the visualisation and sentiment indicators enable faster problem diagnosis to support response coordination:

“[Knowing that there is damaging] information about my brand that’s going on, I’ll use it [DiscourseDash] to understand the sentiment of the audience […] it would help in making decisions [according to] response protocol and there’s like a graph to back it up showing the range, so it’s very explicit and it makes things easy to comprehend.”

DiscourseDash helped navigate overwhelming discussions by providing a structured overview of concerns for quicker identification of root causes and targeted actions. For example, E5 explained how it could pinpoint a specific problem during a product backlash:

“[I] need to know why people are talking [negatively] about [my brand], so [for example] if we realise it’s the price, that way we can try to [fix it quickly]. If you don’t know exactly the problem, it’s going to take longer to find a solution. So the [dashboard] is gonna help us get to the root of the problem, help us find the starting point to solve these problems.”

These reflections indicate the potential for a timely understanding of the ‘why’ behind negative public reactions or emerging narratives, which experts valued more than merely detecting issues like sentiment shifts. In crises, fast decision-making impacts outcomes, so DiscourseDash’s support in navigating social media to identify root causes is crucial for effective response and strategy alignment.

Sub-theme 4: Shaping decisions dependent on understanding public views. Experts perceived DiscourseDash guiding reflective decisions around brand positioning, internal policy, and strategic communication. Rather than immediate responses, these uses focus on understanding how public perceptions evolve. By analysing social media discourses in detail, experts track shifts in tone, values, and alignment with intended identity. Thus, DiscourseDash serves as more than a monitoring tool; it’s a resource for staying informed through broader narratives. For example, E7 explained how discourse insights inform workplace policy design:

“In terms of designing the perks and benefits for all our employees, that is something we can incorporate to understand the trends […] there are a lot of culture changes […] So we can redesign [things] with a set of trendy [ideas] that could improve health, productivity, and the happiness index.”

This example shows how DiscourseDash helps align decisions with a broader understanding of employee sentiments and social media discourses, supporting a healthier workplace identity. Instead of relying only on assumptions or surveys, experts use DiscourseDash to gain accurate insights by analysing employees’ discussions about their work environment to inform strategic reflections.

Sub-theme 5: Following public voices to shape societal values. Experts valued DiscourseDash’s potential to track evolving narratives in policy debates, public health, and social movements. Beyond reactive monitoring, it uncovers deeper public discourse values, concerns, and perspectives for strategic alignment with emerging policies. For example, E11 described using DiscourseDash to inform government decision-making:

“If the government asked me to survey what is currently going on, on ways to improve people’s perception about them, I would use everything you shared with me [DiscourseDash] to include people’s opinions, what people are trying to pass on to the government. Like, if John Doe says, ‘This is a way the government can help the people […]’, it’s my job to write that down and explain [to the government] what led to that suggestion.”

This highlights DiscourseDash as a tool for mediated listening, translating public voices from social media into actionable suggestions for institutions. It illustrates DiscourseDash’s potential to help governments identify emerging concerns, trace the roots of suggestions, and communicate these strategically, bridging public expression and formal decision-making.

Theme 2: Adoption barriers in real-world settings. Experts recognised DiscourseDash’s value but noted barriers to adoption, including organisational constraints and concerns about data, ethics, and contextual fit.

• Resource constraints: Many experts, especially from smaller organisations, noted limited budgets and technical capacity as barriers to adopting new tools. It needs to be easily integrated with existing systems, with minimal training, and no major infrastructure changes. Drawing on experience with similar tools, they emphasised long-term maintainability and reliability to avoid disruptions or rising costs from lack of upkeep or organisational changes. This highlights the need for sustainable, dependable tools that support ongoing workflows.

• Accountability: Concerns were raised around data privacy, ethical use, and transparency in algorithmic processing, reflecting wider sensitivities about monitoring public discourse at scale. They stressed that design must prioritise ethical safeguards, especially for sensitive topics, including protecting identities, ensuring privacy, and promoting fairness. These concerns align with sociotechnical accountability for ethical governance throughout DiscourseDash’s lifecycle.

• Organisational structure: Workflows with hierarchical or collaborative settings require role-specific permissions and access controls, similar to how platforms like Meta manage page roles according to the organisation’s position of individuals. DiscourseDash must support team workflows with varying responsibilities and sensitivities, raising design challenges around shared workspace analysis that respects organisational protocols.

• Real-time performance and integration: In time-sensitive situations, experts emphasised low latency, real-time data processing, and seamless compatibility with existing workflows and platforms. Performance and interoperability are core to adoption, especially where discourse insights inform decisions. This requires back-end optimisation and flexible system design adaptable to real-world workflows.

• Language and cultural adaptability: Experts noted limitations in supporting diverse languages, regional dialects, and culturally specific slang. Inclusivity in multilingual, culturally aware analysis pipelines is important for future tool development to reflect public discourses.

• Trust and reliability: Some experts, especially those with qualitative backgrounds, expressed scepticism about automated classifications oversimplifying complex discourses. While useful for initial exploration, there is still space for improvement within DiscourseDash regarding reliability in managing politically or culturally sensitive contexts. Hence, they emphasised the need for better interpretation in design for users to understand how outputs are generated, how data sources are used, how assumptions are made, and confidence levels of indicators, which are all key factors for real-world adoption.

Theme 3: Designing for evolving analytical workflows. Experts reflected on how their workflows might adapt to the rapidly changing social media landscape. These forward-looking insights provide valuable guidance for future dashboard development. Hence, incorporating these perspectives helps ensure the design remains relevant and adaptable, supporting decision-making workflows that evolve along with the dynamic nature of social media.

• Integration for more customised AI-based features: Experts envisioned future workflows enhanced by collaborative AI to support proactive, informed decisions. Beyond text retrieval, they envision a potential for AI-assisted processes to outline strategies based on past events and guide responses. Suggestions included incorporating instruction-based prompts and adjustable recommendations within DiscourseDash. Some mentioned incorporating features, such as X’s (formerly Twitter’s) Community Notes to identify misleading content and track virality with clear explanations.

• Adapting temporal analysis beyond real-time: Experts emphasised extending temporal analysis to support strategic understanding over time, not just immediate reactions. Tools should capture how discourses evolve before, during, and after key events. They suggested using discourse patterns to predict emerging trends or risks and anticipate misinformation before escalation. They noted: “You want to know where people stand at the beginning, how that shifts after an announcement or crisis.” This highlights the value of integrating real-time data with historical patterns to reveal shifts for more adaptive or forward-looking decisions.

Improving interpretation of analysis outputs

There is a clear need for more detailed and accurate interpretations of the analysis output. General users struggled to understand the meaning behind specific labels such as harmfulness, stance, or sentiment. This highlights the importance of providing concise and accessible explanations, such as through tooltips (Dai et al., 2015) or in-context guides, to help users quickly grasp the meaning of each metric. This aligns with broader insights from explainable AI (XAI) research, which emphasises that the interpretation of analysis outputs is context-dependent and should reflect users’ cognitive models (Miller, 2019; Doshi-Velez & Kim, 2017). Experts, on the other hand, expressed interest in more refined categorisations that could better reflect the complexity of online discourse. For example, rather than using broad categories, future versions of DiscourseDash could explore hierarchical or layered classification systems that first identify the presence of an issue and then allow for deeper investigation into its nature (e.g., types of harm, intensity of emotions). Hence, these insights suggest two main directions for future work: (1) to improve interpretation for non-expert users and (2) to offer more granular analytical options for advanced users. Future work should address these gaps to ensure that these analysis not only integrates seamlessly into interaction experiences, but also increase in value in various professional contexts.

Handling variability in data sources

Future iterations should offer flexibility in data sources, supporting real-time and offline inputs to better align with diverse and dynamic workflows. This aligns with prior work in crisis management studies, which highlights the importance of accommodating diverse data contexts and situational demands (Saroj & Pal, 2020; Jin, Liu & Austin, 2014). The current prototype presented an analysis of vaccine-related topics from the COVID-19 pandemic, which, while suitable for initial testing, does not fully represent the evolving nature of social media discourses. Some participants found it difficult to relate the prototype to current or personally relevant topics. Additionally, expert feedback emphasised the importance of flexible data input methods. Some experts work with curated or sensitive datasets, such as anonymised surveys, interview transcripts, or internal organisational data, which are not presented on public social media platforms. In these contexts, where public discourse is culturally or politically restricted, reliance on online data might not reflect accurate insights for high-stakes decisions. They emphasised that allowing user-uploaded data would better align with practical adoption. This flexibility is critical in workflows where decisions rely on understanding complex perspectives that are not always visible in public discourse. For example, E08, an expert, shared that public expression is sometimes limited due to cultural norms, but accurate community insights are necessary for informed policy-making decisions. Hence, supporting real-time integration and secure offline data upload ensures that DiscourseDash can be adapted to diverse use cases (Adepoju et al., 2022), improving DiscourseDash’s utility and ethical responsiveness across different sociocultural contexts. Although the current prototype was evaluated using COVID-19 discourse from Reddit, the underlying framework is designed to be scalable and adaptable to other social media platforms such as Twitter/X and Facebook towards developing solutions for domains such as brand monitoring and crisis communication. Future iterations could adapt these evaluated analyses to real-time data on social media for real-world deployment.

Incorporating demographics considerations

Our study revealed a strong emphasis on the importance of demographic context, consistent with prior work on demonstrating its influence in social media analysis (Sadah et al., 2016). These findings reinforce the value of demographic data in improving the contextualisation of social media discourses. This inclusion of demographics will allow for a deeper understanding of how different groups perceive and engage with specific topics for a more accurate analysis during decisions on interventions. For example, categorising responses by demographic factors such as community affiliation, cultural background, or belief systems could offer valuable insight into varied behavioural patterns, allowing DiscourseDash to address diverse perspectives. This is evident from experts suggesting that the lack of demographic insights limits DiscourseDash’s practical relevance (see Page 26). For many experts, demographic segmentation forms a core component of their decision-making workflows when accurate and culturally sensitive strategies are required.

In addition, we suggest that future designs could benefit from integrating social network analysis to move beyond static demographic categorisation. This design direction responds to expert feedback and points to the value of supporting relational analysis (Peng et al., 2018). By visualising patterns of interaction, influence, and community formation, this can capture how narratives spread across different groups for a more interconnected understanding of online discourses. However, representing user relationships raises important privacy considerations of sensitive or polarised topics. Therefore, we recommend that future work explore how to balance the analytical benefits of social network visualisation with ethical and responsible design practices.

Predictive and actionable features

Although our approach takes a step in this direction by offering generalised support, our findings suggest that more flexibility is needed to accommodate a greater need for preferences. There is a clear indication from both user groups that DiscourseDash features extend beyond descriptive analytics to incorporate predictive capabilities and actionable guidance (Phillips et al., 2017). With the advantages of LLMs, several suggestions were made within this scope. Firstly, participants consistently emphasised the value of temporal prediction, specifically the ability to anticipate emerging trends based on ongoing discourses or past events, such as the COVID-19 pandemic. To enhance DiscourseDash’s strategic value, incorporating predictive functionality would further support a shift from reactive to proactive decision-making.

In both studies, we observed the importance of allowing users to engage with discourse at varying levels of granularity through optional summarisation features. Among general users, some valued access to detailed analyses of discourses, while others would prefer concise summaries for quicker insights. Building on this, experts can leverage such flexibility to adjust their engagement according to the specific demands of their investigations. This can better support decision-making across a spectrum of time sensitivities, urgencies, and depths of analysis. This highlights the need for a summarisation feature that accommodates diverse user needs by offering a choice of detail levels, balancing rapid insights with opportunities for deeper exploration. This finding aligns with HCI literature on designing for user preferences and flexible data presentation to meet the needs of diverse user groups (Goodman, Langdon & Clarkson, 2007; Perer & Shneiderman, 2008).

Expanding the features of DiscourseDash could further support users in identifying effective courses of action. Based on the analytical outputs, these features can generate suggestions informed by analytical output, such as referencing a repository of predefined knowledge or guidelines that reflect established practices, or recommending interventions based on historical events. Experts highlighted that analytical insights are most valuable when paired with practical recommendations to support decision-making. With advances in LLMs and Retrieval-Augmented Generation (RAG), future iterations of DiscourseDash could evolve with features that allow users to perform queries with LLMs with more grounded recommendations. This can be achieved by linking social media discourses to hierarchical knowledge graphs or allowing a RAG system to retrieve and refine relevant output information from real-time discourses.

Our findings suggest that while generic solutions can provide baseline support for expert decision-making workflows, there is significant scope for future work to address more complex and varied analytical needs. This highlights a key design implication where analytical tools should be adaptive and configurable to support diverse goals and contexts of different users (Feigh, Dorneich & Hayes, 2012). Our study concluded that allowing users to customise prompts based on their specific objectives can increase both the applicability and effectiveness of the system. This is because we found that while some users prefer automation to streamline tasks, others remain sceptical of AI’s reliability and prefer to retain more control. Therefore, future work should develop AI-enhanced systems that are not only flexible but also subject to ongoing evaluation and refinement to remain relevant, accurate, and usable across varied domains.

Inclusivity and representation

Future design of analytical systems must consider accessibility for a wide range of users, regardless of their linguistic, cultural, or resource constraints (Abascal & Nicolle, 2005). An important point raised by our experts was the accessibility of these tools. There is a strong preference for open-sourced solutions, which were seen as a more practical way to promote broader use, encourage collaboration, and reduce cost-related barriers. These tools should be affordable, considering user groups that are also making an impact in smaller teams or researchers with limited budgets.

In addition, our findings extend the ongoing discussions in NLP about language diversity and multilingualism (Joshi et al., 2020; Hershcovich et al., 2022). Participants stressed the importance of designing tools that extend beyond the English language. For example, the global pandemic was a shared crisis that affected people worldwide. However, if analytical tools do not support multilingual analysis, they risk excluding researchers and communities who work in languages other than English. To enhance fairness and encourage wider adoption, future tools should address linguistic diversity by accommodating local dialects and underrepresented languages in regions with frequent dialectal variation.

Furthermore, the discourse on social media reflects only a partial view of societal realities, shaped by cultural and contextual factors. An expert (E9) highlighted that the usefulness of DiscourseDash can vary depending on cultural settings. For example, without such considerations, these tools can risk misinterpretation or overlook important contextual meanings, which can lead to reduced trust or limited applicability in certain settings. To ensure broader relevance and effective use across diverse user groups, future design efforts should be grounded in local practices and informed by cultural contexts.

Building trust in analytical tool

Establishing trust is a fundamental requirement for the adoption of AI-driven analytical tools. Both user groups in our study emphasised the importance of system transparency around how such a tool generates output, processes data, and ensures the fairness of algorithms. This reflects existing HCI research that calls for explainability and accountability in AI system design to support user trust (Binns et al., 2018; Abdul et al., 2018). Future designs of such analytical tools should be adapted from our approach and strengthened by making each underlying process understandable when involving real-world decision-making.

In Phase 1, two participants (G6 and G16) highlighted noticeable mismatches between their interpretations and the sentiment or stance labels generated by DiscourseDash. Similarly, in Phase 2, E9 expressed strong scepticism toward current AI-based solutions for qualitative analysis in sensitive domains. These observation highlights the importance of transparency in the system. On one hand, it demonstrates that DiscourseDash’s design makes discourse data and model outputs visible for users to detect inconsistencies rather than rely on automated labels. This aligns with DiscourseDash’s goal of supporting human–machine collaboration in decision-making rather than replacing expert judgement. On the other hand, these disagreements highlight the need for more targeted studies to understand how annotation mismatches influence users’ trust in AI-assisted analysis. This reflects broader concerns in HCI about the early application of machine learning in socially complex environments, as our approach is not yet sufficiently validated or trustworthy (Suresh & Guttag, 2019; Ehsan & Riedl, 2020). More systems should incorporate methodological transparency and allow users to provide feedback on the system output when needed (Amershi et al., 2019).

To address these concerns, future design efforts should integrate mechanisms for explaining system logic (e.g., how classifications are made), communicate how data are sourced and protected, and allow users to engage with AI outputs at varying levels of interpretive depth. Hence, trust-based designs must be continuously strengthened and evolve together with the system’s use and user feedback (Miller, 2019). Future work should consider how to address limitations in AI-generated insights, for example, by presenting the confidence level of the output (Rechkemmer & Yin, 2022), or explainable methods (Danilevsky et al., 2020; Liu, Yin & Wang, 2018). This way, trust can be encouraged where users can regulate their reliance accordingly in domains where judgment of qualitative insights is subjective.

In addition, there should be transparent documentation that outlines how analytical tools are intended to evolve, be maintained, and be supported over time. When users place trust and reliance on such systems, there is a reasonable expectation that these tools will remain accessible and functional. Lack of clarity or sudden withdrawal can undermine trust and create significant disruptions in research and decision-making workflows. For example, E10 noted that the discontinuation of Meta’s CrowdTangle and the removal of research access by X (formerly Twitter) caused considerable setbacks for many analysts and researchers who had come to depend on these platforms. These cases highlight the need for sustained transparency and clear communication about long-term support and platform commitments when tools are used in critical analytical workflows.