Enhancing transparency understanding using machine learning and visual analytics

Model architectures and integration

• SpaCy-based NER Model: This model represents our primary, custom-trained solution. We fine-tuned a spaCy model built upon the RoBERTa architecture (Briskilal & Subalalitha, 2022; Satheesh et al., 2020) on a domain-specific dataset. This process involved extensive data collection from software Terms and Conditions (e.g., from Facebook, Google, OpenAI), manual annotation of entities, and configuration of a custom training pipeline using spaCy’s DocBin for efficient data handling. The fine-tuning process, detailed in ‘Participant Demographics’, adapted the model to our specific entity types, enhancing its accuracy for legal and policy-related texts.

• Transformers-based NER Model: This model leverages the Hugging Face Transformers library (Lin et al., 2022) to utilize the pre-trained “wikineural-multilingual-ner” model. This model is a fine-tuned multilingual BERT (mBERT) model capable of recognizing named entities in nine languages, providing broad coverage without requiring task-specific training.

• BERT-based NER Model: This model utilizes a BERT architecture (Garrido-Merchan, Gozalo-Brizuela & Gonzalez-Carvajal, 2023) pre-trained on large-scale text corpora. Its bidirectional context representations are highly effective for capturing complex linguistic patterns, making it a strong baseline for NER tasks without further fine-tuning on our dataset.

• XLNet-based NER model: This model is based on the XLNet architecture (Habbat, Anoun & Hassouni, 2022), which uses a permutation-based language modeling objective. This approach enables the model to consider all permutations of the input sequence, often leading to improved performance on tasks requiring modeling of complex, bidirectional dependencies.

Custom model development: the SpaCy pipeline

The development of our custom spaCy model involved a meticulous, multi-stage process to ensure high-quality entity recognition tailored to the domain of software transparency.

Data curation and annotation: We compiled a corpus from the Terms and Conditions of major software entities including Facebook, WhatsApp, Google, Microsoft, and OpenAI. The raw text was manually annotated using a specialized NER tool, labeling entities according to predefined categories such as PERSON and ORG. For example, organization entities included mentions like Google LLC, Meta Platforms, and Microsoft Corporation, while person entities included examples such as John Smith and Jane Doe. These annotations provided a structured representation of the text for downstream tasks.

Preprocessing and fine-tuning: The annotated corpus underwent rigorous preprocessing, including text normalization, deduplication, and standardization of entity naming conventions (e.g., standardizing “Google LLC” and “Google, Inc.” to “Google”). This refined dataset was converted into spaCy’s binary DocBin format. The model was then fine-tuned using a custom configuration file, optimizing hyperparameters like batch size, learning rate, and dropout over multiple iterative training cycles on a GPU. Error analysis was conducted between cycles to correct misclassifications and incrementally improve the model’s accuracy.

Evaluation: The final dataset was split into training (80%) and testing (20%) subsets. The model’s performance was evaluated on the held-out test set to ensure its ability to generalize to unseen data, solidifying it as a reliable tool for stakeholder extraction in transparency modeling.

Visual encoding

As described before, we enhance the TranspVis tool (https://youtu.be/sIxm5pzzbeA) with other features and methods, as shown in Fig. 2⁴ . The components use the same colors, such as green, red, yellow, and blue. A circle surrounds these components, and users can update or rotate the circle using the interaction tools at the top of the main view. By clicking on one of the components, users can explore its details on the right side of the main view. We have added buttons such as “Add Automatically” for each component. Additionally, we have included a description on the right side indicating whether the element was generated by a model or created manually.

For stakeholder addition (see Fig. 3A), the interface displays a dedicated window for automatic suggestions highlighted in red, showing new components proposed by the selected model (see Fig. 3B). Meanwhile, existing components with modified weights appear in green, allowing users to selectively add them if desired (see Fig. 3C). Components that remain unchanged with identical weights are displayed with low black opacity, indicating that no modifications are required.

For adding relationships, a window appears when the user selects two components to create a relationship between them. The user has the option to create it manually or automatically. If the user chooses the automatic option, a proposed relationship type appears in red at the top of the window (see the top of Fig. 4A). Additionally, a text description with the selected components is highlighted with a yellow background to give the user an idea of the chosen relationship type (see the middle of Fig. 4A). The user can review the information in this window and select the appropriate relationship type (see the down of Fig. 4A). Experts in transparency have also validated this window’s visual encoding.

Furthermore, interactions are added to the description section—if the user clicks on a weight, they can change it manually or use the proposed method (see Fig. 4B).

Evaluation metrics

The performance of our named entity recognition models is evaluated using the following metrics:

• Precision: The ratio of true positives (TP) to all predicted positives (TP + FP):

  (4) $$P=\frac{TP}{TP+FP}.$$

• Recall: The ratio of true positives (TP) to all actual positives (TP + FN):

  (5) $$R=\frac{TP}{TP+FN}.$$

• F1-score: The harmonic mean of precision and recall:

  (6) $$F1=2\times \frac{P\times R}{P+R}.$$

• Accuracy: The ratio of all correct predictions (TP + TN) to total predictions:

  (7) $$A=\frac{TP+TN}{TP+TN+FP+FN}$$

where:

• TP = True Positives (correctly identified entities)

• FP = False Positives (incorrectly identified entities)

• TN = True Negatives (correctly rejected non-entities)

• FN = False Negatives (missed actual entities).

NER model’s evaluation

Model comparison and performance analysis:

To evaluate the impact of training data composition on model performance, we conducted two distinct training experiments. The first model was trained exclusively on Facebook’s Terms and Conditions, while the second model was trained on a diverse dataset comprising terms from multiple software platforms, including Anaconda, Flipboard, Feedly, Google, Facebook, Freemind, Bard, and Git.

The performance comparison revealed distinct learning patterns between the two approaches. In the first test, focusing solely on Facebook’s data, the model demonstrated strong initial performance with a starting F1-score of 57.53%, which rapidly improved over training epochs to reach 87.50%. Precision and recall showed consistent parallel improvement, achieving 89.74% and 85.37% respectively, as illustrated in Fig. 5. This rapid convergence suggests efficient learning of platform-specific patterns.

In contrast, the second test utilizing multi-platform data began with a substantially lower initial F1-score of 39.46%, indicating the greater complexity of learning from diverse sources. However, the model showed steady improvement throughout training, eventually reaching a competitive F1-score of 76.28%. Precision and recall followed similar trajectories, achieving 77.36% and 75.23% respectively, as shown in Fig. 6. While the final performance metrics were lower than the Facebook-specific model, the multi-platform model demonstrated remarkable adaptability across different textual environments.

The performance comparison reveals a fundamental specialization-generalization trade-off in NER systems. The single-platform model achieved higher accuracy through focused pattern recognition, while the multi-platform model gained broader applicability at the cost of lower precision. This suggests a hybrid approach—combining diverse pre-training with domain-specific fine-tuning—would optimally balance these competing objectives for software policy analysis.

NER evaluation:

Table 5 presents the results of the evaluation of NER models:

• SpaCy: As shown in Table 5, SpaCy achieved a precision of 77.36%, a recall of 75.23%, and an F1-score of 76.28%. While SpaCy is known for its efficiency and speed, these results indicate that it may not be the best choice for tasks requiring the highest levels of accuracy and comprehensiveness. It is important to note that SpaCy was trained on our data and it extracts more than standard stakeholders. SpaCy extracts other stakeholders like the terms: users, businesses, suppliers, and organizations, while others cannot.

• Transformers: Transformers significantly outperformed the other models with a precision of 95.40%, a recall of 95.80%, and an F1-score of 95.60%. These exceptional scores highlight the model’s superior ability to accurately and comprehensively process language, making it the top performer in our analysis.

• BERT: According to BERT, it demonstrated strong performance with a precision of 86.98%, a recall of 88.33%, and an F1-score of 87.65%. BERT is highly accurate and effective, showing a solid balance between precision and recall, though it falls slightly short of the Transformers model.

• XLNet: Finally, we have XLNet. XLNet achieved a precision of 87.51%, a recall of 89.54%, and an F1-score of 88.51%. These scores indicate that XLNet is highly effective and slightly outperforms BERT, especially in recall and F1-score.

Relation model’s evaluation

Table 6 presents the results of the evaluation of relation models:

• Binary Model: Table 6 presents the performance comparison of Binary, LSTM, and Bi-LSTM models for relation extraction. The Binary model demonstrated superior performance with the highest precision and accuracy among the evaluated approaches. It achieved a precision of 94%, recall of 66%, F1-score of 77%, and accuracy of 67%. These results underscore the Binary model’s exceptional precision and well-balanced performance across metrics. The 94% precision indicates that when the model predicts a positive relation, it is correct 94% of the time. The recall and F1-score reflect a reasonable trade-off between completeness and reliability, while the accuracy confirms its overall prediction correctness.

  Figure 7 presents the confusion matrix of the binary relation model, which summarizes the model’s classification performance between positive and negative classes. The matrix reveals the distribution of true and false predictions across both classes, providing insight into model behavior beyond global accuracy metrics. The model performs well in identifying the positive class, achieving an F1-score of 0.77, while its performance for the negative class remains limited, with an F1-score of 0.41. The high precision (0.94) for the positive class indicates that when the model predicts a positive case, it is usually correct. However, the recall (0.66) shows that a significant proportion of actual positive instances are not detected. This highlights a clear trade-off between precision and recall for the positive class.

• Long Short-Term Memory (LSTM): Next, we have the LSTM model, which is a type of recurrent neural network (RNN). The LSTM achieved a precision of 67%, a recall of 40%, an F1-score of 50%, and an accuracy of 54%. While LSTM is known for handling sequential data effectively, these results suggest that it may struggle with binary classification tasks compared to the other models. The lower recall indicates it misses a significant number of positive instances, and the F1-score and accuracy further reflect its moderate performance.

  The multi-class relation model achieved an overall accuracy of 54%, with performance varying notably across the four relationship types. Figure 8A presents the detailed confusion matrix and classification report, illustrating distinct patterns in the model’s behavior. The model showed particularly high precision (1.00) for the “undecided” class but suffered from very low recall (0.25), suggesting it was overly conservative in predicting this category. The “obligatory” class demonstrated moderate precision (0.67) but low recall (0.40), resulting in an F1-score of 0.50. The “optional” and “production” classes exhibited more balanced results, with F1-scores of 0.57 and 0.54, respectively, though both metrics indicate room for improvement.

  This performance pattern is likely influenced by class imbalance, as the “undecided” class has only four samples, compared with 25 for “optional” and 22 for “production”. Moreover, the model’s architecture may not fully capture the subtle distinctions between relationship types.

• Bidirectional LSTM (Bi-LSTM): Finally, we have the Bi-LSTM model, which processes data in both forward and backward directions, providing a more comprehensive understanding. The Bi-LSTM achieved a precision of 83%, a recall of 50%, an F1-score of 62%, and an accuracy of 54%. These results show that Bi-LSTM outperforms the standard LSTM, particularly in precision and F1-score, demonstrating its enhanced ability to capture and process information from both directions. However, its recall and accuracy indicate that there is still room for improvement.

  The multi-class relation model achieved a moderate overall accuracy of 0.54. As shown in Fig. 8B, performance varies substantially across classes. The obligatory class attained a precision of 0.83, recall of 0.50, and $F_1$-score of 0.62. The optional class achieved precision 0.61, recall 0.44, and $F_1$-score 0.51. The production class demonstrated lower precision 0.44 but higher recall 0.73, yielding an $F_1$-score of 0.55. The undecided class obtained perfect precision 1.00 but poor recall 0.25, resulting in an $F_1$-score of 0.40. These results indicate a bias toward majority classes and difficulty recovering minority examples, likely due to class imbalance and limited data diversity. To address these issues, we recommend applying targeted data augmentation and oversampling techniques such as SMOTE (Synthetic Minority Over-sampling Technique) to increase representation of underrepresented classes, while validating performance on a held-out test set to ensure improved recall without introducing noise.

Model configuration and experimental procedure

All experiments were conducted in Google Colab using a Tesla T4 GPU and 16 GB of RAM. The NER task was evaluated using four models: SpaCy, a generic Transformer model, BERT, and XLNet. Fine-tuning for BERT and XLNet was performed with Adam optimization (learning rate $5\times {10}^{-5}$), batch size 128, gradient accumulation of 3, and early stopping based on validation loss.

For relation extraction, two models were evaluated: a unidirectional LSTM and a bidirectional LSTM. The relation dataset consisted of 301 labeled pairs of stakeholders and information elements, with an 80–20 train–test split. Each model was trained to predict whether a relationship existed and, in the case of the LSTM variants, to classify the relationship into one of five types: Obligatory, Optional, Production, or Undecided.

Error analysis

Named Entity Recognition. The NER models achieved strong quantitative performance; however, qualitative analysis revealed several recurring error types:

• False Positives: The models occasionally overgeneralized by labeling generic nouns as organizations. Examples include “application” or “website” being incorrectly tagged as ORG (organization). This typically occurred when the model encountered policy-related terminology resembling company or service names.

• False Negatives: Some models failed to recognize multi-token entities such as “Meta Platforms, Inc.”, often due to punctuation issues or limited exposure to such entities during fine-tuning.

These findings suggest that while pretrained models such as BERT and XLNet are effective for recognizing common stakeholder types, they remain sensitive to domain-specific phrasing and inconsistencies in capitalization.

Relation Extraction. For relation extraction models, several characteristic errors were observed:

• False Negatives: The Binary and LSTM-based models frequently missed legitimate but rare relationships, such as (developer, API usage information) or (customer, cookies), due to their underrepresentation in the dataset.

• False Positives: The models sometimes predicted relations for unrelated pairs such as (partner, anonymized statistics) or (vendor, service logs), primarily because high-frequency co-occurrences biased the classifiers.

The Binary TF–IDF model showed the best balance between precision and interpretability, while Bi-LSTM demonstrated stronger contextual understanding but remained affected by data scarcity.

Scalability discussion

Transformer-based Named Entity Recognition (NER) models such as BERT and XLNet leverage extensive pretraining on large corpora, enabling them to generalize effectively to broader and more diverse text collections with minimal additional fine-tuning. In this study, the NER models demonstrated robust extraction of stakeholder entities with high precision, showing strong adaptability and scalability despite limited annotated data.

Experts note that selecting entity classes is difficult and context-dependent, so it is best to identify the correct class initially rather than adjust classifications afterward.

In contrast, the relation extraction models—particularly the LSTM-based variants—require retraining or augmentation through semi-supervised or weakly supervised labeling to handle larger and more diverse corpora. While the relation extraction dataset contained only 300 annotated examples, the underlying model architectures were designed to scale effectively. Expanding this dataset would help mitigate class imbalance and improve coverage of rare stakeholder–information relationships.

From a computational standpoint, scaling beyond several thousand examples would necessitate GPU batching and efficient sampling strategies to maintain feasible training times. Although the relation extraction models provided strong interpretability, they struggled with underrepresented classes compared to the NER models. Increasing dataset diversity and size—combined with the integration of contextual embeddings and data augmentation—is expected to enhance both performance and generalizability across larger legal text corpora.

Case study N^o 1: Facebook

This case study focuses on Facebook, one of the world’s largest and most influential social media platforms. After pasting Facebook’s terms and conditions into the input form, we initiated the prediction process and awaited the results.

Figure 9A illustrates the main visualization interface in TranspVis. The display presents extracted data alongside stakeholder complements, with each element identified by associated IDs. Users can interact with these IDs by clicking on them to visualize the corresponding detailed information.

Stakeholders detection: The NER model has extracted 15 different stakeholders represented in Fig. 9B. These stakeholders were added by the BERT model, which is the default model in our application. These stakeholders are standards because the BERT model was trained on commonly named entities.

We also have a choice of exploring and extracting other stakeholders that may not have been extracted by the BERT model by using a different NER model: spaCy, XLNET, or transformer-based models. Figure 10 represents a few results of the spaCy NER model.

We notice that there are many new results (in red), and we can choose the stakeholders that we need and that will help us in our decision-making process. Unlike the other NER models, spaCy was trained on our specific data, which is why it can extract stakeholders’ names like users, businesses, and advertisers. This is the best model for our case studies.

Relations Between Entities: Based on the main visualization circle in Fig. 9A, we notice that the binary model has predicted that there is an existing relation between several stakeholders and information elements. An example of a relation is shown in Fig. 11A.

The application added those relations as “Undecided” because we don’t know their types yet. To change the relation type, we need to click on the relation card and change it. Figure 11B represents how it is processed. Users can check the type relation model for suggestions on relation types. For example, the model might predict that a relation type should be “PRODUCTION” based on its analysis. They can apply their domain expertise to evaluate the model’s suggestion. In this case, we determined that “OPTIONAL” would be more appropriate than the suggested “PRODUCTION” relation type. Users can then select their chosen relation type from the drop-down menu in the interface. For this specific case, they would select “OPTIONAL” to finalize the relation type assignment.

We can do this with all the other relations. This method allows the user to choose the types freely and make their own decision based on the provided text and suggestions.

Case study N^o 2: WhatsApp

Moving on to our second case study, we shift our focus to WhatsApp, a leading messaging platform known for its widespread adoption and communication functionalities. After pasting the WhatsApp terms and conditions in the form, then waiting for the prediction. Figure 12 represents the main visualization in TranspVis.

Stakeholder detection: The NER model has extracted 15 different stakeholders represented in Fig. 13A. These stakeholders were added by the BERT model, which is the default model in our application.

We also have a choice of exploring and extracting other stakeholders that may not have been extracted by the BERT model by using a different NER model: spaCy, XLNET, or transformer-based models. Figure 13B represents a few results of the Transformers NER model.

We notice that this model gave us some new stakeholders, and it is our decision to choose whether we keep them or not.

Relations Between Entities: Based on the main visualization circle in Fig. 12, we notice that the binary model has predicted that there is an existing relation between several stakeholders and information elements. An example of a relation is shown in Fig. 14A.

The application added those relations as “Undecided” because we don’t know their types yet. To change the relation type, we need to click on the relation card and change it. Figure 14B represents how it is processed. Users can check the type relation model for suggestions about relationship types. In this case, the model has predicted that “PRODUCTION” would be the appropriate relation type. They can evaluate the model’s suggestion and decide whether to accept it. Here, we have chosen to follow the model’s recommendation of “PRODUCTION”. Users can then select the recommended “PRODUCTION” type from the drop-down menu in the interface. Finally, users can click the “Update” button to confirm and save the new relation type.

We can do this with all the other relations. This method allows the user to choose the types freely and make their own decision based on the provided text and suggestions.

Assessment approach

This study employs an online evaluation protocol (Qu & Hullman, 2017; Lam et al., 2011) to gather user feedback on the TranspVis application through a structured form, following a user experience (UX) evaluation scenario (Lam et al., 2011).

The protocol consists of four sequential steps:

• 1

  Understanding the working environment: Users received instructions for downloading the tool from GitHub (if not already installed), along with guidance on using TranspVis and understanding its context and objectives.

• 2

  Data explanation and visualizations: Participants were provided with an instructional video demonstrating TranspVis functionality and workflow.

• 3

  Specifying evaluation constraints: A structured questionnaire with multiple sections was created to facilitate comprehensive evaluation.

• 4

  Presentation: The evaluation was conducted online using Google Forms⁷ .

Participant demographics

The evaluation involved 18 participants representing three primary groups: (1) undergraduate and master’s students (aged 18–24) from information systems and computer science programs; (2) transparency and legal experts, including the authors and their colleagues; and (3) university professors specializing in digital ethics and policy.

Participants were recruited through academic mailing lists and professional networks to ensure representation from both technical and legal domains. The study employed a mixed-format approach: some experts accessed TranspVis directly from the GitHub repository in their professional environments and demonstrated its functionalities to colleagues, while students and other participants completed evaluations in supervised settings.

To ensure consistent understanding, all participants watched a short instructional video summarizing TranspVis’s key features before beginning evaluation tasks.

Evaluation tasks and metrics

Participants performed structured tasks designed to assess newly integrated TranspVis features, particularly entity detection, stakeholder weighting, and visual relationship exploration. The tasks required participants to:

• 1.

  Load a terms and conditions document into the system;

• 2.

  Identify and highlight key stakeholders detected by the model;

• 3.

  Explore visual relationships among entities using the transparency graph;

• 4.

  Interpret stakeholder weighting scores displayed in the interface.

After task completion, participants rated the clarity, usefulness, and intuitiveness of each functionality using a 5-point Likert scale (1 = very poor, 5 = excellent). The questionnaire also captured participants’ frequency of using applications that require terms and conditions review, with the following distribution:

• Always: 27.8%

• Frequently: 38.9%

• Occasionally: 16.7%

• Rarely: 16.7%

Evaluation results

The Table 7 below summarizes the form’s questions about various aspects of the “TranspVis” application that were evaluated by a scale [1…5]:

The previous table’s results are visualized in the following bar chart (See Fig. 15).

Users generally perceive TranspVis positively across various aspects such as functionalities, features, and overall satisfaction. The app receives high ratings for specific functionalities and features indicating strong performance. However, there are also areas identified for potential improvements.