Real-time food allergen detection using OCR-enhanced machine learning techniques

Limitations and gaps

The study on the food allergen dataset faces several limitations in the literature. Many datasets provide only basic information, such as food product names, primary ingredients, and allergens, while often lacking comprehensive data on sweeteners, fats, oils, and seasonings, which are crucial for understanding allergenic potential. Additionally, the datasets may oversimplify predictions by labeling food products without accounting for complex interactions between ingredients and allergens, limiting their applicability in diverse real-world scenarios. This study presents a novel machine learning-based system designed to read ingredient lists from scanned product labels. The system analyzes the extracted information to detect the presence of allergenic components, thereby supporting the prevention of allergic reactions and promoting public health. The proposed system takes food images as input, detects the presence or absence of allergens, and provides detailed information about identified allergens. It is designed for seamless integration with existing platforms, such as mobile applications. The study evaluates the performance of various machine learning classifiers using different feature extraction methods. The main contribution points of this study are:

• Development of an innovative machine learning-based system capable of analyzing ingredient lists from scanned product labels to detect the presence of allergenic components, contributing to the prevention of allergic reactions and promoting public health.

• Integration of Optical Character Recognition (OCR) for real-time extraction of ingredient information from product labels, ensuring accurate and efficient data retrieval.

• Proposal of a novel machine learning approach leveraging Logistic Regression (LR) for rapid and accurate detection of allergens, demonstrating superior performance in predicting allergic and non-allergic elements.

• Design of a system for seamless integration with existing platforms, such as mobile applications, enabling user-friendly access to allergen information for enhanced consumer safety.

The remainder of the article is organized as follows: ‘Related Work’ reviews related work and discusses key findings from previous research. ‘Proposed Methodology’ introduces the proposed system architecture, outlines the requirements, and summarizes the machine learning approaches used for food allergy detection. ‘Proposed Methodology’ details the dataset and methodological strategies for data preprocessing and feature engineering. The experiments and their outcomes are documented in ‘Results and Discussion’, while ‘Conclusions’ presents the conclusions drawn from the research and offers recommendations for future work.

Dataset

The dataset of food ingredients and allergies is sourced from Kaggle Dat (2023). The dataset consists of a collection of information regarding allergens present in various food items. The data contain information about allergy-related ingredients in different food items. The dataset consists of 400 records and seven attributes, each representing a specific food item and its associated allergens. A detailed list of all the allergens present in the food products is included in the dataset. These allergens include a broad spectrum of foods, including dairy, wheat, and nuts (almonds, peanuts, pine nuts), seafood (anchovies, fish, shellfish), grains (oats, rice), ingredients derived from plants (celery, mustard, soybeans), animals (pork, poultry), and common components (cocoa, eggs). Additionally, the dataset contains entries where no specific allergens are listed. Dataset attributes and their description are given in Table 2 and sample size and counts are shown in Table 3.

Data preprocessing

Data preparation is a crucial step that aims to increase the accuracy of the model, facilitate the processing of data by model, and enhance data quality (Shafique et al., 2024a). The food allergy dataset includes unstructured text data, containing elements such as stemming, stopwords, punctuation, and capital and lowercase characters. Text pre-processing is a vital step in the process of standardizing raw data. Table 4 shows some samples before and after applying the preprocessing steps. As shown in Table 5, individual food attributes such as sweeteners and allergens are merged into a unified representation.

Feature engineering methods

Feature engineering is the process of selecting, creating, or transforming features (input variables) in a dataset to improve the performance of machine learning models (Raza et al., 2022). It involves identifying relevant information from raw data and representing it in a format that is suitable for predictive modeling. This is done to help programs learn in a machine-learning system. Algorithms for machine learning can perform better when feature engineering is used. “Trash in, trash out” is a machine-learning proverb that is frequently used. According to this, the meaningless input leads to a meaningless output. However, more informative data could also lead to desired outcomes. Consequently, feature engineering may be used to extract relevant features from unprocessed data, hence improving the accuracy and consistency of learning algorithms. We employed three feature engineering techniques in this study: BoW, GloVe, and TF-IDF.

Machine learning model

In this section, we discuss the machine learning models used for allergen food detection. Machine learning techniques were implemented using Natural Language Toolkit (NLTK) and the Scikit-learn modules. The Sci-Kit module was utilized to implement the four supervised machine-learning algorithms using Python. Regression and classification issues are frequently resolved using supervised machine-learning techniques. A linear model is called LR, whereas a tree-based technique, RF is used. Other methods used to check the efficacy of the proposed system are KNN and Naive Bayes (NB) (probability). Table 10 displays the implementation details of the algorithms along with their hyperparameters.

The hyperparameter settings for the algorithms highlight a balance between simplicity and optimization tailored to the specific models. LR and NB use default settings, reflecting their robustness without extensive tuning. KNN employs n_neighbors = 3 to capture fine details in the data. For RF, the parameters include n_estimators = 300, random_state = 2, and max_depth = 100 to enhance model stability and prevent overfitting. Deep learning models such as CNN, GRU, and LSTM share advanced settings, including multiple filter sizes (16, 32, 64), the Adam optimizer for efficient learning, binary_crossentropy as the loss function, Dropout = 0.5 for regularization, and epochs = 100 for sufficient training. LSTM further incorporates maxpooling1D = 1 × 1 for feature selection. These configurations reflect a strategic approach to leveraging each algorithm’s strengths for optimal performance.

Evaluation method

The proposed model was evaluated using a two-fold approach. First, its performance was benchmarked against several state-of-the-art machine learning algorithms, including RF, KNN, LR, and NB, utilizing feature representations such as TF-IDF, BoW, and GloVe. Additionally, we explored deep learning architectures including LSTM, GRU combined with Recurrent Neural Network (RNN), CNN integrated with LSTM, and a hybrid model combining Gated Recurrent Unit (GRU), LSTM, and RNN. Second, the system underwent real-time testing on an independent dataset comprising 200 previously unseen food product images captured using a mobile camera, to assess its effectiveness in detecting allergens under varying real-world conditions.

Machine learning performance using TF-IDF features

Table 12 highlights the performance of machine learning models using TF-IDF features, with RF achieving the highest accuracy of 0.99, followed by LR at 0.96, NB at 0.95, and KNN at 0.87. TF-IDF enhances feature representation by emphasizing word importance, benefiting RF, LR, and NB with their ability to leverage weighted features effectively, while KNN struggles due to the high-dimensional feature space. RF excels due to its capacity to capture intricate relationships, while LR and NB show strong balanced metrics. Overall, TF-IDF improves model performance significantly, except for KNN, which faces limitations in handling high-dimensional data.

Machine learning models results using BoW features

Table 13 presents the results of machine learning models using BoW features, with LR achieving the highest accuracy (0.99), followed by RF (0.98), NB (0.96), and KNN (0.88). BoW transforms text into fixed-length vectors based on word frequencies, and this feature extraction technique benefits models like LR and RF, which effectively leverage frequency-based feature representations. RF performs strongly, achieving high precision and recall, especially for the “Contains” class. LR also performs exceptionally, with balanced precision and recall across both classes. KNN experiences performance degradation due to the sparsity and high dimensionality of BoW features, while NB performs decently with relatively balanced metrics. BoW improves performance for linear and tree-based models but presents challenges for distance-based models like KNN.

Machine learning performance using GloVe features

Table 14 presents the results of machine learning models using GloVe features, with RF achieving the highest accuracy of 0.79, followed by LR at 0.77, KNN at 0.74, and NB at 0.67. GloVe provides dense, pre-trained word embeddings that capture semantic relationships between words, which benefits models like RF and LR. However, the performance with GloVe features is lower than with other techniques such as TF-IDF or BoW. RF performs relatively well but struggles with class imbalance, showing lower recall for the “Does not Contain” class. LR exhibits similar performance, with a good precision-recall trade-off for “Contains” but struggles for “Does not Contain.” KNN’s performance is also hindered by the high-dimensional and dense nature of GloVe embeddings, while NB performs poorly, especially with the “Does not Contain” class. Although GloVe improves semantic understanding, but its impact on accuracy is less pronounced for these models than simpler methods like TF-IDF or BoW. The results shown in Fig. 3 compare the accuracy of three different feature extraction methods: TF-IDF, BoW, and GloVe. TF-IDF and GloVe achieved the highest accuracy, consistently scoring around 0.95–0.99, indicating their effectiveness in capturing features for classification tasks. In contrast, the BoW method showed lower accuracy, ranging between 0.67 and 0.88, reflecting its limitations in capturing semantic relationships between words. This highlights the strength of TF-IDF and GloVe in delivering better performance for tasks involving natural language understanding.

Figure 4 illustrates a LR training set, where each point represents a sample with two features plotted on the x and y axes. The purple points correspond to the positive class (labeled as “Contains”), while the green points represent the negative class (“Does not contain”). LR uses this labeled data to learn a decision boundary that separates the two classes by estimating the probability that a new point belongs to the positive class. The model aims to find the best line (or curve) that clearly distinguishes purple points from green ones, thereby enabling it to predict whether future, unseen samples are more likely to “contain” or “not contain” the target attribute.

In comparison, the BoW based confusion matrices demonstrate slightly lower performance, though they still reflect reliable classification results. While RF with BoW remains highly accurate, NB exhibits a higher rate of false positives and false negatives, suggesting that BoW may not capture text features as effectively as TF-IDF, as shown in Fig. 5. The confusion matrices for TF-IDF show strong classifier performance, with high true positive rates and minimal false negatives across all models. For example, the RF classifier with TF-IDF achieves 57 true positives and just one false negative, indicating near-perfect classification. LR also performs well, with only a few misclassifications as shown in Fig. 6.

The confusion matrices in Fig. 7 illustrate the performance of the machine learning model on a binary classification task using GloVe features. KNN correctly predicted 60 instances and misclassified 24, while LR performed slightly better, with 62 correct predictions and 20 misclassifications. NB showed relatively lower performance, with 56 correct predictions and 34 incorrect predictions. RF performed the best, achieving 64 correct predictions and 20 misclassifications, tied in misclassification count with LR but surpassing it in incorrect classifications.

Deep learning model performance for allergen detection

Table 15 compares the performance of four deep learning models (LSTM, GRU + RNN, CNN + LSTM, and GRU + CNN + RNN) on allergen food detection, based on accuracy, precision, recall, and F1-score. The GRU + RNN model achieves the highest accuracy (0.98), likely due to GRU’s ability to capture long-term dependencies in sequential data, and RNN’s effectiveness in handling temporal relationships. LSTM follows with a high accuracy of 0.96, as it is also adept at handling sequential data but typically performs slightly worse than GRU models in this context. The CNN + LSTM model, which combines convolutional and sequential processing, has the lowest accuracy (0.90), likely because CNNs are more suited for spatial tasks, and the sequential nature of the problem isn’t fully leveraged. The GRU + CNN + RNN model performs well with an accuracy of 0.95, but it doesn’t surpass the simpler GRU + RNN model, possibly due to added complexity and less optimal tuning. Overall, the GRU + RNN model stands out as the most effective for allergen detection in text-based data due to its ability to maintain relevant context across sequences. Machine learning models outperform deep learning approaches, especially in binary classification tasks (Sokoliuk et al., 2020), due to data requirements and model complexity differences. Deep learning models typically require larger datasets to learn effectively, and their performance may degrade with limited data due to overfitting or insufficient feature learning. Conversely, machine learning models can excel with smaller datasets, particularly with well-engineered features. Additionally, factors such as imbalanced data, suboptimal hyperparameter tuning, unsuitable architectures, and insufficient training time can hinder the performance of deep learning models. In our case dataset is small and has a smaller feature set, so the performance is lower compared with machine learning algorithms. These challenges emphasize the importance of matching the model complexity to the dataset characteristics and task requirements. The results shown in Fig. 8 compare the performance of various deep learning models like LSTM, GRU + RNN, CNN + LSTM, and GRU + RNN + CNN across metrics such as accuracy, precision, recall, and F1-score. All models perform consistently well, with scores ranging from 0.9 to 0.98. GRU + RNN + CNN and CNN + LSTM show slightly higher metrics, particularly in accuracy and precision, reaching up to 0.98, indicating their superior ability to capture and process complex patterns. Overall, these results highlight the robustness of hybrid architectures in improving performance across classification tasks.

Table 16 presents the performance of various deep learning models for food allergen classification. The evaluation metric used is the mean accuracy, which reflects the average classification performance over multiple runs. The standard deviation ( $\pm$std) indicates the variability or consistency of the model’s performance across those runs. A lower standard deviation suggests that the model performs consistently across different runs. Among all models, the GRU + RNN architecture achieved the highest mean accuracy of 0.98 with a low standard deviation of 0.02, indicating both high accuracy and reliability. The LSTM model also performed well with a mean accuracy of 0.96. Models like CNN + LSTM and GRU + CNN + RNN showed slightly lower accuracy, possibly influenced by the fact that they were trained on a smaller subset of the dataset due to data constraints. As expected, more complex models required longer training times, as reflected in the wall time column.

The results from our 10-fold cross-validation experiments reveal valuable insights into the comparative performance of different classification models and feature representation techniques as shown in Table 17. LR combined with BoW emerged as the most effective approach, achieving the highest accuracy of 0.91 ( $\pm$0.11). This performance is comparable to that of the more complex RF model, which scored 0.91 ( $\pm$0.09) using BoW. However, LR has the added advantages of being a simpler, more interpretable model with a probabilistic foundation, making it well-suited for many practical applications. Notably, LR outperformed its own variants when paired with TF-IDF (0.86 $\pm$ 0.07) and GloVe embeddings (0.68 $\pm$ 0.12), indicating that BoW provided the most effective representation for the dataset in question. Similarly, NB achieved the same accuracy as LR with BoW (0.91 $\pm$ 0.11), but LR is generally more robust and flexible, especially for imbalanced or complex data distributions. KNN, while showing reasonable performance with BoW (0.83 $\pm$ 0.13), lagged behind LR in both accuracy and computational efficiency.

Across all models, GloVe embeddings resulted in lower performance, suggesting that pre-trained word embeddings may not align well with the domain-specific vocabulary or context of the dataset. This further supports the effectiveness of simpler, corpus-driven methods like BoW in certain classification tasks. Regarding computational efficiency, wall time comparisons show that LR with BoW offers a practical balance. Although it is marginally slower than RF (132 vs. 1.52 ms per fold), the trade-off is justified given LR’s interpretability and comparable accuracy. Furthermore, LR is significantly more efficient than KNN, which requires 83.6 to 86.4 ms per fold yet delivers lower accuracy. These results confirm that LR with BoW is not only highly accurate but also computationally feasible and interpretable, making it a strong choice for text classification tasks in real-world settings.

Performance evaluation of the proposed model through real-time testing

In the online testing phase, we employed OCR using the EasyOCR package to extract text from an image. A total of 20 images per class were captured in real time and were processed to extract multiple blocks of text, including phrases such as “Low FAT VANILLA FLAVOURED YOGHURT” and “INGREDIENTS: Skim Milk, Concentrated.” Images of food product labels were captured using a Samsung Galaxy S21 smartphone equipped with a 12 MP rear camera, under typical indoor lighting conditions (300–500 lux). The device was held at a fixed distance of approximately 20 cm from the product surface to minimize distortion. To ensure consistency, images were taken against a plain background with minimal ambient glare. The EasyOCR library (version 1.7.2) was used for text extraction, running in a Python 3.8 environment. To assess robustness, label images were captured under three lighting conditions: natural daylight, fluorescent indoor lighting, and low light (below 150 lux) environments. Lighting-induced inconsistencies were addressed by converting images to grayscale and applying histogram equalization using OpenCV to normalize contrast. This preprocessing improved OCR text detection accuracy by reducing the impact of uneven illumination and shadow artifacts.

We evaluated the OCR system using product labels with font sizes ranging from 6 to 18 pt. EasyOCR maintained an accuracy of character recognition of 90% for fonts 10, with a minor decline (5%) in recognition for fonts below 8 pt. Labels with larger fonts showed near perfect OCR output. These results confirm the model’s robustness across practical font variations commonly found on commercial packaging. Once the text was extracted, it was concatenated into a single string and split into rows using commas as delimiters. These rows were further cleaned by removing extra whitespace and were then compiled into a new DataFrame. For preprocessing, the following steps were performed: the text was converted to lowercase, HTML tags and URLs were removed, numeric values were stripped, and stopwords were eliminated. Additionally, tokenization, stemming, and lemmatization were applied to process the words. After preprocessing, an LR machine learning model was employed to predict labels based on the cleaned data. For curved product surfaces (e.g., cylindrical bottles), the OCR pipeline includes a text localization step using bounding boxes to isolate text regions. These regions are corrected using affine transformations to reduce perspective distortion. The grid generator described in Fig. 2 assists in normalizing slanted or wrapped text blocks into flat 2D planes. This pre-correction significantly enhanced the readability and accuracy of OCR outputs on non-planar labels. Of the 40 images, 36 were correctly classified under the labels “Contains,” and “Does not Contains” indicating the presence of allergenic ingredients in the respective products with approx. 91% accuracy scores. In contrast, only one image was incorrectly predicted. The classification report and confusion matrix are presented in Table 18 and Fig. 9, respectively. This result further validates the effectiveness of the proposed model as a reliable tool for allergen prediction in various products.

The performance of the four deep learning models varies across epochs as shown in Fig. 10. All models achieve rapid convergence in training mean square error (MSE), with near-zero values indicating effective learning. However, validation MSE fluctuates significantly for LSTM and GRU-RNN, suggesting instability in generalization, while CNN-LSTM and GRU-CNN-RNN show consistently higher validation MSE, highlighting generalization challenges despite minimal fluctuations. These trends indicate potential overfitting or complexity issues affecting validation performance across models. In our proposed framework, through real-time testing, as shown in Table 19, the system successfully detected allergen-related terms (e.g., “nuts”) while avoiding false positives on allergen-free items. This visual demonstration supports the quantitative findings reported earlier and reflects the practical utility of the proposed approach in consumer-facing applications.

Table 19:

OCR results and allergen classification in real-time food label analysis.

Image	Extracted OCR text	Predicted label	Ground truth
	PARBOILED RICE	No allergens detected	Correct (No allergen)
	Energy: 117 kcal
	Salt: <0.01g
	Fat: <0.5 g
	LIDL product	Likely contains allergens	Correct (Possible allergen)
	Fat: 48.8 g
	Saturates: 8.5 g
	Protein: 21.3 g
	Omega-3: 9.9 g	Contains allergens (nuts)	Correct (Nut allergen)
	Protein: 15.5 g
	Note: may contain nuts

DOI: 10.7717/peerj-cs.3338/table-19

Comparison with other state-of-the-art models

To show the importance of our recommended plan more clearly, we have compared its results with some recent top-notch systems. Table 20 shows how well the new model beats the known best methods in Allergy classification found so far. It is clear from this comparison that our proposed idea works better than current top-notch literature techniques for classifying allergens. It also shows the given model can find allergens with 99% accuracy on average. This is better than other advanced techniques in studies.

While previous allergen datasets have contributed valuable insights, they are often limited in scope, containing only basic ingredient and allergen labels without accounting for complex food compositions such as sweeteners, fats, oils, and seasonings. These missing components may play a critical role in allergenic responses but are often overlooked, reducing the dataset’s effectiveness for comprehensive analysis or real-time application. It is important to note that the datasets used in the comparative studies vary significantly in terms of sample size, feature granularity, and target allergen classes. For example, the study by Ktona et al. (2022) used only 155 medical records focused on detecting allergies in Albania, while Wang et al. (2021) used large; multi-source datasets (e.g., SDAP, NCBI). In contrast, our proposed model was evaluated on a balanced and moderately sized dataset containing detailed ingredient-level attributes. Therefore, although the proposed method outperforms others in accuracy, it is essential to interpret these results within the context of dataset complexity, generalizability, and domain specificity. Our proposed system addresses these gaps by incorporating a more detailed feature set extracted via OCR from real product labels. This includes secondary and hidden ingredients, enabling a more nuanced and realistic allergen detection model. Furthermore, by integrating this approach into a real-time OCR-enhanced machine learning pipeline, our system not only improves detection accuracy but also ensures practical usability for consumers, distinguishing it from previous predominantly offline or small-scale efforts.

To further investigate the impact of learning rate and batch size on the performance and convergence behavior of deep learning models, we conducted additional experiments by fixing the batch size to 128 and varying the learning rate across 0.1, 0.01, and 0.02. These configurations were chosen based on standard tuning practices and to address fluctuations observed in earlier training loss curves. The performance metrics for each model under these settings including accuracy, precision, recall, and F1-score are summarized in Table 21. The results indicate that all models maintained strong classification performance, suggesting that the selected hyperparameter ranges support effective learning and reduce the likelihood of overfitting or underfitting.

Limitations and future directions

While this study makes a significant contribution to real-time allergen detection, several limitations remain that provide opportunities for further improvement. One key challenge lies in enhancing the accuracy of real-time predictions, which can be impacted by OCR errors due to poor image quality, complex label designs, or spelling mistakes in extracted text. For future work, we plan to extract additional features from images to enrich the input data and further improve classification performance. We also intend to incorporate a large language model to correct ingredient spelling errors, which may arise from OCR inaccuracies and negatively affect the prediction outcomes. Specifically, we plan to expand the dataset by integrating food items from diverse and authoritative sources, such as government allergen databases and international food regulatory bodies. This enhancement aims to improve the dataset’s representativeness and the model’s generalizability across regional and global food products, ultimately contributing to the novelty and broader applicability of our proposed method. Additionally, we are exploring the deployment of the proposed framework as an Android application, with the goal of enabling real-time allergen detection and improving accessibility for end users in practical settings.