AI detectors are poor western blot classifiers: a study of accuracy and predictive values

Sample size determination

The number of western blot images included in the accuracy study was determined by a sample size calculation based on the following formula (construction of an interval based on the normal approximation when the classification status is known at the time of sampling) (Hajian-Tilaki, 2014):

$$n={\displaystyle \frac{Z_{\left(1-\alpha /2\right)}^2\ast S\ast \left(1-S\right)}{M^2}}$$where Z is the standard normal value at 1−α/2, S is the anticipated sensitivity (or specificity), and M is the maximal margin of error. A crude pilot investigation showed that the sensitivity and specificity of online AI detectors are relatively low when trying to identify western blots; they detected fake images about half of the time. With Z = 1.96, a predicted sensitivity of 0.6, and a margin of error set at 0.1, the calculation gives an estimated sample size of 92. In the absence of more precise information about the actual sensitivity and specificity of AI detectors, it was decided to evenly balance the number of AI-generated images and authentic western blot images (46/46) in the final library, thus giving a prevalence of 0.5 for the feature to be detected (i.e., a fake image). We decided to increase the sample size to 48 in each group to account for potentially unusable images.

Statistical analysis

Analyses were performed using RStudio (version 2023.06.2 + 561; RStudio Team, 2023) or GraphPad Prism (version 10) as specified in the openly provided files. Graphics were made with GraphPad Prism. The confidence level was set at 95%, which corresponds to a false positive risk (type I error) of 0.05 (i.e., 5%). The sensitivity (the proportion of AI-generated images that are correctly categorised), specificity (the proportion of authentic western blots categorised as fake), and accuracy (the proportion of correctly categorised images) were calculated for each AI detector by using Microsoft Excel for Mac (v. 16.77.1) and confirmed using the caret package in R using the counts of detector outcomes as follows:

$$Sensitivity={\displaystyle \frac{CorrectlycategorisedAI-generatedblots}{CorrectlycategorisedAI-generatedblots+Falselyauthenticblots}}$$

$$={\displaystyle \frac{TP}{TP+FN}}$$

$$Specificity={\displaystyle \frac{Correctlycategorisedauthenticblots}{Correctlycategorisedauthenticblots+FalselyAI-categorisedblots}}$$

$$={\displaystyle \frac{TN}{TN+FP}}$$

$$Accuracy=\left(0.5\ast Sensitivity\right)+\left(0.5\ast Specificity\right)$$where TP, FN, TN, and FP indicate counts of true positive, false negative, true negative and false positive results, respectively. PPV and NPV were calculated for each AI detector in Microsoft Excel for Mac (v. 16.77.1) as follows, using an increasing AI prevalence (from 0 to 0.5):

$$PPV={\displaystyle \frac{Sensitivity\ast Prevalence}{\left(Sensitivity\ast Prevalence\right)+\left[\left(1-Specificity\right)\ast \left(1-Prevalence\right)\right]}}$$

$$NPV={\displaystyle \frac{Sensitivity\ast \left(1-Prevalence\right)}{\left[\left(1-Sensitivity\right)\ast Prevalence\right]+\left[Specificity\ast \left(1-Prevalence\right)\right]}}$$

The AI probabilities are given to one decimal place, whereas the sensitivity, specificity, accuracy, PPV, NPV, and the area under the receiver operating characteristic curve (ROC AUC) are reported to four decimal places.

Generation of fake western blot images

Fake western blot images were generated with ChatGPT 4 (https://chatgpt.com), which uses the DALLE-3 interface. Prompts, chats, and image collection were done between 20 May and 23 May 2024. The query method was based on pilot tests that evaluated the efficacy of ChatGPT 4 at creating western blots. We used repeated prompts asking for a ‘realistic image of a western blot’ while progressively changing the query to get new images, each time trying to guide the algorithm to a realistic western blot image. The entire prompting history was documented and saved. Every realistic western blot image from which four distinct bands from different lanes could be isolated was saved and used for pre-processing. Images that were not satisfactory were also saved for documentation. Four distinct chats were used to create 10–15 images each. The WEBP images created by ChatGPT were converted to PDF files to reproduce the initial format of authentic western blot images sampled from articles.

From a single query, the number of western blot bands displayed on the images generated by ChatGPT 4/DALLE-3 can vary greatly. To our knowledge, this fickleness is not, controllable by the prompt, apparently because ChatGPT does not easily correctly identify the term ‘band’ or ‘lane’. However, the number of bands in a western blot might influence the classification by AI detectors. Therefore, the choice was made to standardise the images by cropping them (by selective screen capturing, in the PNG format) to keep only four lanes, with no space to the left of the left-most band or the right of the right-most band, and leaving a space equivalent to one tenth of the width of a lane above and below the bands (see Fig. 1). To preserve independence between images and to reduce intraclass correlation, only one cropped image was taken from each AI-generated image. To investigate how the number of lanes in an image affected detector performance, we created a second dataset by isolating just two lanes from the original images. This was achieved by cropping each image to retain only the two centremost lanes.

Sampling of authentic western blot images

Authentic western blot images were sampled from published articles. To mitigate the risk of collecting AI-generated images, photos were collected from articles published in 2015, which was before the rise of GenAI content that has been observed since 2020 (https://www.wipo.int/web-publications/patent-landscape-report-generative-artificial-intelligence-genai/en/introduction.html). Images were sampled from four journals that frequently publish western blot figures:

• The Journal of Biological Chemistry (electronic International Standard Serial Number (ISSN) 1559-1182, 12 articles);

• Oncogene (electronic ISSN 1476-5594, 12 articles);

• Public Library of Science (PLOS) Biology (electronic ISSN 1545-7885, 12 articles);

• Cancer Research (electronic ISSN 1538-7445, 12 articles).

A systematic sampling scheme was used to collect the articles as follows. The final 2015 issue of each journal was examined, and each article with a figure containing a western blot image was sampled (PDF file). Only one image was sampled from each article, and only blots with one band per lane that is seen distinctly within four adjacent lanes was sampled. If the figures showed the detection of multiple proteins, then priority was given to the housekeeping protein (e.g., actin, tubulin); if no housekeeping protein was displayed, then blots of the first probed protein (scanning from top to bottom) with a single band was sampled. If multiple western blot figures were present in one article, the first western blot appearing in the article that fulfilled the aforementioned conditions was sampled. This sampling method was applied to suitable western blot images from consecutive articles, chosen while moving from the beginning to the end of the issues, until the required number of images had been obtained.

The following exclusion criteria were predefined and applied:

• Blots from immunoprecipitation or pull-down assays (because they might have specific background or signal intensity);

• Blots either shown in colour or displaying white bands against a dark background (although these are the result of natural luminescent image acquisition, they are colour-reversed images compared with convention);

• Images with less than four lanes;

• Images with inserts such as a highlighted area, framed zones, arrows, or text;

• Conference abstracts, reviews, or perspectives articles.

Authentic western blot images were captured using selective screenshots, and the same protocol was employed as for AI-generated blots (i.e., keeping only four lanes, leaving no space to the left and right of the bands, a space corresponding to one tenth of a lane width above and below the bands, and generating another image by cropping the photograph to keep the two central lanes).

Selection and performance analysis of AI detectors on Western blots displaying four or two lanes

A Google search using the query ‘Free AI image detector’ was performed on 17 May 2024 in Lausanne (Switzerland). The first three results that corresponded to free websites that did not require a subscription were used: Is It AI? (https://isitai.com/ai-image-detector/), Hive Moderation (https://hivemoderation.com/ai-generated-content-detection), and Illuminarty (https://app.illuminarty.ai/). Each image was scanned with all three of the AI detectors. The output of the AI detectors is a probability, given as a percentage, that the image is AI generated. Each image was classified as a TP, a TN, an FP, or an FN, with positivity defined as a detector output probability above 50%. In addition, the PPV and NPV were calculated for each detector. Two detector outputs were included in the analysis: (1) image classification determined by the detector outputs, which were used in the confusion matrix to calculate the sensitivity, specificity, accuracy, PPV, and NPV; and (2) the absolute AI probability returned by the detectors.

Data storage and sharing

AI images were saved in the WEBP format generated by ChatGPT, converted to the PDF format, and then cropped screenshot images in the PNG format were created and stored for analysis. Authentic western blot images were saved, processed, stored, and analysed in the PNG format from screenshots; no digital modifications, such as contrast, were applied to the images. All data were saved, stored, and shared on a publicly available Figshare repository (https://figshare.com).

A data folder accessible at https://doi.org/10.6084/m9.figshare.26300464 contains:

• The entire prompting history used to create the images;

• An Excel spreadsheet that summarises all quantitative analyses;

• An Excel spreadsheet that provides details of all sampled articles;

• Comma-separated value (CSV) files for each specific data set;

• The R codes used to analyse the data;

• GraphPad Prism files used to generate the figures.

A data folder accessible at https://doi.org/10.6084/m9.figshare.26300515 contains:

• The cropped versions of the authentic western blots;

• The full AI-generated images and their cropped versions;

• The unused (failed) ChatGPT 4 images.

A preprint version of this study is available at https://doi.org/10.48550/arXiv.2407.10308.

AI detectors showed highly variable abilities to distinguish between artificial and authentic western blots

Following the assessment of the western blots with four lanes, the returned AI probabilities spanned a very wide range for each AI detector (Fig. 2A). Illuminarty gave the highest average AI probabilities (median = 86.2, interquartile range (IQR) [42.2–98.7] for AI-generated western blots, median = 81.1, IQR [19.2–99.1] for authentic western blots). The lowest AI probabilities were returned by Hive Moderation (median = 17.0, IQR [9.5–30.3] for AI-generated western blots, median = 18.2, IQR [8.3–37.0] for authentic western blots). Is It AI? produced an intermediate output that was visibly dissimilar between AI-generated western blots (median = 85.1, IQR [69.8–93.1]) and authentic western blots (median = 47.3, IQR [26.9–66.0]). This variability in detector output was reflected in the formal confusion matrices and accuracy analyses presented in Fig. 3B. The proportion of false positives was 22/48 for Is It AI?, 6/48 for Hive Moderation, and 28/48 for Illuminarty, and the proportion of false negatives was 2/48 for Is It AI?, 39/48 for Hive Moderation, and 14/48 for Illuminarty. As presented in Fig. 4A, the consequence of these high rates of misclassifications is that the performance of the detectors often fell short of usual standards (Is It AI?: sensitivity = 0.9583, specificity = 0.5417, accuracy = 0.7500, ROC AUC = 0.9028, 95% CI [0.8435–0.9621]; Hive Moderation: sensitivity = 0.1875, specificity = 0.8750, accuracy = 0.5313, ROC AUC = 0.5100, 95% CI [0.3930–0.6270]; Illuminarty: sensitivity = 0.7083, specificity = 0.4167, accuracy = 0.5625, ROC AUC = 0.5449, 95% CI [0.4276–0.6622]).

As shown in Fig. 2B, when the scanned western blots were cropped from four lanes to two lanes, the detectors returned markedly reduced AI probabilities (Is It AI?: median = 62.8, IQR [38.5–80.8] for AI-generated western blots, median = 11.7, IQR [6.2–28.8] for authentic western blots; Hive Moderation: median = 6.2, IQR [3.1–18.0] for AI-generated western blots, median = 12.2, IQR [5.6–24.1] for authentic western blots); Illuminarty: median = 15.4, IQR [7.8–43.5] for AI-generated western blots, median = 9.1, IQR [2.4–29.0] for authentic western blots). Consequently, there was a dramatic increase in the number of false negatives (Is It AI?: 20/48; Hive Moderation: 46/48; Illuminarty: 38/48) and a reduction in the number of false positives (Is It AI?: 4/48, Hive Moderation: 4/48, Illuminarty: 8/48; Fig. 3C). Figure 4B shows that although the overall test accuracies were not noticeably impacted by reducing the number of bands (Is It AI?: accuracy = 0.7500, ROC AUC = 0.8974, 95% CI [0.8361–0.9586]; Hive Moderation: accuracy = 0.4792, ROC AUC = 0.6252, 95% CI [0.5118–0.7386]; Illuminarty: accuracy = 0.5208, ROC AUC = 0.6248, 95% CI [0.5115–0.7380]), there was reduced sensitivity (Is It AI?: 0.5833; Hive Moderation: 0.0417; Illuminarty: 0.2083) along with increased specificity (Is It AI?: 0.9167; Hive Moderation: 0.9167; Illuminarty: 0.8333).

Evaluation of the PPVs and NPVs of the AI detectors

Sensitivity and specificity are intrinsic properties of the detectors that describe their ability to correctly classify images as authentic or fake. However, in editorial contexts where the question is whether a given image can be deemed authentic, the PPV and NPV would be more useful; these measures are functions of both sensitivity and specificity, as well as the prevalence of AI-generated images in the literature. Therefore, we calculated the PPV and NPV for each of the three AI detectors in scenarios in which AI prevalence increased sequentially from 0 to 0.5 (Fig. 5). The data from western blots with four lanes (Fig. 5A) showed that the PPV of each detector was very low when the AI prevalence was set below 0.1, with a maximum value of 0.1885 for Is It AI? (0.1885) when the AI prevalence was set at 0.1. This indicates that most of the western blots categorised as AI generated would be false positives. When further reducing the AI prevalence in the simulations, the PPV become dramatically low. For example, at a prevalence of 0.005 (meaning that we expect one western blot out of 200 to be AI generated), the PPV was 0.0104 for Is It AI?, 0.0075 for Hive Moderation, and 0.0061 for Illuminarty. Conversely, the NPV was greater than 0.9 for all detectors at this realistic AI prevalence, indicating that false negatives would be rare in this scenario. At higher AI prevalences, the PPV for each detector increased steadily as the NPV decreased, although a discrepancy was observed between the relatively high NPV of Is It AI? (NPV = 0.9285 at an AI probability of 0.5) and the NPV of Hive Moderation (NPV = 0.5185 at an AI probability of 0.5) and Illuminarty (NPV = 0.5185 at an AI probability of 0.5). The PPVs and NPVs calculated using western blots with two lanes (Fig. 5B) showed patterns for Hive Moderation and Illuminarty that were similar to those obtained from four-lane blots. However, Is It AI? showed a higher PPV than that of the two other detectors, even for low AI prevalences, and a lower NPV than when blots with four lanes were analysed.