Generative artificial intelligence and machine learning methods to screen social media content

Rationale for the case example topic: E-cigarette and pregnancy content on TikTok

In the current study, we examine the capacity of OpenAI’s ChatGPT as a generative AI model to screen social media content from TikTok related to a salient public health topic: use of e-cigarettes during pregnancy. We chose this topic due to limited research examining social media content in this topic area, despite its relevance to public health. This topic was chosen as an illustrative example, but our methods can be applied across a broad range of other topics on social media.

Electronic cigarettes (i.e., e-cigarettes) have been the most frequently used nicotine/tobacco product among youth and young adults for the past decade (National Center for Chronic Disease Prevention and Health Promotion (US) Office on Smoking and Health, 2016; Cullen et al., 2018; Cornelius et al., 2020), many of whom are now reaching child-rearing age. E-cigarette use among pregnant people was selected as a case example because it is a timely topic with clinical and public health relevance. Combustible cigarette smoking during pregnancy is linked with clear and well-substantiated risks for adverse health outcomes for the pregnant person and baby, as communicated to the public through effective public health messaging and advice from healthcare providers (Office of the Surgeon General (US) & Office on Smoking and Health (US), 2004; Wickstrom, 2007; Allen, Prince & Dietz, 2009; Einarson & Riordan, 2009; Himes et al., 2013; Allen, Oncken & Hatsukami, 2014; National Center for Chronic Disease Prevention and Health Promotion (US) Office on Smoking and Health, 2014; Allen et al., 2018; Gould et al., 2020; Morales-Prieto et al., 2021).

In contrast, the potential impacts of e-cigarette use during pregnancy are less thoroughly understood, with mixed findings from human studies as to the risks of e-cigarette use during pregnancy, including from clinical trials, pregnancy cohort studies, and population survey data (DeVito et al., 2021). Although the data is still incomplete, the current available evidence (clinical, epidemiological data, preclinical, and in vitro data) taken together indicate that e-cigarette use (exclusive e-cigarette use or dual use of e-cigarette and combustible cigarettes) during pregnancy most likely poses greater risk than abstaining from all tobacco products, but exclusive e-cigarette use may be less harmful than smoking combustible tobacco. These mixed findings can complicate the development and dissemination of cohesive messaging on platforms such as social media. Analysis of social media content created by the general public can also provide a snapshot into the public’s current understanding and opinions about the use of e-cigarettes during pregnancy.

Power of combining machine learning with generative AI to screen social media content

Machine learning methods are valuable for establishing a systematic and automated framework to analyze data from social media platforms (Visweswaran et al., 2020). While machine learning can examine large amounts of data quickly, datasets typically still require human review. For example, unsupervised models such as clustering techniques can distill large amounts of data into fewer parts (i.e., clusters) for further analysis. However, clusters become uninterpretable if there is too much noise resulting from irrelevant data. Therefore, human review is typically needed to ensure that posts include content related to the primary topic(s) of interest (e.g., e-cigarettes and pregnancy).

Additionally, traditional machine learning classifiers can be stringent and rely on a finite dictionary list of words, allowing limited flexibility in text data classification through processes like stemming and lemmatization. Similar limitations exist for screening methods such as keyword filtering, which requires pre-identification of a comprehensive set of keywords to identify relevant content. Due to the rapidly evolving nature of social media, the terms used to describe topics such as vaping and pregnancy are constantly changing, calling for more flexible approaches that look for patterns in text rather than discrete terms. Human review of content is also frequently used to screen for relevant content, particularly for complex topics where computational methods such as keyword filtering may miss content. However, human review can be a time consuming and cumbersome process for large-scale social media datasets.

Advancements in artificial intelligence, as seen in generative AI, present exciting new opportunities to apply data science techniques to both identify and characterize social media content. Recent developments in AI technology have broadened the scope and influence of large-scale social media research. For example, ChatGPT (OpenAI, 2023) is an LLM trained on a vast web corpus that has demonstrated state-of-the-art performance on a range of natural language tasks (Kasneci et al., 2023). ChatGPT’s models, including GPT-4, are trained on a diverse array of sources including books and websites. Its transformer architecture and advanced natural language processing capabilities can be leveraged to efficiently analyze patterns across data and make relevance assessments (OpenAI, 2023). ChatGPT’s natural language capacity can eliminate the need for manual screening and pre-training for concepts like e-cigarettes and pregnancy, instead using existing definitions informed by ChatGPT’s vast corpus of data, elicited using specific prompts.

Prompt engineering is a technique for developing and optimizing requests to elicit a desired output from LLMs such as ChatGPT (Brown et al., 2020). In an application of ChatGPT’s capabilities, Li et al. (2024) utilized the LLM through prompt engineering on a dataset consisting of 3,481 hateful, offensive, and toxic (HOT) comments posted on Reddit, X (formerly Twitter), and YouTube. Their study aimed to compare ChatGPT outputs to a previous coding analysis conducted by human coders with the same dataset. They observed that ChatGPT was generally reliable and consistent, achieving an 80% accuracy rate in comparison to human coders. ChatGPT therefore presents unique potential to streamline social media research by facilitating quicker screening of content as relevant or irrelevant to a specific topic. In our current study, we examine ChatGPT’s capacity to screen TikTok posts for e-cigarette and pregnancy-related content.

However, much is still unknown about ChatGPT and its potential limitations when screening social media content. LLMs like ChatGPT can experience a loss of dialogue or “contextual forgetting” in conversational threads due to limitations in memory processing and retaining extensive context in conversations (Liang et al., 2023). Furthermore, previous work has found that ChatGPT can produce erroneous outputs or “hallucinations” (Alkaissi & McFarlane, 2023). Therefore, further comparison between ChatGPT and other commonly used methods, such as screening by human coders, is needed to evaluate ChatGPT’s capacity to identify specific types of content and topics.

Machine learning techniques to prepare social media data for ChatGPT

Another important limitation of ChatGPT in its current form is its limited capacity to examine videos, a format that is central to popular platforms such as TikTok. ChatGPT has begun to incorporate more image and video analysis features. However, ChatGPT’s capacity to examine text still far exceeds its capacity to examine image-based content. To overcome this obstacle, machine learning techniques such as computer vision and natural language processing can facilitate the conversion of video-based content into written transcripts, facilitating text-based screening of social media content by ChatGPT.

Previous work (Pinto et al., 2024) on social media has taken multimodal approaches to analyze online communities, including using data scraping tools like Zeeschuimer and PykTok to extract community metadata from TikTok (Zehrung & Chen, 2024; Van Berge, 2023). The multimodal approaches used in this study extend these methods by including the acquisition of audio, image, and text data from TikTok posts, converting the audio and image data into text, and using ChatGPT to identify content related to vaping and pregnancy. To convert video-based social media content into text-based data, we developed a multimodal system that automates the extraction and integration of image and audio content into text using computer vision techniques (i.e., object detection and optical character recognition) and natural language processing (i.e., automatic speech recognition).

One computer vision technique that is ideal for converting image data into text is object detection. Object detection is a computer vision technique that identifies and locates objects of interest within images or videos by drawing bounding boxes around the detected objects and classifying them into predefined categories (Mack et al., 2008; Xiao et al., 2020). State-of-the-art object detection algorithms leverage deep learning, feature extraction and integration, and machine learning techniques to accurately recognize and locate objects with minimal training (Xiao et al., 2020; Wang, Bochkovskiy & Liao, 2023). Recent studies have employed object detection technology to accurately identify e-cigarette content on TikTok, detecting the presence of objects such as vape devices and vapor clouds (Murthy et al., 2024), as well as e-cigarette containers, brand names, and warning labels (Vassey et al., 2023). Additionally, previous research has leveraged deep learning technology to identify the presence of e-cigarette devices (Vassey et al., 2020) and FDA warning labels (Kennedy et al., 2021) in Instagram posts. By applying object detection techniques, we can therefore identify posts that are likely to include e-cigarette content by detecting specific objects (e.g., e-cigarettes) associated with specific topics.

In addition to objects, TikTok posts frequently include text incorporated within video content. Optical character recognition (OCR) is a type of computer vision technique that converts images of text into machine-readable and editable text data (Hamad & Kaya, 2016). The process of OCR involves detecting, recognizing, and extracting characters, words, and lines of text from images or scanned documents. OCR algorithms may employ image preprocessing, feature extraction, and machine learning techniques to accurately recognize and extract textual information (Hamad & Kaya, 2016; Memon et al., 2020). OCR may be used in social media research to enable the extraction of textual data from images. Previous work (Salma et al., 2021) has used both object detection and OCR computer vision techniques to detect and extract relevant information from visual data, such as license plates. When applied to social media, the employment of both object detection and OCR techniques can facilitate the conversion of image-based information into text-based data. This text-based data can then be entered into ChatGPT to screen for specific topics such as vaping and pregnancy.

Finally, TikTok posts often include an individual or individuals talking to the camera, resulting in large amounts of relevant information being communicated through audio. Automatic speech recognition (ASR) is a natural language processing technology that enables computers to process and convert human speech into text. Modern ASRs, like OpenAI’s Whisper (OpenAI, 2022), use end-to-end neural network models that can directly map audio waveforms to text, bypassing the intermediate phonetic elements used by traditional ASR systems (Ghai & Singh, 2012). Previous research has showcased Whisper’s potential in health communication by providing a tutorial for the ethical implementation of the software in an audio transcription pipeline for psychiatry, psychology, and neuroscience research (Spiller et al., 2023). Moreover, advancements in deep learning technology have greatly refined ASR accuracy as evidenced by studies examining its improving reliability to capture different languages (Mukhamadiyev et al., 2022) and accents (Najafian & Russell, 2020). ASRs such as OpenAI’s Whisper can therefore facilitate the conversion of speech to written text for further analysis by ChatGPT.

Current study

The primary objective of this study is to develop a multimodal system for converting video-based content into text data, with the resulting text data entered into ChatGPT to examine its capacity to screen TikTok content for specific topics (i.e., e-cigarettes and pregnancy). Our multimodal system automates the extraction and integration of comprehensive information from videos into a large, text-based dataset using machine learning techniques including object detection, optical character recognition, and automatic speech recognition. We then developed a prompt procedure for identifying TikTok posts containing e-cigarette and pregnancy content, comparing ChatGPT’s determinations to a human coder to better understand its strengths and limitations when screening content. Our research questions include:

Research Question 1: How can machine learning be used to develop a multimodal system that automates the extraction and integration of comprehensive text-based data from videos in a large dataset of TikTok posts?

Research Question 2: What is ChatGPT’s capacity and accuracy, as compared to human coding, when screening large social media datasets for two content areas simultaneously (i.e., e-cigarettes and pregnancy)?

Overview of data collection, aggregation, and analytic methods

We obtained TikTok metadata from N = 23,888 videos using Zeeschuimer (Digital Methods Initiative, 2023), a scraping tool. After a deduplication process, we identified N = 11,673 unique videos. We employed the PykTok library (Freelon, 2022) to download video content for these 11,673 posts using their video IDs. We then used Whisper (OpenAI, 2022) to transcribe audio content from each video. Our multimodal approach then integrated computer vision techniques to create a comprehensive text-based dataset. Specifically, we used the OpenCV library (Bradski, 2000) to extract still frames of each video. After extracting still frames, we used the AI service Oracle Cloud Infrastructure (OCI) for image processing (Oracle Corporation, 2025). OCI’s module, Oracle Cloud Vision, is an AI service used to conduct extensive deep-learning-based image analysis. This module leverages OCR to extract and analyze textual information. These methods are summarized visually in Figs. 1A–1G, detailing the sequential steps involved in our data collection and data compilation process. Finally, prompts were generated through an iterative process in ChatGPT-4 to screen the resulting text-based datasets for content discussing e-cigarettes and pregnancy. Additionally, we conducted frequency analyses of the objects and texts detected within posts, enriching our understanding of the dataset. These analytical methods are summarized visually in Figs. 1G–1L.

Data processing steps are illustrated in panels a–f, resulting in a processed dataset (j) comprised of three key components (g–i), which were fed to ChatGPT for classification into vaping, pregnancy, or both vaping and pregnancy categories, represented by (k). Additionally, the objects and text detected in posts (i) underwent frequency analysis (l) through Oracle Cloud Vision.

Our research methods are informed by the Association of Internet Researchers (AoIR) research integrity guidelines including providing a detailed methodology and transparent reporting of results, pursuing review by an ethics committee or institutional review board (IRB), and minimizing use and sharing of identifiable information (Franzke et al., 2020). Specifically, the Yale Institutional Review Board determined the work to be non-human research and therefore exempt from further IRB review. No images were input into ChatGPT to minimize use of identifiable information, and only publicly available content was used for analysis.

Data collection

We searched for e-cigarette and pregnancy-related content on TikTok using 70 pairs of thematic hashtags related to “pregnancy” (e.g., #pregnancytok, #momtok, #pregnancy) and “vape” (e.g., #vape, #smoke, #juul; see Appendix A, Table S1). These hashtags were identified by starting with six broad keywords related to both pregnancy and vaping (“e-cigarette pregnant”, “e-cigarette pregnancy”, “vaping pregnant”, “vaping pregnancy”, “vape pregnant”, “vape pregnancy”). We then examined the hashtags used in posts containing both vaping and pregnancy content to identify the hashtags and hashtag pairs used most frequently across relevant posts. A total of 70 pairs of hashtags were identified as being related to vaping and pregnancy, reflected in Appendix A, Table S1.

We commenced data collection in February 2024 using Zeeschuimer (Digital Methods Initiative, 2023), a browser extension optimized for scraping data from social media platforms, including TikTok. Following methods employed by previous researchers (Van Berge, 2023), we used Zeeschuimer to scrape TikTok data by manually scrolling through the site interface. The Zeeschuimer extension exports scraped metadata in Newline Delimited JSON (NDJSON) format, which is a structured file format commonly used for storing extracted data.

This dataset (N = 23,888) contained unique identifiers (i.e., IDs), titles for each video, video descriptions provided by the content creator, and engagement metrics (e.g., the number of likes, shares, comments, and overall view counts) (see Fig. 1A). To ensure data uniqueness in our extensive dataset, we conducted a deduplication process, by identifying and merging duplicate entries based on the unique video IDs, successfully reducing our dataset to 11,673 unique, unprocessed video metadata entries (see Fig. 1B).

We used PykTok library (Freelon, 2022) to download the identified TikTok videos (see Fig. 1C). The PykTok module retrieves data directly from TikTok web pages and undocumented application program interfaces (APIs), enabling the download of videos and metadata for local storage. PykTok streamlined the download process by generating direct URLs for each identified video, which we used to acquire video content corresponding to the cleaned dataset of video IDs.

Dataset preparation using machine learning

Data screening using ChatGPT

ChatGPT relevance classifier results

The ChatGPT Relevance Classifier analysis identified TikTok posts related to both vaping and pregnancy, using the capabilities of ChatGPT-4 through prompt engineering (see Fig. 1K). ChatGPT determined that 44.86% (N = 5,237) of the screened videos were relevant to pregnancy only, while 36.91% (N = 4,311) of videos pertained to vaping only. Only 8.91% of the dataset (N = 1,040) contained videos that ChatGPT determined were relevant to both pregnancy and vaping.

Figure 2 features a still where both “pregnant” and “vape” appear in the textual content. The transcript and captions explicitly mention pregnancy and vaping, highlighting the classifier’s ability to accurately identify and categorize posts that intersect these two topics. In cases where the video only mentioned topics related to “pregnancy” and “vape,” ChatGPT would generate a “no” response.

Object detection results

To further illustrate how the objects and text were detected, we examined the frequency of objects detected across the 1,040 posts identified by ChatGPT as relevant. OCI Vision identified over 75% of the videos as featuring people (86.54%), human faces (38.46%), and clothing (36.06%), with the potential for overlapping categories. Moreover, women (31.25%) were detected in over three times as many videos as men (9.62%). Other less frequently detected objects included accessories (17.31%), furniture (8.65%), and vehicles (5.77%) (see Fig. S1). The text detected in videos covered topics across vaping and pregnancy (see Fig. S2). The most prevalent term, “pregnant,” appears in approximately 26.44% of the videos. Terms directly related to smoking, such as “smoking” (19.23%), “smoke” (11.54%), “vaping” (14.42%), and “vape” (13.94%) were featured frequently. “Pregnancy” is mentioned in 16.35% of the videos, “baby” is present in 13.94%, and “mom” is found in 7.21% of the videos. Action-oriented terms such as “quit” (11.54%), “like” (11.35%), “get” (8.65%), and “stop” (7.21%) were also present.

Human verification results

To further validate the accuracy of ChatGPT’s classifications, a human coder cross-checked the 1,040 posts that ChatGPT marked as relevant to both pregnancy and vaping. Upon evaluation, the coder verified n = 861 posts (82.79% of the 1,040 posts identified by ChatGPT) as containing pregnancy content, n = 555 posts (53.37%) as containing e-cigarette content, and n = 472 posts (45.38%) as discussing e-cigarette use during pregnancy.

A human coder also reviewed 10% of the 10,633 posts that ChatGPT determined to not be relevant to both pregnancy and vaping. Out of the 1,063 human-reviewed posts, n = 10 (0.94%) were determined to actually include pregnancy and vaping content (i.e., were misclassified by ChatGPT as irrelevant), resulting in 99% agreement between ChatGPT and the human coder for screened-out posts.

Limitations and future directions

Our work has several limitations. We converted all image and audio data into text using machine learning techniques because, at the time, ChatGPT had limited capacity in image processing. As LLMs are in their nascent stage of image processing, future studies may not need to preprocess images and videos using computer vision models. Nonetheless, sensitivity analyses are still needed comparing the accuracy of ChatGPT’s determinations based on visual (i.e., videos and images) vs. text-based (i.e., transcripts) inputs.

Additionally, even if ChatGPT and other generative AI tools improve their abilities to assess and analyze images and videos, there could be issues of privacy in inputting sensitive information such as faces. Therefore, computer vision techniques that can convert image and video-based data into text may still have utility. Our team did not enter image data into ChatGPT and only used text from publicly available TikTok posts. However, all use of generative AI models and platforms such as ChatGPT for research purposes should be done carefully, with effort to minimize the use of private and identifiable information, as data inputted into ChatGPT can be used to train its model. For future works, analyzing the objects and text commonly found in relevant posts, contrasted with those in irrelevant posts, could help discover patterns or features that distinguish meaningful content from noise. Comparisons in objects between relevant and irrelevant content can also identify common and unique objects present in posts discussing specific topics.

Other LLMs (e.g., Microsoft’s large language model—Copilot) could provide additional insights into how pregnancy and vaping content can be classified. Sensitivity analyses should assess which tools provide the most accurate determinations, given that all LLM service algorithms are proprietary. Other models can be used for cross-validation in future work (e.g., Claude, Gemini, LLaMA), as well as different versions of ChatGPT (e.g., GPT-4 vs. GPT-4o). Our overarching protocol comparing ChatGPT-4 to human determinations could serve as a benchmark for the ethics and execution of combining human oversight with LLMs as these algorithms become more sophisticated. Similar procedures can also be used to: 1) examine the sensitivity of generative AI platforms when screening social media content compared to other commonly used methods, such as keyword filtering, to identify when simpler or more complex analytic approaches are needed; and 2) compare generative AI’s accuracy when screening social media content depending on what information is entered into the model (e.g., audio transcripts, post descriptions, and objects detected) to better understand the benefits and cost to each data element in our multimodal system. Future research can also include TikTok user comments, instead of solely relying on post content and descriptions.

Moreover, future studies should compare results and content across other social media platforms to understand the generalizability of our findings, as other social media services have different demographics (e.g., Facebook with older users, YouTube with a wider spread) that can offer different perspectives on topics such as e-cigarette use during pregnancy (Zote, 2024). Additional research using ChatGPT to examine social media data should continue to examine ChatGPT’s potential limitations, ways to optimize prompts to increase ChatGPT’s accuracy, and when human verification and coding is most essential.