Design of a hazard prediction system with intelligent multimodal fusion based on artificial intelligence & internet of things technology: taking a crib as an example

Design of real-time monitoring module

This research has employed the ESP32-CAM development board to achieve real-time monitoring of infants’ activity status. The board also transmits video stream information to parents’ mobile phones. The ESP32-CAM development board incorporates WiFi and Bluetooth modules, which makes it easy to connect to the internet. The board’s camera can provide panoramic real-time monitoring of the baby’s condition and transmit the monitoring footage to parents’ mobile phones. The key components and functions of the ESP32-CAM development board module circuit design include:

Firstly, a 5V DC power supply is used, and the voltage is stabilized at 3.3V through an AMS117-3.3V voltage regulator chip. This provides the required power for ESP32-CAM and the camera.

Secondly, the ESP32-CAM module is the core aspect of the whole development board. It includes the ESP32 chip (Gaikwad, 2023), peripheral circuits, and WiFi module. The ESP32 chip supports WiFi and Bluetooth communication and offers multiple GPIO interfaces that can be connected to other peripherals.

Additionally, it is equipped with a OV2640 camera module (Gaikwad, 2023), which supports static image shooting and video recording with a maximum resolution of 1,600 × 1,200 pixels. This camera can efficiently and accurately monitor the baby’s activity status and transmit real-time images to the mobile phone. For the convenience of developer communication and debugging, a USB to serial port chip is also configured, allowing the developer to interact with the ESP32-CAM module through the USB interface.

At the same time, it supports functions such as microSD card expansion (Gaikwad, 2023) and GPIO expansion and can be expanded and connected as needed. Through the baby activity monitoring system designed in this way, parents can keep up-to-date with the baby’s situation through a stable internet connection and provide timely attention and care. The design of the real-time monitoring module is shown in Fig. 1A.

Design of turn-over detection module

Turn-over Detection Module design consists of multiple components including the AONI high-definition camera, interfaces, and the PC terminal. These components collectively form the Turn-over Detection Module of the system, which is capable of precise facial detection of infants by leveraging the AONI camera’s interface connected to the PC terminal. The design of the turn-over detection module is shown in Fig. 1B.

The PC terminal serves as the central processing unit of the system, orchestrating the various stages of the turn-over detection workflow. Equipped with robust computational capabilities, the terminal operates in tandem with the AONI camera, receiving real-time image data and invoking the turn-over detection algorithm. Leveraging sophisticated image processing techniques, the algorithm employs facial detection algorithms to accurately detection and track the infant’s facial features.

The integration of this module with the Turn-over Detection Algorithm enables accurate facial detection of infants. By examining the presence or absence of detected facial images, it becomes possible to determine if the infant has turned over. If no facial image is detected within a 5-s interval, it indicates that the infant has turned over and their mouth and nose might be obstructed. In response, the system automatically triggers an alert through a buzzer, prompting parents to take immediate action.

The synergy between the AONI camera, the PC terminal, and the turn-over detection algorithm ensures a robust and reliable Turn-over Detection Module for infant safety. This comprehensive approach not only guarantees real-time monitoring of infants but also provides timely alerts to parents, enabling them to promptly address potential risks associated with the infant’s position during sleep or rest.

Future research endeavors may focus on further enhancing the performance of the Turn-over Detection Module. This could include exploring novel image processing techniques, refining deep learning models, and integrating additional sensor modalities for a more comprehensive understanding of the infant’s behavior and environment.

Design of environmental detection module

The design of the environmental detection module is shown in Fig. 1C. It consists of various components, including the main control chip Arduino UNO development board module, power module, LCD1602A display module, temperature and humidity detection module, sound detection module, ultrasonic detection module, buzzer warning module, ESP32-CAM development board module, OV2640 camera module, AONI high-definition camera module, and sliding resistor. The functions of these modules in the entire hazard prediction system are listed in above Table 1.

The detailed explanation of the Table 1 is as follows: The main function of the Arduino UNO development board module is to control the power module, LCD1602A display module, temperature and humidity detection module, sound detection module, ultrasonic detection module, and buzzer alarm module to work together, responsible for collecting temperature, humidity, decibel value, distance value, and controlling the opening or closing of the buzzer. The power module can provide a stable power supply for the entire system and can convert 9V power into 5V or 3.3V for output power supply.

The DHT11 temperature and humidity detection module, a sound detection module, and an ultrasonic detection module can monitor the relevant information of infants in real-time, such as temperature, humidity, decibel value, and distance value, while the LCD1602A display module can dynamically display and refresh these four data in real-time. The buzzer warning module can issue an alarm when needed to remind parents or guardians. Finally, the environment detection module was simulated and designed using the simulation software Fritzing, as shown in Fig. 2.

This simulation design diagram provides a detailed explanation of the composition structure of the environmental detection module, the wiring relationships of various sensors, and the corresponding pin ports, which is conducive to landing the physical object and conducting scene simulation experiments.

Design of real-time state monitoring algorithm

Firstly, initialize and configure the camera. If the camera model is “ESP32-CAM”, set pins 13 and 14 to input pull-up. Then, initialize the camera and check if the initialization is successful. If initialization fails, please print an error message and end the algorithm. Next, initialize the sensor and make some settings based on the sensor model. Then, check if the LED pins are defined. If defined, please set LED flashing. Then, disable sleep mode and try to connect to WiFi until the connection is successful. Next, start the camera server to transmit the video stream in real-time.

In the main loop, the algorithm will loop infinitely, with each loop delayed by 10 s. This can continuously monitor real-time status and process or display the required video stream data. This algorithm implements a complete real-time status monitoring system that can initialize cameras, sensors, and LEDs, connect to WiFi, and start the camera server to provide real-time video streams and local IP addresses. Finally, print the HTTP address so that other devices can connect to the camera through it. Finally, opening the IP address through the browser (Gaikwad, 2023) allows for real-time viewing of the video screen. The Real-time State Monitoring Algorithm will be presented and analyzed in the form of pseudocode is shown in Algorithm 1.

Important reminder: Before using the ESP32-CAM development board, please make sure to open the WiFi hotspot in advance, otherwise an error message of “WiFi connection failure” may appear. Next, burn the program onto the ESP32-CAM development board and press the RST button on the board to restart it. At this point, the core program will automatically run and connect to a WiFi hotspot LAN named “XXX” with a password of “XXX”. Next, select the serial port monitor under the Arduino IDE compiler interface tool to view the IP address output from the serial port.

In addition, the above program is only applicable to local area networks. If you want to remotely monitor real-time videos within a wide area network, you can try the following approach: design two subroutines, one for running on the ESP32-CAM development board and the other for running on the server. Send video data to the server through the ESP32-CAM development board, and the server runs the receiving program to receive and display it.

The advantage of real-time video stream information being sent to external public network servers is that distant parents can receive remote real-time video monitoring on their mobile phones.

However, it should be noted that even if the video of the impending danger is transmitted back to parents, they will not be able to rush home in time to prevent the tragedy from happening. Therefore, this study only focuses on the scenario where parents are at home (within a local area network), aiming to provide the effect of short-term care.

Design of turn-over detection algorithm

The implementation logic of the Turn over Detection Algorithm is to first create a window to display real-time images captured by the camera. Then load a human face classifier model, which is used to detect faces in the image. In each frame of the image, the program converts the image into a grayscale image and uses a classifier model to detect faces.

If a face is detected, the program will record the current time and add the data to the list of duration and status. The detected face will be marked with a green box on the image. If no face is detected, the program will also record the current time and add the data to the list of duration and status, while setting the status to “Turn over” to indicate flipping. If no face is detected for more than 5 s and the buzzer has not been triggered, trigger the buzzer and set its triggering status to triggered.

After the loop ends, the program will write the list data of duration and status into an Excel spreadsheet. It uses the openpyxl library to create a new workbook and creates column headings on the active table.

Then, it traverses the data in the list and writes the data into the corresponding cells of the table. At the same time, based on the value of the status, set the “Buzzer” column to “Ring” or “Non”. Finally, the program saves the Excel spreadsheet and prints “Finished.” to indicate that the program has completed execution.

This program implements functions such as capturing real-time images from the camera, facial detection, data recording, and writing Excel spreadsheets. It can be used to track the duration of facial appearance, record facial status, and trigger a buzzer to remind you to turn over. Finally, the data written into an Excel spreadsheet will be used for subsequent experiments and analysis on the accuracy of the Turn over Detection function. The pseudocode logic diagram of Turn-over Detection Algorithm is shown in Algorithm 2.

This study uses Turn over Detection Algorithm to achieve bounding box detection of infants’ faces and determine whether the infant has turned over. In the hazard prediction system, this system chooses to call the face detection algorithm in the Open CV library for face detection. There have been many studies on face detection. Open CV is an open-source library for visualization algorithms and image processing developed by Intel. It has become quite mature and has been widely applied in face detection and recognition (Lin et al., 2021). This article selects the Haar cascade classifier. The file of the classifier contains Haar-like features that describe different parts of the human body. It achieves the classification of faces and nonfaces through integrated images, AdaBoost algorithm, and cascaded classifiers (Lin et al., 2021).

The main process of the Haar classifier is to use sliding windows and integral images to achieve fast traversal of grayscale images and calculation of Haar-like features. Next, the AdaBoost algorithm based on Haar-like features is used to train a weak facial classifier and construct a strong classifier. Then, multiple strong classifiers are combined to enhance the classification performance (Lin et al., 2021). Therefore, facial classification can be achieved, and the workflow of the facial classifier is shown in Fig. 3A.

Object detection using Haar cascaded classifier features is a useful method for object detection. This is a machine learning-based method, where the cascade function is trained from multiple positive and negative images. Each section is a single value obtained by subtracting the number of pixels below the white box from the number of pixels below the black box.

The program will automatically search to determine Haar features based on the lines and edges of dark and bright images (Diyasa et al., 2021). There are several types of Haar features, namely edge features, line features, and center features. After determining the Haar features in a specific image region, the image integration will be calculated within that pixel region (Diyasa et al., 2021). The basic calculation process for the image of the Haar cascade classifier is shown in the following Eq. (1) (Diyasa et al., 2021):

(1) $$s(x,y)=i(x,y)+s(x,y)+s(x-1,y)-s(x-1,y-1)$$among them, s(x, y) represents the sum of each pixel value, i(x, y) represents the pixel value intensity value from the input image, s(x ‒ 1, y) represents the pixel value on the x-axis, s(x, y ‒ 1) represents the pixel value on the y-axis, and s(x ‒ 1, y ‒ 1) represents the pixel value on the diagonal (Diyasa et al., 2021). The facial detection effect after running the baby turn-over detection algorithm is shown in Fig. 3B.

An image, contains many Haar features, so a large amount of image integration calculations will also be performed. Afterward, the images will be processed through a cascaded classifier. Cascade classifiers process many features through hierarchical classification (Diyasa et al., 2021).

During the face detection process, the system will automatically search for facial information. If there is a face in the field of vision, the system can detect it within 0.04 s. By using multi-scale algorithms to search facial images at low resolution, the system switches to high-resolution search only when a face-like shape is detected.

In addition, future research may consider replacing facial detection algorithms with FaceIt (Brey, 2004) recognition systems, which use local feature analysis (LFA) algorithms (Kim et al., 2020) to encode facial features and generate facial patterns as unique numerical codes. The system can compare the detected facial patterns with the facial pattern data stored in the database to achieve accurate facial recognition.

Design of environmental detection algorithm

The environment detection algorithm designed in this article utilizes multiple sensors to work together in multimodal mode and monitor the temperature, humidity, sound intensity, and distance of the environment in real-time. Firstly, the values read by each sensor are printed out on the serial port and converted into strings for real-time display on the LCD screen. Through LCD and serial port output, users can conveniently observe the temperature and humidity, sound intensity, and distance information of the environment.

In addition, the algorithm also provides feedback and control to the system based on certain judgment conditions. When the sound intensity exceeds 50, the temperature exceeds 38 degrees Celsius or is below 36 degrees Celsius, the humidity exceeds 90%, or the distance is less than 5 centimeters, the system will control the buzzer to sound, indicating that danger is about to occur, and output a “Ring” on the serial port. On the contrary, it indicates that the environment is normal and “Non” is output on the serial port. These valid data will be imported from the serial port into an Excel spreadsheet for verification and analysis of subsequent experimental results. The pseudocode logic diagram of Environment Detection Algorithm is shown in Algorithm 3.

Test the function of monitoring the real-time status of infant

During this test, aimed to assess its capability to monitor the real-time status of infants and transmit the monitoring image to parents’ mobile phones via the Internet.

To simulate different activity states of the baby, the Real-time Monitoring Module was utilized to capture the real-time panoramic view of the infant. After running the Real-time Monitoring Algorithm, the website address will be output. Open the IP address (Gaikwad, 2023) through a browser to view the baby’s real-time activity scene. The test results indicated that all the baby’s indicators remained within the normal range, and the video transmission was successful. Consequently, the functional test for simulating and monitoring the real-time status of infants is deemed successful, as depicted in Fig. 1D.

Table 2 shows real-time recorded values of various indicators for a small subset of randomly selected infants. The distance value in the third and fourth rows of the table above is 0, which is an abnormal situation. This situation may be due to the ultrasonic sensor being obstructed, the distance value exceeding the range, or the refresh delay causing the value to be 0 during the operation process. Among the 1,107 actual measured data, only two appeared, with an anomaly rate of 0.1807%, which has a small impact and can be ignored.

Test the detection function of the infant kicking open the quilt

In this test, the system will be able to detect whether the baby has kicked open the quilt and issue a warning signal through a buzzer. The most common definition of hypothermia is body temperature ≤36.0 °C (Ramgopal et al., 2023). During the testing process, the temperature difference between the inside and outside of the quilt will be used to distinguish.

When the baby kicks off the quilt, the temperature sensor is exposed and the temperature begins to decrease. When the temperature drops below 36.0 °C (<36.0 °C), the buzzer will sound to inform parents to cover the quilt. The test results indicate that the system can issue timely warning signals when a baby kicks off the quilt. The function test of simulating a baby kicking off the quilt was successful, as shown in Fig. 1E.

Under the premise of ensuring that other factors fluctuate within a small range (not exceeding the set threshold), 1,126 pieces of hyperthermia data were collected for preprocessing, and the calculation is based on the following Eqs. (2) and (3):

(2) $$T=(U/Vs)+b,$$among them, T is the temperature (°C), U is the sensor output voltage (V), V_s is the sensor sensitivity (constant), and b is the sensor zero offset (constant).

(3) $$KT=\mathrm{\Delta }T/\mathrm{\Delta }t=(T2-T1)/(t2-t1),$$among them, KT is the rate of temperature change, usually represented by the symbol △T/△t, in units of (°C/s), where △T is the amount of change in object temperature in degrees Celsius (°C), △t is the amount of change in time, which can be in seconds (s).

By performing formula calculations on 1,126 pieces of hypothermia data, three sets of valid information can be obtained: Temperature (°C), KT (°C/s), and Run Time (s).

Next, input these three sets of data into A(X), B(Y), and C(Z) in the book table of Origin software, convert the worksheet into an XYZ grid matrix, and then use the 3D Smoother tool in Apps to smooth the data. The Smoother method selects the Adjacent Average, outputs the 3D image, and then projects the image to obtain the bottom image. Add the side semi-transparent fill color, adjust the scale line, input the meaning of the coordinate axis, name the image (upper right corner), and export it to PNG format. Finally, the 3D analysis graph is obtained, as shown in the following Fig. 4A.

From the above three-dimensional smooth image, it is not difficult to see that when the baby’s body temperature is too low, the Temperature (°C) changes between 28 and 35 with the passage of Run Time (s), and the temperature change rate KT (°C/s) only changes between −0.75 and 0.75. This can effectively prove that the hazard prediction system can timely issue warning signals when monitoring the baby kicking off the quilt.

Test the detection function of the infant fever

The purpose of this test is to check if the system can monitor infant fever. The temperature value can be displayed in real-time through the LCD1602A screen. According to the widely used definition of fever (temperature ≥38.0 °C) (Kasbekar et al., 2021), the system will trigger a buzzer alarm when the temperature inside the quilt rises to above 38.0 °C, reminding parents that the baby may have a fever and need timely treatment. During the testing process, to maintain a constant humidity, only increasing the temperature and simulating a baby’s fever can be done using the hot air mode of a hair dryer or a cup with a lid filled with hot water for simulation. The test results show that the system can accurately monitor the temperature of infants and give timely alarms. The simulation test of infant fever function is considered successful, as shown in Fig. 1F. In this experiment, 1,283 hyperthermia data were collected for preprocessing, using the same formula as Eqs. (2) and (3).

By performing formula calculations on 1,283 hyperthermia data points, three sets of valid information can be obtained: Temperature (°C), KT (°C/s), and Run Time (s). Next, using the method described in the first test, a three-dimensional analysis diagram was finally obtained, as shown in Fig. 4B. From the above three-dimensional smooth image, it is not difficult to see that when the baby’s body temperature is too high, with the passage of Run Time (s), the Temperature (°C) changes between 38 and 44, and the Temperature Change Rate KT (°C/s) only changes between −0.75 and 0.76. This can effectively prove that when the hazard prediction system detects a baby’s fever, the temperature change is more significant, and it can issue a timely warning signal.

Test the detection function of the infant bedwetting

In this test, the system will be checked to whether it can detect a baby wetting the bed and issue a safety warning signal through a buzzer. During the testing process, the towel will be moistened with warm water at around 37 °C, and the temperature will be controlled within the normal range of 36 °C to 38 °C. Only the humidity will be changed to simulate the baby wetting the bed. When using a wet tissue to wrap the humidity sensor, it is evident that the humidity value on the LCD1602A display screen rapidly soars to over 90%. The test results show that when the system detects a baby wetting the bed, the reading of the humidity sensor will exceed the threshold of 90%, and the buzzer will sound to notify parents to replace the diaper promptly. The function test of simulating the baby bedwetting was successful, as shown in Fig. 1G.

In this experiment, 1,162 bedwetting data were collected for preprocessing, and the formulas used for calculation are as follows Eqs. (4) and (5):

(4) $$H=(U-C3)/(C1+C2\ast R),$$among them, H is the relative humidity (RH); U is the output voltage of the sensor (V); C₁ to C₃ are constants; R is the output resistance of the sensor (Ω).

(5) $$KH=\mathrm{\Delta }H/\mathrm{\Delta }t=(H2-H1)/(t2-t1),$$among them, KH is the rate of change in humidity, usually represented by the symbol $\mathrm{\Delta }$H/ $\mathrm{\Delta }$t, in units of (RH/s), where $\mathrm{\Delta }$H is the amount of change in relative humidity, in units of relative humidity (RH), and $\mathrm{\Delta }$t is the amount of change over time, which can be expressed in seconds (s). By performing formula calculations on 1,162 bedwetting data, three sets of valid information can be obtained: Humidity (RH), KH (RH/s), and Run Time (s). Next, using the method described in the first test, a three-dimensional analysis diagram was finally obtained, as shown in Fig. 4C.

From the three-dimensional smooth image, it is not difficult to see that as the Run Time (s) increases, the humidity (RH) changes between 20% and 95%, while the humidity change rate KH (RH/s) only changes between −2 and 6. This effectively proves that the hazard prediction system monitors the rapid changes in humidity when the baby wets the bed and can issue timely warning signals.

Test the detection function of the infant crying

In this test, the system will be checked to see if it can detect a baby crying and issue a safety warning signal through a buzzer. During the testing process, an audio clip of a baby crying will be played on the phone to simulate a baby crying. When a baby wakes up and starts crying, the surrounding environment changes from quiet to noisy. The sound sensor receives the sound signal of a baby crying. When the reading exceeds the threshold of 50 dB, the buzzer will sound to notify parents to comfort their children. The test results show that the system can accurately monitor the crying situation of infants, with a monitoring accuracy rate of over 95%. The functional test of simulating baby crying was successful, as shown in Fig. 1H.

In this experiment, 1,303 crying data were collected for preprocessing, and the formulas used for calculation are shown in Eqs. (6) and (7):

(6) $$V=20\ast log(U/U0),$$among them, V is the sound intensity (dB), U is the sensor output voltage (V), and U₀ is the reference voltage (V).

(7) $$KV=\mathrm{\Delta }V/\mathrm{\Delta }t=(V2-V1)/(t2-t1),$$among them, KV is the rate of change in sound intensity, usually represented by the symbol $\mathrm{\Delta }$V/ $\mathrm{\Delta }$t, expressed in dB/s, where $\mathrm{\Delta }$V is the amount of change in sound intensity, expressed in decibels (dB), and △t is the amount of change over time, which can be expressed in seconds (s). By performing formula calculations on 1,303 crying data, three sets of effective information can be obtained: Voice (dB), KV (dB/s), and Run Time (s). Next, using the method described in the first test, a three-dimensional analysis diagram was finally obtained, as shown in Fig. 4D.

From the three-dimensional smooth image, it is not difficult to see that as the Run Time (s) increases, the Voice (dB) changes between 0 and 70, and the sound intensity change rate KT (°C/s) only changes between −40 and 45. This effectively proves that the hazard prediction system can detect significant changes in sound intensity when a baby is crying, and can issue timely warning signals.

Test the detection function of preventing the infant from climbing over crib

In this test, it will be checked whether the system can prevent babies from climbing over the crib. When there is no object around the crib, the distance measured by the ultrasonic sensor is a constant value. When the baby climbs near the edge of the crib, the distance displayed by the ultrasonic sensor on the monitor quickly shortens. When the distance value is less than 5 cm, it is determined that the baby is at the edge of the bed, and the buzzer will sound, notifying parents to handle the dangerous situation promptly. The test results show that the intelligent baby crib can effectively prevent babies from climbing over the crib, and the protective effect is very good. During the experiment, the function test to simulate preventing infants from climbing over the crib was successful, as shown in Fig. 1I. In this experiment, 1,255 climbing over data were collected for preprocessing. The calculation was based on the Eqs. (8) and (9):

(8) $$D=K/(U-C),$$among them, D is the distance (cm), U is the sensor output voltage (V), and K and C are the constants of the sensor.

(9) $$KD=\mathrm{\Delta }D/\mathrm{\Delta }t=(D2-D1)/(t2-t1),$$among them, KD is the rate of change in distance, usually represented by the symbol $\mathrm{\Delta }$D/ $\mathrm{\Delta }$t, in units of (cm/s), where $\mathrm{\Delta }$D is the amount of change in distance in centimeters and $\mathrm{\Delta }$t is the amount of change in time, which can be in seconds. By performing formula calculations on 1,255 climbing over data, three sets of effective information can be obtained: Distance (cm), KD (cm/s), and Run Time (s). Next, using the method described in the first test, a three-dimensional analysis diagram was finally obtained, as shown in Fig. 4E. From the 3D smooth image above, it is not difficult to see that as the baby approaches the edge of the bed, the distance (cm) changes between 0 and 175 with the passage of Run Time (s) and the Distance Change Rate KD (cm/s) changes between −80 and 85. This effectively proves that the hazard prediction system can issue timely warning signals when the baby approaches the edge of the bed and the distance value is less than 5 cm.