Multimodal dataset for sensor fusion in fall detection

Ethical approval and study procedures

This study was conducted with the approval of the Ethics Committee of Universidad Andrés Bello, under Approval Act number 032/2023. The committee reviewed and approved all proposed ethical and methodological aspects, ensuring the protection and respect for the dignity and rights of the participants. The research focused on data collection for the “QUIDA dataset”, aimed at enhancing FDSs, specifically designed for the elderly population.

Participants were fully informed about the study objectives, involved procedures, potential benefits, and associated risks of their participation. All activities were conducted in a controlled and safe environment, using gymnastics mats and other safety measures to minimize any risk of injury. Participation was voluntary, and all participants provided written informed consent before participating in the study.

The confidentiality and anonymity of the collected data were paramount. In line with ethical practices recommended by the committee, no digital records of sensitive participant data were made. Only necessary data for the study objectives were encoded and stored, and only authorized personnel had access to this information. This approach ensures the integrity and privacy of the information, complying with the ethical standards required.

Data capture

The data was captured using four different sensors in a room with gymnastics mats placed in the middle and a desktop to the side. The center of the mats is the designated fall area and people move along the length of the mats for the experiments.

The first sensor is the accelerometer of a Samsung Galaxy A14 smartphone, securely placed on the participant’s torso between the T5 and T10 vertebral levels using an adjustable harness. This anatomical location was chosen due to its proximity to the body’s center of mass, providing a more representative measure of trunk balance and control during movement. The smartphone integrates an STMicroelectronics LSM6DSL 6-DoF inertial measurement unit (IMU), whose accelerometer has a maximum acceleration range of ±16 g with a resolution of 0.488 mg. Data was recorded at a sampling frequency of 62.5 Hz (16 ms period), which is well-suited for capturing falls, as their primary frequency components typically lie within the 2–3.5 Hz range (Huynh & Tran, 2021). This sampling rate also exceeds the recommended minimum of 40 Hz for accelerometers used in physical activity monitoring (Meng & Kim, 2011).

Mounted on the ceiling approximately 2.7 m above the ground and directly above the mats, an 8 × 8 STMicroelectronics VL53L5CX LIDAR was installed. It was programmed to operate at 10 Hz (100 ms period), with an approximate field of view of 2 × 2 meters centered on the fall area. About 20 cm from the LIDAR sensor, a Texas Instruments IWR6843AOP 60–64 GHz radar was installed. It operated at 8.3 Hz (120 ms period) and had an approximate field of view of 4 × 4 meters. Given its broad area of vision, the radar was off center from the fall area. Finally, a Melexis MLX90640 32 × 24 far infrared (FIR) thermal camera was strategically mounted on the desktop 1.5 m from and centered on the fall area, at 1 meter above ground and programmed to operate at 2.6 Hz (384 ms period). The FIR camera interfaced with a Raspberry Pi 4 via I²C, the LIDAR sensor was attached to an ESP32 microcontroller via I²C, which then connected to a computer through USB, and the radar was directly connect to a computer through USB, as shown in Fig. 1.

Figure 2 illustrates the block diagram of the multisensor system, depicting the arrangement of the various components including sensors, a Raspberry Pi, and other processing units. The diagram outlines the data flow and connectivity between the devices, such as the harness with a mobile phone, LIDAR, radar, and a far infrared thermal camera, all integrated into a room designed for fall detection experiments.

Throughout the experiments, continuous data recording of all sensors at the same time took place, with the exception of occasional interruptions caused by cable unplugging incidents during the data collection process. The mobile phone’s accelerometer was configured to record continuous blocks of 12 s for each fall and during walking period of the protocol. All sensors, with the accelerometer being the sole exception, employed custom software for the capture and recording of sensor data. Figure 3 shows a sample of the data captured by these sensors.

It should be noted that the sensors used in this study (LIDAR, thermal cameras, accelerometers and radar) intrinsically favor privacy, since the information they capture does not allow personal identification. In addition, the data reported are presented anonymously, protecting the identity of the participants.

Dataset generation

In order to create the dataset, each participant was required to perform a series of tasks. The study involved a phase of walking without falling and simulating 10 different types of falls. The activities were sequenced as follows:

1. Walking without falling

2. Falling backward while walking backward

3. Falling forward caused by tripping

4. Falling caused by fainting

5. Falling backward while attempting to sit down

6. Falling backward with straight legs

7. Falling forward with straight legs

8. Falling forward with knee flexion

9. Falling backward with knee flexion

10. Lateral falling with straight legs

11. Lateral falling with knee flexion.

Except for the mobile phone accelerometer, custom Python programs were used to capture data from the various sensors. All the captured data included a timestamp which was then used to synchronize the time series of all the sensors.

Dataset description

The dataset supporting this study is publicly accessible on the Open Science Framework (OSF) platform at https://doi.org/10.17605/OSF.IO/YJGDV. It is organized with individual directories for each subjects, containing four CSV files corresponding to the different sensors. The mean and standard deviation for the age, weight, height, and Body Mass Index (BMI) of the participants are presented in Table 2. In each sensor’s CSV file, the first column presents the timestamp in Unix time, with subsequent columns providing sensor-specific data.

Columns 2, 3, and 4 of the accelerometer CSV contain acceleration along the X-axis (left–right), Y-axis (craniocaudal), and Z-axis (anteroposterior), respectively, measured in meters per second squared (m/s²).

The FIR camera’s CSV file contains data in columns 2 to 769, representing the temperature in degrees Celsius (^∘C) for each pixel in a 32 × 24 matrix.

In the LIDAR CSV, columns 2 to 65 detail the ambient light received by the Single-photon avalanche diode (SPAD) array, quantified in kilo-counts per second per SPAD (kcps/SPAD). Columns 66 through 129 in the dataset represent the count of targets identified for each matrix element. In these columns, a value of 1 denotes a successful distance measurement at the respective matrix element, whereas a value of 0 indicates an inability to measure the distance. Columns 130 to 193 show the number of SPADs active for the current measurement. Columns 194 to 257 represent the photon count during the vertical-cavity surface-emitting laser (VCSEL) pulse, also in kcps/SPAD. The standard deviation of the measured distance in meters is specified in columns 258 to 321. Columns 322 to 385 provide the distance measurements in meters for the matrix elements, and columns 386 to 449 describe the status of each matrix element, with more details available in the UM2884 document by STMicroelectronics. Finally, columns 514 to 577 provide the reflectance percentage of the object.

The radar CSV file uses a variable number of columns on each row depending on the number of points detected. Specifically, the number of columns equals 1 + 7n for n detected points. Every block of seven consecutive columns represents a single point and their rows indicate the (x,y,z) position (m), Doppler (m/s), SNR (dB), noise (dB) and tracking ID.

The file named ‘Falls.csv’, located in the root directory, records the timings of falls for the different participants. The data is organized such that each column corresponds to a different subject, arranged sequentially. The first column pertains to the first subject, the second column to the second subject, and this pattern continues across columns. Additionally, each row in the file represents a separate instance of a fall.

Dataset segmentation

To separate fall and no-fall events within the dataset, the following procedure was used to segment each recording into discrete frames. This segmentation was based on manual annotations marking the exact instances of falls captured in the dataset. Following each annotated fall, a predefined interval determined by user-defined parameters, e.g., 2 s, defines a window both 2 s before and 2 s after the annotated fall event. This window size is chosen to encompass the entire fall event, ensuring that all relevant data points within the segment are captured.

For cases not classified as falls, segments of equal width were defined using measurements outside the segments labeled as falls, maintaining a balance between fall and no-fall samples at a ratio of 1:N, in this case N = 3. This segmentation strategy not only guarantees the representation of both event types within the dataset, but also allows the synchronization of events across different sensor data streams, despite their different sampling rates.

The synchronization of the four sensors was ensured by performing a specific action that generated a clear, simultaneous peak across all signals, achieved through a sudden movement accompanied by arm flapping. The RGB camera was also included in this synchronization process. The exact moment of the fall was determined by analyzing the visual recording, allowing for precise alignment of the sensor data with the observed fall event. This approach provided a consistent reference point for data analysis across all modalities.

Thus, each segment, whether representing a fall or a no-fall event, is synchronized to represent the same event across all sensor modalities, albeit with a different number of data points due to the different sampling frequencies of each sensor. This approach ensures a comprehensive and consistent representation of each event, facilitating subsequent analysis and classification based on these segmented data sets. To visualize the segmentation process described, refer to the schematic illustration provided in Fig. 4.

Feature extraction

For characterizing the dataset, our aim is to distinguish between fall and non-fall events using simple yet effective features. This approach helps to emphasize the fundamental differences between these event categories, facilitating their future identification and analysis.

To characterize the captured sensor data from matrices such as those from LIDAR, radar, and thermal cameras, the Frobenius norm (or L2 norm) is employed in two distinct ways: instantaneously and as a difference between consecutive frames. The Frobenius norm, a measure of a matrix’s magnitude, is calculated by squaring each element of the matrix, summing all these squared values across both axes, and then taking the square root of this sum (Amir et al., 2024; Jefiza et al., 2017).

The mathematical formulations for instantaneous metric is as follows: (1)${\displaystyle \text{Instantaneous Matrix Norm}=\sqrt{\sum _{i=1}^m\sum _{j=1}^n|a_{ij,t}{|}^2}}$

where A denote the signal matrix A_t at time t, where x and y represent the rows and columns of the matrix, respectively. This norm reflects the instantaneous power or intensity of the sensor data.

The mathematical formulations for difference between the norms of consecutive frames: (2)${\displaystyle \text{Matrix Norm Difference}=\sqrt{\sum _{i=1}^m\sum _{j=1}^n|a_{ij,t}-a_{ij,t-1}{|}^2}}$

For the three signals from the accelerometer, the Euclidean norm (or L2 norm) is utilized to analyze the signals’ aggregate intensity at any given instant as well as the intensity changes between consecutive readings.

The mathematical formulation for the instantaneous metric is expressed as follows: (3)${\displaystyle \text{Vector Norm}=\sqrt{x_t^2+y_t^2+z_t^2}.}$

This equation calculates the Euclidean norm for the accelerometer data at time t, where x_t, y_t, and z_t denote the acceleration values along the X, Y, and Z axes, respectively. This measure reflects the overall intensity or “power” of the motion captured by the accelerometer at that specific moment.

To capture the dynamic nature of the data by analyzing changes between consecutive readings, the difference in norms between two successive frames is computed as: (4)${\displaystyle \text{Vector Difference Norm}=\sqrt{{\left(x_t-x_{t-1}\right)}^2+{\left(y_t-y_{t-1}\right)}^2+{\left(z_t-z_{t-1}\right)}^2}.}$

This formula determines the change in intensity of the accelerometer’s signals from one time frame to the next, illuminating the rate of change in motion. This comparison between the instantaneous norm and the difference norm aids in understanding both the magnitude and the variation of movement, facilitating the differentiation between fall and no-fall events.