Attribute-based people search: open attribute recognition for person re-identification in surveillance and security

Text-based person re-identification

Although using images for person ReID is effective for certain surveillance and security applications, it is not always practical in real-world scenarios. For instance, when an initial image of the target person is not easily available, relying only on image-based ReID is insufficient. This frequently occurs due to problems including low image quality, subject occlusion, or limitations in surveillance systems (Yadav & Vishwakarma, 2024). On the other hand, text-based ReID provides a useful alternative by utilizing natural language descriptions provided by witnesses or extracted from case files (Li et al., 2024a; Yang & Zhang, 2024). This approach proves particularly useful in cases involving missing persons or criminal investigations, where verbal descriptions might be the only available lead (Huang, Qi & Chen, 2024; Wu et al., 2023a).

Several person ReID and retrieval techniques handling text and images have been developed and are categorized depending on their methods for handling cross-modal data. Different methodologies in cross-modal and multi-modal ReID offer unique advantages and limitations, influencing their suitability for various scenarios, and their features are represented in Table 1. Some methods concentrate on global feature alignment, in which models are trained to embed text and image data into a shared space and calculate similarities using these global representations (Zhai et al., 2022; Zheng et al., 2024a; Zhao et al., 2024; Hu et al., 2025). For instance, early methods implement visual geometry group (VGG) and long short-term memory (LSTM) networks, while more recent methods employ ResNet and bidirectional encoder representations from transformers (BERT) backbones (Huang, Qi & Chen, 2024). Other methods incorporate attention mechanisms in their methodology to improve feature alignment (Lv, Sun & Nian, 2024; Wang et al., 2022b; Zhai et al., 2022). Local feature alignment, which aims to match specific parts of an image with corresponding phrases or words in the text, is examined by other techniques. These methods often use part-level or fine-grained features to achieve more refined alignments, occasionally incorporating body-part relationships. However, more computational power is needed to implement local matching (Wu et al., 2024c; Qin et al., 2024; Zhai et al., 2022). Some methods approach the modality gap between images and text by extracting modality-shared features and reducing inter-modal differences (Shao et al., 2024; Huang, Qi & Chen, 2024; Wu et al., 2023a, 2023b; Wang et al., 2022b). More recent approaches utilize the rich alignment knowledge of vision-language pre-training models such as CLIP for both local and global alignments (Li, Sun & Li, 2023; Cao et al., 2024). Finally, some methods address the challenge of noisy correspondence in image-text pairs by specifically modeling the uncertainty in visual and textual data using Gaussian distributions and multi-granularity uncertainty estimates (Qin et al., 2024; Zhao et al., 2024). These methodologies are often complementary rather than mutually exclusive and can be combined to provide a more comprehensive understanding of the cross-modal data, such as merging global and local feature alignment techniques (Wang et al., 2022b).

Person attribute recognition

Within the rapidly evolving fields of computer vision and machine learning, person attribute recognition (PAR) strives to identify specific semantic characteristics of individuals from visual data, such as images or video frames (Wang et al., 2022a). These characteristics, including gender, age, style of clothing, accessories, and even behavioral characteristics, provide useful contextual information for real-world applications such as autonomous systems, human-computer interaction, and security monitoring. Recent developments in deep learning, convolutional neural networks (CNNs) and attention mechanisms have significantly improved the accuracy and robustness of PAR systems (Liu et al., 2017; Bui et al., 2024; Wang et al., 2024b).

Different aspects of attribute modeling that are emphasized by PAR techniques can be categorized into several main approaches. Attribute localization is performed using part-based models that divide the human body into parts or attention-based mechanisms that focus on informative regions using multi-scale or channel-aware attention (Wang et al., 2022a; Weng et al., 2023). An attribute localization module may be employed to enhance attribute-related features (Li et al., 2022). Local part-based models extract the fine-grained details that global image-based models struggle to capture at the expense of increased complexity. Attention-based procedures balance accuracy and efficiency by dynamically focusing on informative regions in the image (Liu et al., 2017; Bui et al., 2024; Wang et al., 2024b). Attribute correlations between attributes or contextual correlations between different image regions can be investigated through relation-aware models that apply graph convolutional networks or relation mining, occasionally combined with an attribute correlation module (Zhao et al., 2022; Zuo et al., 2024; Yu et al., 2024; Wu et al., 2024b; Weng et al., 2023). Some approaches represent attributes as semantic properties and use attribute prompts or textual descriptions (Bui et al., 2024; Yang et al., 2023b; Wang et al., 2024b). Additional methods include attribute-aware style adaptation for semantic consistency, grouped knowledge distillation to minimize the disparity between visual and text embeddings, and adaptive feature scale selection to select relevant feature maps (Qu et al., 2023; Shen et al., 2024; Li et al., 2024b; Zhang et al., 2023). Transformer-based models, often combined with CNNs, are used to capture long-range dependencies and model global context (Zhang et al., 2023; Wang et al., 2024b; Bui et al., 2024; Bui, Le & Ngo, 2024; Cormier et al., 2024; Weng et al., 2023). Multi-scale feature aggregation and knowledge distillation can be used to further improve the model’s robustness (Zhang et al., 2023; Bui et al., 2024; Yu et al., 2024; Wu et al., 2024b). To address the challenge of data imbalance, some methods introduce novel loss functions or weighting strategies (Zhang et al., 2023; Bui et al., 2024; Wang et al., 2024b; Yu et al., 2024; Wu et al., 2024b). Multi-modal fusion techniques can be implemented to integrate visual and textual data and improve person retrieval (Wang et al., 2024b; Cormier et al., 2024; Galiyawala, Raval & Patel, 2024).

The reviewed methods offer diverse approaches to PAR and attribute-based person retrieval tasks, each excelling in different scenarios, and the methods used in each article are highlighted in Table 2. Person open attribute recognition (POAR) (Zhang et al., 2023), with its Transformer-based Embedding + Attribute Search (TEMS) approach, leverages general knowledge distillation (GKD) to enhance cross-dataset transferability, achieving higher Recall@1 in PETA-to-PA100K and PETA-to-RAPv1 setups. However, it underperforms in reverse dataset transitions. UFineBench (Zuo et al., 2024) introduces a novel ultra-fine-grained dataset (UFine6926) and the mSD metric, excelling in textual descriptiveness but lacking robust numerical performance comparisons with other methods. Cross-Transformers with Pre-trained Language model is All you need for Person Attribute Recognition and Retrieval (CLEAR) (Bui et al., 2024) enhances embedding vector separation through hybrid attribute representations, outperforming other models in retrieval accuracy on Market-1501 and PA100K. In contrast, Label2Label (Li et al., 2022) sets a benchmark in mA, accuracy, and F1-score for PA100K, utilizing a language modeling framework. For text-image retrieval, Multi-Attribute and Language Search (MALS) (Yang et al., 2023b) stands out, offering significant Recall@1 improvements using the Attribute Prompt Learning and Text Matching (APTM) framework and large-scale image-text pairs. Video-based person attribute recognition (PAR) benefits from VTFPAR++ (Wang et al., 2024b), which uses spatiotemporal tuning for superior accuracy and F1-scores on Matching Attribute-aware Representations for Text-based Sequential Recommendation (MARS)-Attribute and DukeMTMC-VID-Attribute. Meanwhile, methods like Exploring Attribute Localization and Correlation (EALC) (Weng et al., 2023) and Knowledge Distillation Pedestrian Attribute Recognition (KD-PAR) (Wu et al., 2024b) focus on attention mechanisms and knowledge distillation, respectively, achieving competitive accuracy and F1-scores with lightweight designs. For datasets emphasizing pose discrimination, HydraPlus-Net (HP-Net) (Liu et al., 2017) offers state-of-the-art top-1 accuracy through multi-scale key-point learning. In cross-modal attribute relationships, Multi-Correlation Graph Convolutional Network (MCGCN) (Yang et al., 2023a) uses graph convolutional networks for high recall but lacks dominance in mA on PA100K. Finally, the UPAR-Challenge 2024 baseline (Cormier et al., 2024) highlights the need for improved harmonic mean metrics in benchmark challenges. Overall, the method selection depends on task requirements: CLEAR for retrieval accuracy, MALS for cross-modal granularity, and VTFPAR++ for spatiotemporal analysis, while Label2Label dominates attribute prediction tasks.

Open attribute recognition

Despite significant advancements in person search, the exploration of open attribute recognition, where attributes are not predefined, remains a largely under-explored field. One article that pioneered in the field is POAR, a novel approach introduced by Zhang et al. (2023) that uses image-text search to predict pedestrian attributes beyond predefined classes. POAR employs a Transformer-based encoder with a masking strategy to focus on attributes of specific pedestrian parts and encode them into visual embeddings. A many-to-many contrastive loss with masked tokens is applied to handle multiple attributes. The effectiveness of this method as a baseline for POAR is demonstrated through evaluations on three PAR datasets in an open-attribute setting.

The methodologies presented for open attribute recognition vary significantly depending on the task, but some common themes and contrasts emerge. For object retrieval, the MS-UGCML method’s unsupervised, collaborative metric learning approach (Tang et al., 2024) offers a more flexible alternative to methods that rely on labeled data. However, its dependence on feature embedding dimensionality might limit its performance. Attribute extraction benefits from OpenTag’s active learning and attention mechanisms (Zheng et al., 2018), which are better for handling complex, real-world product descriptions than older, hand-engineered feature methods. Similarly, SOAE’s multi-component approach combining Semi-Open Attribute Extraction (SOAE) and Ontology-based Attribute Extraction (OBAE) is effective for extracting a wide range of attribute information (Zhang et al., 2021a). While open-set recognition methods often incorporate semantic information to identify the unknown, approaches like Dynamic Attribute-guided Few-shot Open-set Network (DAFON) offer potential advantages. DAFON, for instance, utilizes attribute-guided feature alignment, which may make it more robust and interpretable than traditional methods, particularly when facing scarce data and unknown distributions in medical imaging (Luo et al., 2024). However, the reliance on attribute information is also a limitation when it is not available. For zero-shot learning (Long, Liu & Shao, 2017), the semantic-to-visual embedding approach is a departure from visual-to-semantic embeddings and is more suitable when the focus is on generating visual representations based on semantic descriptions. This is particularly useful in fine-grained settings, where visual differences are subtle. When analyzing the VQA models, Tri-VQA’s triangular reasoning is better than joint embedding methods for medical VQA due to the explicit reasoning process, improving the answer’s reliability (Fan et al., 2024). Finally, the survey on open vocabulary learning (Wu et al., 2024a) shows that knowledge distillation techniques are beneficial, but their ability is tied to the teacher model’s capabilities. The multi-attribute open-set recognition method explored here uses a simple extension of single-task open-set recognition (Saranrittichai et al., 2022) with multiple prediction heads for each attribute. This is the least sophisticated method for open-set recognition with multiple attributes, especially compared to methods such as DAFON.

Discussion

Cross-modal and multimodal ReID aims to bridge the gap between different modalities, such as images, text, and sketches, to improve person re-identification. Cross-modal interaction modules (Wang et al., 2022b) are effective for integrating information from different modalities while preserving unique features; however, they may not address issues like background clutter or occlusions. For handling multi-modal data, multi-modal margin loss can be beneficial for learning modality-specific representations. Methods like descriptive fusion models (Zhai et al., 2022), which combine text and sketches, excel in scenarios where detailed descriptive information is available, but are complex and computationally intensive. Multi-modal uncertainty modeling (Zhao et al., 2024) is advantageous when dealing with inherent data uncertainty, offering a probabilistic approach, but may add complexity to the model. Approaches using prototypical contrastive learning (Zheng et al., 2024b) are useful for weakly supervised settings, leveraging prototypes to overcome intra-modal variations and cross-modal gaps, but their performance is closely tied to the quality of the generated prototypes. Methods like robust dual embedding (RDE) (Qin et al., 2024) are specifically designed for noisy environments, making them suitable when data is not perfectly aligned. Mask-guided dynamic dual-task collaborative learning (Shao et al., 2024) is beneficial for handling background clutter by using masks but may require precise mask generation. Multi-granularity contrastive consistency models (MGCC) (Wu et al., 2024c) are suited for handling occlusions by using token selection mechanisms. Multi-level cross-modality learning (Wu et al., 2023a) is valuable for extracting and aligning both global and local features and is suitable when both are critical for performance. Finally, multimodal large language models (Yang & Zhang, 2024) are powerful for leveraging the capabilities of LLMs in ReID, especially with their instruction-following capabilities, but they are computationally intensive and may not be suitable for edge devices. Choosing a method depends on the specific needs of the task, such as available data, presence of noise or occlusion, and computational resources.

NLP module

The natural language processing (NLP) module, shown in Fig. 4, transforms user-provided text queries into a structured format suitable for attribute-based people search. The module consists of three main steps: keyword extraction, attribute verification, and synonym generation. This is done to ensure that textual descriptions are correctly mapped to relevant visual features for integration with the system’s subsequent components.

The process begins with keyword extraction, which employs a deep learning approach based on bidirectional encoder representations from transformers (BERT) to identify descriptive terms from the input sentence. BERT’s bidirectional transformer architecture allows it to process the input sequence concurrently from both left to right and right to left, thereby capturing the contextual relationships of words within the entire sentence. For instance, in a query like “Find a tall man wearing a red jacket,” keywords such as “tall,” “man,” and “red jacket” are isolated. Tokenization, part-of-speech tagging, and named entity recognition (NER) are used in this step. The extracted keywords are then inserted into the attribute verification stage, where the keywords are mapped to a predefined set of 26 attributes derived from the PA100k dataset, including categories such as clothing color, gender, and carrying objects.

The module then incorporates a synonym generation mechanism. Synonyms for recognized attributes are generated using a T5 (Text-to-Text Transfer Transformer) model. This allows the system to handle different linguistic variations in user queries using T5’s robust capabilities in text generation and transformation. For instance, synonyms like “blonde” for “fair-haired” or “coat” for “jacket” are aligned with the predefined attributes. This module produces a structured output that includes a list of verified attributes and their synonyms, which are used for the subsequent tasks of image attribute extraction, ranking, and retrieval.

Person detection module

The Person Detection module, shown in Fig. 5, is responsible for accurately identifying and localizing individuals within an image or video frame. This module uses a YOLOv9 model augmented with RetinaFace model, which is a state-of-the-art deep learning framework for person detection that offers high accuracy and speed. The model has a single-stage detection architecture. The feature extractor used has a ResNet backbone to extract hierarchical features from input images. Additionally, a Deformable Convolutional Networks (DCNs) is incorporated to enhance contextual understanding by adaptively adjusting the receptive fields to focus on areas with high relevance, such as regions that are likely to contain faces or body parts. For the detection, a cascade regression strategy is used to enhance the bounding box predictions iteratively for better accuracy and more precise bounding boxes that encapsulate the detected individuals. This approach ensures that the bounding boxes closely align with the detected individuals, minimizing false positives and missed detections. A multi-task loss function is used to simultaneously minimize errors in bounding box regression and landmark localization. Finally, the feature pyramid network in RetinaFace enables multi-scale detection, ensuring that both small and large individuals within the same frame are accurately localized. The bounding box predictions are processed in real time which makes RetinaFace an ideal choice for video surveillance systems that require high-throughput processing.

Person open attribute recognition module

The POAR model is divided into two stages: the traditional PAR model and the open attribute recognition model. The PAR model, shown in Fig. 6, is designed to extract and identify various attributes of individuals in surveillance video. The model is trained to recognize a wide range of open-set attributes, meaning it can identify attributes that are not limited to a predefined set. This offers flexibility in recognizing diverse and dynamic features. Once the open attributes are obtained from the people detected in video frames, the attributes are compared to the attributes extracted from the NLP query to find and retrieve the best-matching individual image.

Open attribute recognition

This methodology improves the detection of comprehensive person attributes by introducing a two-stage attribute recognition process. The predefined attributes from the previous module are extracted and structured as key-value pairs, representing attribute types and their corresponding values. The system then employs an open attribute recognition stage, shown in Fig. 7, where both the structured attributes and visual data are input into a multimodal model. This model utilizes the BLIP-2 architecture and a 6.7 billion parameter OPT backbone to expand attribute detection beyond the initial set effectively.

To start, the image is analyzed using a fixed visual processor, similar to Vision Transformers (ViT), to convert it into a set of precise, numerical representations of its different sections. A key improvement is the introduction of a Q-former as a refining mechanism by using adjustable ‘prompt’ indicators to interact with the encoded image data. The Q-Former within BLIP-2 functions as an information bottleneck and condenses the visual information from the person’s image using cross-attention mechanisms. This efficient compression enables the large language model to focus its attention on the most relevant visual aspects alongside the input attribute pairs. This targeted attention facilitates the recognition and extraction of unseen attributes, surpassing the limitations of conventional systems restricted to predefined attribute lists.

The condensed visual data is then fed into OPT, a pre-trained language model. The language model uses attention mechanisms to reference previously generated words and refined visual information to generate detailed and unseen attributes. The identified novel attributes are organized into key-value pairs and grouped under the ’additional attributes’ key to maintain data consistency and integrate seamlessly into the person ranking module. Furthermore, the corresponding person image is linked to the attribute key-value pairs, enabling robust multimodal reasoning.

Experimental settings

The framework was developed using Python and PyTorch. All experiments were conducted on Google Colab with a Tesla V100 GPU with 32 GB of VRAM and an Intel Xeon Gold 6130 CPU with 4 vCPUs and 25 GB of RAM. The PAR module training initially used a batch size of 16, a learning rate of 0.01, and the stochastic gradient descent (SGD) optimizer. However, overfitting was observed and adjustments were made to the training parameters. The optimized training setup included an increased batch size of 125, a reduced learning rate of 0.001, and a switch to the Adam optimizer. The performance of the models was evaluated using accuracy for overall prediction correctness, mean accuracy for the average accuracy across all attributes, precision for the accuracy of positive predictions, recall for the model’s ability to identify all relevant instances, F1-score for balancing precision and recall, and Rank-1 accuracy to evaluate if the top predicted result matches the ground truth of the retrieved person.

Data description

The PA-100k dataset (Jin et al., 2024), shown in Fig. 8, is considered the largest available for pedestrian attribute recognition and is used for both training and evaluation. It contains 100,000 images captured from outdoor surveillance cameras, with 26 binary attributes (yes/no) labeled for each image. The dataset is split into 80,000 images for training, 10,000 for validation, and 10,000 for testing. The images were collected from 598 outdoor surveillance cameras which gives a rich and diverse set of real-world scenarios. Due to the diverse quality and resolution of the images, some are blurry, which makes precise attribute recognition more difficult. Additionally, several binary attributes are infrequent in the dataset, which may introduce class imbalance and affect model training performance. The images vary in resolution, ranging from 50 × 100 pixels to 758 × 454 pixels. The dataset attributes can be categorized as follows:

• Gender: “Female” (yes/no).

• Age Groups: “AgeOver60”, “Age18-60”, “AgeLess18”.

• Viewpoint: “Front”, “Side”, “Back”.

• Accessories: “Hat”, “Glasses”, “HandBag”, “ShoulderBag”, “Backpack”, “HoldObjectsInFront”.

• Clothing Style: “ShortSleeve”, “LongSleeve”, “LongCoat”, “Trousers”, “Shorts”, “Skirt&Dress”, “Boots”.

• Clothing Patterns and Details: “UpperStride”, “UpperLogo”, “UpperPlaid”, “UpperSplice”, “LowerStripe”, “LowerPattern”.

NLP module results

The NLP module’s performance in processing input textual queries for person re-identification is detailed in this subsection, focusing on key performance metrics across keyword extraction, attribute verification, and synonym generation.

Regarding the keyword extraction stage, we investigated various models that employ different techniques. As Table 3 shows, DistilBERT, a BERT-based model, achieved superior keyword extraction performance, with an F1-score of 0.98, precision of 0.97, and recall of 0.99. This strong showing stems from its Transformer architecture and self-attention mechanism, which allow it to identify highly relevant and semantically meaningful keywords by analyzing complex word relationships rather than just frequency. Compared to traditional methods like TF-IDF and TextRank, DistilBERT demonstrated significantly better contextual understanding, accurately detecting keywords even in complex text. Attribute verification results were assessed against the PA-100K dataset, showing strong agreement between predicted and ground truth attributes. The model achieved high accuracy for various attributes, especially in well-represented categories like gender, accessory presence, and clothing color. The overall attribute verification accuracy reached 94.3%. This outcome highlights the model’s reliability in attribute recognition and its ability to generalize effectively to real-world situations.

For synonym generation, the T5 (Text-to-Text Transfer Transformer) model’s capabilities were assessed, focusing on its effectiveness in maintaining semantic accuracy while producing varied outputs. BERTScore and METEOR metrics were employed to evaluate T5’s performance against other language models, specifically ChatGPT API and GPT-3, in regard to paraphrased semantic text quality. As Table 4 details, T5 consistently outperformed both ChatGPT and GPT-3, achieving the highest BERTScore of 0.935 and a METEOR score of 0.7958. This indicates its superior ability to generate synonyms that closely preserve the original sentences’ semantic meaning. Furthermore, T5 models are generally known for their efficiency, often requiring fewer GFLOPs for inference compared to larger, more complex models like ChatGPT, making them more computationally efficient for real-world scenarios.

Person detection module results

The person detection module, utilizing YOLOv9 and the RetinaFace model, demonstrated outstanding performance across test video surveillance footage under different conditions. The YOLOv9 model was evaluated on the WIDER FACE (Yang et al., 2016) benchmark dataset for person detection. Figure 9 shows the [email protected] and [email protected]:0.95 performance of YOLOv9 across various IoU thresholds, showcasing its robustness and precision in person detection across varying detection scales.

Table 5 summarizes the quantitative performance metrics of YOLOv9 in comparison to state-of-the-art models, including YOLOv8, RetinaNet, and Faster R-CNN. YOLOv9 showed superior performance compared to YOLOv8 and other state-of-the-art models in both [email protected] and [email protected]:0.95, indicating its effectiveness in detecting individuals with high precision and recall. Specifically, YOLOv9 achieved a [email protected] of 89.7%, which is a 2.5% absolute improvement over YOLOv8. Despite the improved accuracy and speed, YOLOv9 maintains a tiny model size of 85 MB, comparable to YOLOv8 but significantly more compact than RetinaNet and Faster R-CNN. Therefore, YOLOv9 can be used for edge deployment in resource-constrained environments, including video surveillance, autonomous systems, and human-computer interaction.

The YOLOv9 model augmented with the RetinaFace module was tested on bird’s eye view footage in both daylight and nighttime, and Fig. 10 demonstrate the results.

The integrated module achieved 98.1% average precision in daylight bird’s-eye view footage with 6,832 frames and 93% average precision in nighttime bird’s-eye view footage with 10,871 frames. The module has outperformed other person detection models, including ResNet and YOLOv9, across the test video footage, highlighting the robustness of RetinaFace in handling different environments and conditions such as low resolution, blurred images, and partial occlusions.

Person attribute recognition module results

The performance of the person attribute recognition module was evaluated by comparing the individual components (MNet, AF1, AF2, AF3) and the integrated whole model. The model was trained for 40 epochs using the fine-tuned weights of MNet, AF1, AF2, and AF3 modules. The initial training of MNet faced issues of overfitting due to a batch size of 16 and a learning rate of 0.01, with SGD as the optimizer. To resolve this, several optimizations were implemented to the module, including increasing the batch size to 128, reducing the learning rate to 0.001, and switching to the Adam optimizer. These changes yielded significant performance improvements. The use of Adam optimizer mitigated the overfitting issue and enhanced the MNET model’s convergence by adaptively adjusting the learning rate for each parameter and correcting gradient estimations, leading to improved learning efficiency and memory utilization. As shown in Table 6, the whole model consistently outperforms each individual component across all metrics, achieving an accuracy (Acc) of 83.7%, a mean accuracy (mAcc) of 83.3%, and an F1-score of 90.4%. MNet, which acts as the backbone of the module, achieved an accuracy of 82.4% and an F1-score of 89.7%. The attribute feature models AF1, AF2, and AF3 exhibited varying degrees of effectiveness, with AF3 achieving the highest performance among the individual components, with an accuracy of 82.8% and an F1-score of 89.8%. While each component contributes to the model’s overall performance, the integrated model demonstrated superior precision, recall, and recognition capabilities, highlighting the importance of combining these components for optimal person attribute recognition. The robustness of the model was further demonstrated by its stable performance across different data splits, emphasizing its capacity to generalize well to unseen data. The implementation of optimization techniques, particularly the Adam optimizer, played a critical role in these performance improvements, setting a strong foundation for future developments in pedestrian attribute recognition.

As shown in Table 7, the proposed model significantly outperforms Hydraplus-Net across all evaluation metrics. The proposed model achieves an accuracy (Acc) of 83.7%, which is 17.83% higher than the accuracy of Hydraplus-Net of 65.87%. It also surpasses Hydraplus-Net in mean accuracy (mAcc) by 16.26%, with the proposed model achieving 83.3% compared to 67.04% for Hydraplus-Net. The precision of the proposed model of 92.8% is 10.72% higher than that of Hydraplus-Net, which scored 82.08%. The proposed model outperforms Hydraplus-Net in recall by 13.66%, achieving 88.1% compared to 74.44%. Finally, the proposed model’s F1-score of 90.4% is 12.32% higher than the Hydraplus-Net F1-score of 78.08%. The results distinctly demonstrate significant improvements in the proposed model’s performance, indicating its higher performance in person attribute recognition tasks.

Person open attribute recognition module results

Figure 11 provides a visual comparison of images before and after processing through the open attribute recognition (OAR) module and shows the capacity of the model to refine and improve attribute extraction. The module employs cross-attention mechanisms to identify and highlight key visual details, allowing it to infer attributes beyond those explicitly seen during training. The qualitative results confirm its effectiveness in generating detailed and contextually appropriate attribute descriptions and generalizing across various visual inputs.

The images in Fig. 12 illustrate a selection of outputs after processing through the OAR module and demonstrate its ability to generate meaningful attribute descriptions in diverse scenarios. The results illustrate how the model adapts effectively to new visual inputs, recognizing attributes beyond predefined categories through its attention-based approach.

The evaluation of the proposed end-to-end retrieval system is conducted using Rank-1 Accuracy of the person retrieval as the primary metric, which is a standard measure in retrieval-based tasks that measures the proportion of test queries for which the system correctly identifies and retrieves the most relevant person as the top-ranked result. The evaluation was performed on the benchmark dataset PA100k annotated with detailed person attributes, ensuring a rigorous and standardized testing environment. By leveraging this metric, the results demonstrate the system’s retrieval accuracy and its practical usability in real-world applications.

Currently, only one study has explored POAR, which is the article by Zhang et al. (2023). However, it does not utilize an end-to-end framework. It also formulates attribute recognition as an image-text search problem and relies on similarity computations between visual and textual embeddings. On the other hand, our method integrates attribute extraction within a fully end-to-end framework. This eliminates the need for predefined attribute descriptions and manual text encoding, allowing our model to generalize more effectively to unseen attributes.

Table 8 provides a comparative analysis of rank-1 accuracy between the two approaches. As shown in Table 8, our proposed methodology achieves superior Rank-1 accuracy, which demonstrates the effectiveness of our end-to-end, attention-driven architecture in accurately recognizing both seen and unseen attributes.

With a Rank-1 accuracy of 84.8%, the proposed method outperformed POAR (TEMS) and POAR (TEMS + GKD) by 1.5% and 3.5%, respectively (Zhang et al., 2023). This indicates that the proposed method not only performs better than standard approaches but also demonstrates how well it handles open attribute recognition jobs and provides a more reliable solution in correctly identifying person attributes and retrieving the correct individual.

Table 9 presents a comparison of our proposed approach against various text-based and attribute-based person search methods on the PA100K dataset. Rank-1 accuracy is used as the evaluation metric, representing the proportion of correctly identified matches at the top rank. Traditional approaches such as 2WayNet and CMCE exhibit lower accuracy, while more recent techniques like ASMR and WSFG show improvements. Our method outperforms all existing models, achieving an impressive Rank-1 accuracy of 84.8% and proving its effectiveness in text-based person search.

End-to-end deployment

To validate the practical applicability of the proposed surveillance framework, an end-to-end deployment in a cloud environment that simulates real-world conditions was conducted. This comprehensive evaluation aimed to assess the system’s responsiveness and processing efficiency in retrieving the best-matching individual from a surveillance video based on a user-inputted textual query. The computational complexity of each module in the pipeline is shown in Table 10.

The surveillance system is designed to operate in two distinct phases. The initial phase, where we create the image gallery’s attributes, is done offline since it’s computationally intensive. The second phase, however, handles making text query attributes from the user searches and then ranking and retrieving the person of interest by checking against the image gallery attributes. This part happens in real-time, so users can quickly type in a query and get results for a person from videos already processed by the system. This division makes the system efficient and scalable while still giving us rich details for describing and searching for people. For evaluation, the complete system was deployed on Google Colab Pro using an NVIDIA Tesla V100 GPU (16 GB). A 60-s surveillance video at 1,920 $\times$ 1,080 resolution and 12 frames per second (FPS) was used as the video input.

The image-based pipeline processes visual surveillance data offline to construct a structured attribute set for each image in the gallery. Due to the high computational cost of models such as BLIP2 and the large volume of image data within surveillance environments, this phase of processing is not executed in real-time. The process begins by extracting key frames at a rate of one per second from the video for the person detection module, which effectively manages workload while capturing all key actions. Powered by the V100 GPU, YOLOv9-Small quickly detects people, while DeepSORT tracking enables the system to store the first instance of each person in the gallery. Following this, the PAR module creates the closed-set attributes for each person in the image gallery. Finally, the OAR module uses BLIP2-OPT to generate the open attributes. The overall computational cost for the image-based pipeline is approximately 700–1,000 GFLOPs per person instance, which is mainly affected by the OAR module. However, performing this entire phase offline makes the system feasible for real-world use, allowing for the comprehensive and rich descriptions of surveillance footage without impacting the real-time operations of the person search.

On the other hand, the NLP module that processes the user text query to create the open text attributes is activated in real-time. This process begins with DistilBERT extracting keywords from your query, followed by the attribute verification step and a T5-base model to generate synonyms. The system then finds the best matching person of interest by performing person ranking and retrieval. The lightweight nature of these modules ensures minimal delay, with a computational complexity of 40-60 GLOPS per user query, making the retrieval system interactive and real-time, even under constrained computational environments.

We tested the end-to-end person re-identification system using three one-minute surveillance videos. On average, the system detected 20 people in these videos. The system processed the 20 detected people and compared their information to all 20 gallery attributes to retrieve the best-matching individual. We then calculated the GFLOPs needed by running the system on a Google Colab Pro V100 GPU. The real-time online text part was incredibly efficient, using only about 41.5 GFLOPs. In contrast, the offline image part is more computationally intensive and requires about 15,984 GFLOPs. Nevertheless, even with that higher demand, a powerful GPU can enable us to process small video segments in real-time with the offline system.