Enhancing scene graph generation via hybrid co-attention and predicate reweighting for long-tail robustness

Scene graph generation

SGG has evolved along two primary architectural paradigms to bridge visual perception and semantic understanding, yet persistent challenges remain in cross-modal alignment and long-tail distribution handling. Single-stage methods, such as RelDN (Zhang et al., 2019), employ contrastive losses to mitigate entity confusion, while JMSGG (Xu et al., 2021) jointly models objects and relations by capturing their inter-dependencies. However, these methods may not effectively capture complex relationships between objects and relations, and their ability to handle cross-modal alignment is limited. Conversely, two-stage methods excel at hierarchical reasoning. For example, MotifNet (Zellers et al., 2018; Lu et al., 2021) and VCTree (Tang et al., 2019) establish classical sequence/tree-based context modeling, and RU-Net (Lin et al., 2022b) with unrolled optimization and HL-Net (Lin et al., 2022a) handle graph heterophily. But even these advanced methods have limitations. GPS-Net (Lin et al., 2020) utilizes message-passing mechanisms to facilitate information exchange between visual regions and textual predicates. However, this method suffers from a semantic granularity mismatch. The message-passing process may not effectively align the fine-grained semantic details of visual features with the more abstract textual predicates, resulting in suboptimal cross-modal alignment and limiting the accuracy of the generated scene graphs. BGNN (Li et al., 2021) adopts simple addition or concatenation operations for multi-modal feature fusion. This straightforward approach often results in incoherent representations as it fails to adequately capture the complex relationships and semantic interactions between different modalities. To address the issue of semantic ambiguity, Hiker-SGG (Zhang et al., 2024) employs a hierarchical attention mechanism to incorporate multi-scale knowledge. To further explore efficient architectures, Hydra-SGG (Chen et al., 2025) proposes a hybrid relation assignment mechanism within a one-stage framework. Furthermore, with the advent of large language models, recent work (Chen, Li & Wang, 2024) has begun to leverage role-playing large language models (LLMs) to refine scene graphs by utilizing the powerful commonsense knowledge embedded in pre-trained models. Despite these advances in architecture, the treatment of bias remains crucial. Recent efforts like Lyu et al. (2022) and Dong et al. (2022) address bias mitigation through fine-grained predicates learning and stacked hybrid-attention, respectively, while Zheng et al. (2023) achieves competitive performance via prototype-based embeddings, but they overlook the dynamic inter-predicate correlations crucial for long-tail robustness. These limitations highlight the need for a more sophisticated approach that can effectively address both cross-modal alignment and long-tail distribution issues. To tackle these challenges, we propose a unified framework to offer a more comprehensive solution.

Cross-modal attention mechanism

Recent advances in multimodal fusion (Bayoudh, 2024) and cross-attention modeling (Ren et al., 2024; Han et al., 2025) have garnered increasing interest across various vision-language tasks. As noted in Li et al. (2022b), the naive addition fusion used in current SGG methods (e.g., Motifs; Zellers et al., 2018) disproportionately amplifies high-frequency predicates, biasing predictions toward common relations. Notably, attention-based mechanisms like the Stacked Hybrid-Attention (SHA) network (Dong et al., 2022) have been applied to SGG, which combines Self-Attention (SA) and Cross-Attention (CA) units in parallel layers to enhance intra-modal refinement and inter-modal interaction. However, SHA’s parallel design processes SA and CA concurrently within each layer, which improves efficiency but lacks progressive alignment between modalities. This issue is further complicated by the inherent semantic gap between visual region of interest (ROI) features (typically extracted from Faster R-CNN; Zhao, Wei & Xu, 2024) and learned linguistic predicate embeddings. Additionally, few methods have adequately addressed the weak fusion between object proposals and their corresponding class names in SGG. To bridge this gap and overcome the limitations of existing cross-modal attention mechanisms, we propose the HCA to improve multi-modal representation learning and addressing the semantic granularity mismatches and weak fusion issues present in previous methods.

Long tail recognition

In real-world scenarios, the distributions of object and relation categories are inherently imbalanced, with natural data exhibiting a long-tail pattern (Zhang et al., 2017; Wang et al., 2021a; Zhao et al., 2024). Various methods have been proposed to address long-tail bias. TDE (Tang et al., 2020) employs counterfactual causality to address bias in scene graph generation. Although this method can mitigate the impact of spurious correlations to some extent, it has weak contextual modeling capabilities. It does not effectively account for the rich contextual information present in images and text, which is essential for accurately capturing the relationships between objects and their attributes. BA-SGG (Guo et al., 2021) categorizes predicates based on information content using Shannon entropy and splits them into common and informative groups for resampling. However, such resampling strategies have significant drawbacks. They are computationally inefficient when handling images with multiple object instances and lack the flexibility of re-weighting approaches. More importantly, these methods operate under the questionable assumption of a linear frequency-information correlation, thereby neglecting the context-dependent nature of predicate relationships, especially for tail categories where semantic meaning is often scene-specific. To address this, we propose the PR module. Unlike previous methods that treat all tail predicates uniformly, PR adaptively adjusts the weights of predicate categories and their inter-predicate relevance. This allows our model to better handle the long-tail distribution problem and improve the performance of scene graph generation by preserving the semantic coherence of the generated graphs.

Problem formulation

The scene graph generation task is essentially a multi-classification problem. To capture the relationships between objects and relations, we adopt a two-stage process (Li et al., 2017): first, detecting all objects within an image using an object detector like Faster region-based convolutional neural network (Faster R-CNN), and then identifying the relationship predicates between pairs of objects. Based on these relationships, we construct a scene graph composed of triplets (subject, predicate, object). Specifically, given an image X, Faster R-CNN is employed as the object detector to detect all objects $O={\left\{o_i\right\}}_{i=1}^N$. Next, for each subject-object pair $\left(o_i,o_j\right)$, predict their visual relationship $\left(r_{ij}\right)$. So we can obtain the triplet for this pair of entities $h\left(o_i,r_{ij},o_j\right)$. For the image X, we use all triplets to construct the scene graph $G=\left\{\left(o_i,r_{ij},o_j\right)|o_i,o_j\in O,r_{ij}\in R\right\}$, where R represents the set of all possible predicate relationships for the subject-object pairs $\left(o_i,o_j\right)$. For subject-object pairs with no identifiable relationship, the relation is designated as NA.

Framework overview

As illustrated in Fig. 2, the ReBalance-HCA framework comprises two key components: the HCA and the PR module. The framework’s pipeline is as follows:

• 1.

  Feature extraction: Visual and textual features are extracted by Faster R-CNN (Zhao, Wei & Xu, 2024) and GloVe (Pennington, Socher & Manning, 2014), respectively.

• 2.

  HCA: The HCA is the backbone of our framework for multi-modal feature fusion. It consists of two key components: Intrinsic Feature Refinement (IFR) and Context-Guided Attention propagation (CGA). The IFR is designed to refine features within each modality, capturing the intrinsic characteristics of visual and textual features respectively. The CGA, on the other hand, focuses on aligning cross-modal semantics, using information from one modality to influence the attention distribution of another. This dual-component structure enables the HCA to effectively capture both intra-modal and inter-modal relationships, producing more comprehensive and discriminative multi-modal representations.

• 3.

  Predicate reweighting (PR) Module: After obtaining the fused multi-modal features from the HCA, the PR module dynamically adjusts the predicate category distribution. It takes into account both the empirical training data distribution and the inter-predicate semantic relevance, thereby generating a more balanced predicate space.

• 4.

  Balanced fine-tuning: Finally, the initial SGG model, trained on the source domain with imbalanced predicate distribution, is fine-tuned on the balanced target domain produced by the PR module. This fine-tuning process, followed by Guo et al. (2021), focuses on the last layer of the network to maintain efficiency and prevent overfitting to head predicates.

Hybrid co-attention network

The HCA is a critical component of our framework, responsible for multi-modal feature fusion. It is built upon an encoder-decoder architecture and consists of multiple stacked IFR and CGA layers.

Intrinsic feature refinement

As shown in Fig. 3A, the IFR is designed to refine features within each individual modality. It encodes input vectors into query, key, and value vectors, which are then processed through a multi-head attention layer. Each attention head independently computes the attention scores, allowing the model to learn different correlation information in different representation spaces. It computed as

(1) $$\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(Q,K,V)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{Q{K}^{\mathrm{T}}}{\sqrt{d_k}}\right)V$$where Q, K, and V represent the query vector, key vector, and value vector, respectively, and $d_k$ denotes the dimensionality of the vectors. The computation equation for each attention head can be expressed as

(2) $${\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}}_i=\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left(Q{W}_i^Q,K{W}_i^K,V{W}_i^V\right)$$where $W_i^Q,W_i^K,W_i^V$ represents the learnable weight for the $i$-th attention head. The last multi-head attention layer is composed of $m$ such attention heads, and its feature input denoted as:

(3) $$\mathrm{M}\mathrm{A}(Q,K,V)=\left[{\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}}_1,{\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}}_2,\dots ,{\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}}_m\right]W^o.$$

Next, the output of the multi-head attention layer is further refined through a feed-forward layer, followed by residual connections and layer normalization. This process results in a weighted feature representation Z, which better captures the intrinsic characteristics of the input features. The IFR enhances the model’s ability to understand and represent individual modalities, providing a solid foundation for subsequent cross-modal fusion.

Predicate reweighting module

To mitigate the challenge posed by the long-tail distribution of predicate categories, we propose a Predicate Reweighting module that dynamically adapts the context-aware penalty adjustments in the loss function by leveraging semantic correlations among predicates. This module consists of two core components: (1) predicate semantic correlation, and (2) a correlation-based reweighted cross-entropy loss.

Datasets

Visual Genome. We conducted our experiments on the Visual Genome dataset, which comprises over 108,000 images and 2.3 million pairs of relationship instances. Similar to Lin et al. (2020), Tang et al. (2020), we experimented with the VG150 split, which is widely used in SGG and includes the 150 most frequently occurring object categories and 50 predicate relationship categories. Following the VG150 protocol, we split the dataset into 70% for training and 30% for testing. Additionally, we sampled a 5,000-image validation set from the training set, consistent with Zellers et al. (2018).

OpenImages v6. This large-scale dataset is designed for tasks like object detection and visual relationship detection, containing 126,368 images for training, 1,813 for validation, and 5,322 for testing. It includes 301 object classes and 31 predicate classes, with high-quality annotations.

Evaluation metrics

Visual Genome. Based on the prior works, we employed three tasks to comprehensively evaluate performance: (1) Predicate Classification (PredCls): Predicts relationships between all paired objects by utilizing given real bounding boxes and categories. (2) Scene Graph Classification (SGCls): Predicts the categories of objects and the paired relationships between them, utilizing given real object bounding boxes. (3) Scene Graph Detection (SGDet): Detects all objects in the image, predicting their bounding boxes, categories, and paired relationships.

Following recent works, we adopted the Mean Recall@K (mR@K) metric as the evaluation metric. This metric calculates the average recall for each predicate category, providing a fair evaluation method for SGG. Due to R@K’s bias toward head categories in imbalanced datasets (Wang et al., 2021b), we adopt mR@K. This metric computes recall per predicate category before averaging, ensuring fair evaluation of head and tail predicates. Higher mR@K indicates better performance. For each task, we bold the highest mR@K and underline the second.

OpenImages v6. Following the previous works (Zhang et al., 2019; Lin et al., 2022b; Zheng et al., 2023), we utilized the Recall@50 (R@50), weighted mean AP of relations ( $\mathrm{w}\mathrm{m}\mathrm{A}{\mathrm{P}}_{rel}$), weighted mean AP of phrase ( $\mathrm{w}\mathrm{m}\mathrm{A}{\mathrm{P}}_{phr}$) as the evaluation metrics. The final composite score, $\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}{\mathrm{e}}_{wtd}$, is calculated as:

$\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}{\mathrm{e}}_{wtd}=0.2\times R@50+0.4\times \mathrm{w}\mathrm{m}\mathrm{A}{\mathrm{P}}_{rel}+0.4\times \mathrm{w}\mathrm{m}\mathrm{A}{\mathrm{P}}_{phr}$.

Implementation details

All experiments were conducted on Linux-based systems using one RTX 3090 with CUDA support. For the first-stage object detector, we maintain the same configuration as the baseline model, employing Faster R-CNN (Zhao, Wei & Xu, 2024) with ResNeXt-101-FPN backbone pre-trained on Tang et al. (2020).

Data Preprocessing. Training images undergo: (1) Color jittering (brightness/contrast/saturation/hue), (2) random horizontal flipping (50% probability), (3) configurable vertical flipping, (4) aspect ratio-preserving resizing. Evaluation uses resizing only. Semantic embeddings are extracted using GloVe (Pennington, Socher & Manning, 2014).

Training protocol. Initial SGG training uses SGD (lr = 0.001, batch size = 16) for all tasks (PredCls, SGCls, SGDet) with mixed precision, warmup scheduling, gradient clipping, and validation-based early stopping. Specifically, the model is trained for a total of 16,000 iterations. We employ a warmup scheduler: the learning rate starts from $0.001\times 0.1=0.0001$ during the warmup phase and increases linearly to the base rate ( $0.001$). Subsequently, the learning rate is dynamically reduced by a factor of $0.1$ when the validation metric fails to improve for two consecutive validation checks (patience = 2), with a maximum of three reductions. Validation is performed every 2,000 iterations. Our HCA uses a four-layer encoder (four IFRs) and decoder (four CGAs). During target domain adaptation, only the last layer undergoes fine-tuning (Guo et al., 2021).

Reparameterization. Correlation threshold $\beta =0.9$ distinguishes strong/weak predicate-pairs, with reweighting intensity $t=2$.

Results and analysis

As shown in Table 1, ReBalance-HCA outperforms competitive methods in mR@K across all tasks (PredCls, SGCls, SGDet) on the Visual Genome dataset. We attribute this to: (1) HCA achieves effective fusion of visual and textual features, and (2) PR can handle the issue of predicate distribution imbalance and accurately infer appropriate predicates. The results demonstrate that the proposed method consistently learns multimodal feature representations.

To further validate the generalization capability of ReBalance-HCA, we extended our evaluation to the OpenImages v6 dataset. As shown in Table 2, ReBalance-HCA achieves a competitive weighted score ( $\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}{\mathrm{e}}_{wtd}$) of 44.9, matching the PE-Net, while excelling in key metrics such as $\mathrm{w}\mathrm{m}\mathrm{A}{\mathrm{P}}_{rel}$ (36.7) and R@50 (76.8). This performance highlights the method’s robustness in handling complex scenes and varied predicate distributions, which are characteristic of OpenImages v6. The consistency in results across datasets can be attributed to the HCA module’s ability to align cross-modal semantics effectively and the PR module’s dynamic adjustment of predicate correlations, which mitigate domain-specific biases.

Ablation study

In this section, ablation experiments were performed on the VG dataset for the ReBalance-HCA’s model to evaluate the effectiveness of HCA as well as PR in the model, and the results are shown in Table 3. We further evaluated RP by integrating it into several benchmark SGG models for comparison.

Effectiveness of hybrid co-attention network

As shown in Table 3, integrating HCA leads to significant performance improvements. The IFR-only variant increases the baseline PredCls mR@100 by +1.4 points (17.2 vs 15.8), validating its effectiveness in intra-modal refinement. Meanwhile, the CGA-only variant demonstrates stronger cross-modal conditioning, achieving PredCls mR@100 = 17.6 (+1.8 over baseline). The full HCA model achieves optimal synergy with PredCls mR@100 = 17.8, highlighting HCA’s efficacy in multimodal feature fusion.

To qualitatively demonstrate how HCA improves cross-modal alignment, we visualize the attention evolution in Fig. 4. The heatmap focused on the decorated elephant (Fig. 4B) reflects the IFR mechanism, which distills features by enhancing salient regions (e.g., decorative patterns). Conversely, the attention shift to the rider–elephant interaction (Fig. 4C) illustrates the CGA mechanism, where semantics are grounded by linking phrases like “person riding elephant” to visual predicates (e.g., riding).

Effectiveness of predicate reweighting module

As demonstrated in Table 3, the proposed PR module yields significant performance gains across three evaluation tasks. The PR Variants analysis reveals that disabling the reweighting mechanism ( $t=0$) produces identical results to the baseline, while activating full PR yields substantial gains. This improvement can be attributed to two key mechanisms: First, the module dynamically adjusts the loss weights for different predicate categories based on their information content and contextual importance, effectively rebalancing the distribution. Second, it preserves crucial semantic relationships for tail predicates while suppressing dominant head categories, enabling more robust feature learning.

Furthermore, to verify the effect of the PR module on model performance, we integrate it into several SGG benchmark models, Motifs and VCTree, for comparison. As shown in Table 4, our PR consistently improves performance across various SGG benchmark models, outperforming most competitive methods such as BPL (Guo et al., 2021), Inf (Biswas & Ji, 2023), and SKD (Sun et al., 2024). VCTree+Ours achieved the same optimal performance (9.9) as VCTree+BPL on the SGDet task’s mR@20 metric, while Motifs+Ours (9.8) achieved suboptimal performance at a slight disadvantage compared to Motifs+NICE (9.9). This confirms the effectiveness and superiority of our PR and also demonstrates the versatility of PR as a plug-in debiasing module.

Table 4:

Comparation of our PR with competitive SGG models via integrating it into SGG models.

The best performance is highlighted in bold, and the second best is underlined.

	PredCls			SGCls			SGDet
Method	mR@20	mR@50	mR@100	mR@20	mR@50	mR@100	mR@20	mR@50	mR@100
Motifs (Zellers et al., 2018)	11.5	14.6	15.8	6.5	8.0	8.5	4.1	5.5	6.8
+ Debiasing (Cui et al., 2019)	18.8	28.1	33.7	10.6	15.6	18.3	7.2	10.5	13.2
+ TDE (Tang et al., 2020)	18.5	25.5	29.1	9.8	13.1	14.9	5.8	8.2	9.8
+ GCA (Knyazev et al., 2021)	16.4	17.8	18.3	9.6	11.2	12.6	8.0	9.0	10.2
+ STL (Chiou et al., 2021)	13.3	20.1	22.3	8.5	12.8	14.1	5.4	7.6	9.1
+ PCPL (Chiou et al., 2021)	19.3	24.3	26.1	9.9	12.0	12.7	8.0	10.7	12.6
+ DLFE (Chiou et al., 2021)	22.1	26.9	28.8	12.8	15.2	15.9	8.6	11.7	13.8
+ BPL (Guo et al., 2021)	22.6	27.1	29.1	13.0	15.3	16.2	9.7	12.4	14.4
+ NICE (Li et al., 2022a)	23.7	29.9	32.3	13.6	16.6	17.9	9.9	12.2	14.4
+ Inf (Biswas & Ji, 2023)	15.7	24.7	30.7	10.2	14.5	17.4	6.6	9.4	11.7
+ SKD (Sun et al., 2024)	22.3	29.4	32.5	12.2	15.8	17.2	7.4	10.5	13.1
+ PR (Ours)	27.3	36.2	40.3	16.0	19.8	22.0	9.8	13.7	16.8
VCTree (Tang et al., 2019)	11.7	14.9	16.1	6.2	7.5	7.9	4.2	5.7	6.9
+ TDE (Tang et al., 2020)	18.4	25.4	28.7	8.9	12.2	14.0	6.9	9.3	11.1
+ STL (Chiou et al., 2021)	14.3	21.4	23.5	10.5	14.6	16.6	5.1	7.1	8.4
+ PCPL (Chiou et al., 2021)	18.7	22.8	24.5	12.7	15.2	16.1	8.1	10.8	12.6
+ DLFE (Chiou et al., 2021)	20.8	25.3	27.1	15.8	18.9	20.0	8.6	11.8	13.8
+ BPL (Guo et al., 2021)	23.8	28.4	30.4	15.6	18.4	19.5	9.9	12.5	14.4
+ SKD (Sun et al., 2024)	22.3	29.9	32.9	14.3	18.9	20.8	6.9	9.6	11.7
+ PR (Ours)	27.6	36.3	40.3	15.7	20.1	22.3	9.9	13.6	16.3

DOI: 10.7717/peerj-cs.3548/table-4

Moreover, we quantify the computational tradeoffs introduced by the PR module. As shown in Fig. 5A, the pairwise correlation computation between all predicate categories increases training time by 14.36% compared to the baseline. This overhead primarily stems from the operation of conditional power adjustments. Crucially, Fig. 5B demonstrates that during inference, where cached weights replace dynamic calculations, the cost remains negligible. Given the significant performance gains (Fig. 5C), the additional training overhead is considered acceptable.

Visual analysis on long tail robustness

To illustrate intuitively the robustness of ReBalance-HCA on long tail predicates, we compare R@100 results on each predicate for Motifs with/without ReBalance-HCA. As shown in Fig. 6, he ReBalance-HCA gives a significant performance boost to almost all predicates (e.g.,“riding in,” “working on,” and “parked on”), especially those with long tails. The slight loss of head predicates, such as “on”, “has”, may be due to weight adjustments during reweighting; while the large number of long-tail predicates, such as “growing on”, “painted on” and “made of”, may be due to PR’s dynamic punishment of head-class and tail-class sample pairs according to strong and weak correlations during reweighting.

To better visually demonstrate the performance in robust rebalancing on long tail predicates, we present an example from the VG dataset. As shown in Fig. 7, compared to Motifs, our method is more accurate in predicates prediction, particularly for both: (1) common relations (e.g., ‘person-riding-horse’, ‘person-on-horse’), and (2) semantical long-tail predicates (e.g., ‘person-covered-in-forest’, ‘person-across-grass’). This performance improvement stems from the HCA, which refines cross-modal semantic alignment through its stacked fusion process. Furthermore, the PR enhances the model’s discriminative capacity for rare but semantically vital predicates (e.g., spatial relations like ‘across’ and ‘covered in’) while maintaining head category performance. This demonstrates the effectiveness of joint HCA and PR in addressing long-tail challenges for scene understanding.

Hyperparameter sensitivity analysis

As illustrated in Fig. 8, we performed sensitivity experiments on the correlation threshold $\beta$ to evaluate its role in balancing head-tail predicate distributions. For fair and comprehensive comparison, SKD (Sun et al., 2024) was selected as the benchmark model. Defined in Eq. (10), $\beta$ determines the boundary between strong and weak inter-predicate correlations, directly influencing the adaptive reweighting mechanism. We varied $\beta$ from 0.1 to 1.0 in 0.1 increments and evaluated mR@100 performance across three SGG subtasks on the VG dataset. The performance reached its peak at $\beta =0.9$, with the highest mR@100 scores (PredCls: 41.3, SGCls: 22.7, SGDet: 18.7). These results confirm $\beta$’s critical importance in robust long-tail reasoning and empirically validate our selection for balanced reweighting.