Learning with semantic ambiguity for unbiased scene graph generation

Scene graph generation

SGG is dedicated to transforming visual images into semantic graph structures, thereby playing a critical role in merging vision and language. Early methods such as VTransE (Zhang et al., 2017) focused on identifying objects and relations using separate networks, overlooking the wealth of contextual information. Subsequently, iterative message passing (IMP) (Xu et al., 2017) introduced an iterative message-passing mechanism to refine object and relation features, highlighting the substantial role contextual information plays in enhancing relation prediction accuracy. Motifs (Zellers et al., 2018) emphasizes the critical importance of contextual interplay among objects, utilizing BiLSTM to disseminate contextual data effectively. Similarly, Transformer (Tang et al., 2020) captures rich contextual representations of objects by encoding features through self-attention layers. To address the challenges posed by noisy information during message passing, VCTree (Tang et al., 2019) proposes a tree-structured method to efficiently leverage global contexts among objects. Additionally, KERN (Chen et al., 2019) attempts to incorporate prior knowledge into SGG models to improve the precision of relation predictions. Nonetheless, these methods overlook the long-tail distribution in data, resulting in a propensity for predictions to favor high-frequency relations. Such relations tend to be less informative, thereby constraining the utility of these models for downstream tasks.

Unbiased scene graph generation

Unbiased scene graph generation methods aim to rectify the prediction biases stemming from the long-tail distribution of data, with a particular focus on enhancing the model’s performance across various relations. They can be broadly classified into four categories: re-sampling (Dong et al., 2022; Li et al., 2021), re-weighting (Yu et al., 2020; Yan et al., 2020), biased-model-based (Tang et al., 2020; Chiou et al., 2021), and data transfer (Zhang et al., 2022; Li et al., 2022a). Stacked hybrid-attention and group collaborative learning (SHA+GCL) (Dong et al., 2022) employs a median re-sampling strategy, adjusting the sample rates to balance the training sets according to the median relation count within each classification space. Bipartite graph neural network (BGNN) (Li et al., 2021) utilizes a bi-level re-sampling method to achieve a balance in data distribution during the training phase. CogTree (Yu et al., 2020) leverages semantic relations across different categories to devise a loss function that rebalances the weights. Predicate-correlation perception learning (PGPL) (Yan et al., 2020) dynamically identifies appropriate loss weights by recognizing and leveraging relation category correlations. TDE (Tang et al., 2020) calculates the difference between the original and counterfactual scenes to remove context bias, ensuring unbiased scene graph generation. Dynamic label frequency estimation (DLFE) (Chiou et al., 2021) dynamically estimates label frequencies by maintaining a moving average of biased probabilities, allowing the model to recover unbiased probabilities.

Although these methods alleviate bias and improve low-frequency relations performance, they often compromise high-frequency relations performance and neglect the semantic ambiguity inherent in visual relations. Recent works (Zhang et al., 2022; Li et al., 2022a) argue that semantic ambiguity could be alleviated if there is a reasonable and sound dataset. IETrans (Zhang et al., 2022) introduces an Internal and External Data Transfer method to achieve the transfer of high-frequency to low-frequency relations and the relabeling of relations for unannotated samples. NoIsy label correction (NICE) (Li et al., 2022a) redefines SGG as a noisy label learning issue, presenting a strategy for noisy labels correction aimed at bias mitigation. It effectively cleanses noisy dataset annotations to equalize the data distribution.

These methods treat relation classification as a single-label problem and use one-hot target labels to train the relation classifier in SGG models. In one-hot target labels, each relation is represented as a binary vector where only one relation is set to 1 (indicating the target relation), and all other relations are set to 0. This method is highly effective for clear and mutually exclusive classification tasks. However, it fails to capture the nuances in scenes with semantic ambiguities, where relations are not mutually exclusive. In contrast, soft labels assign a probability to each relation, indicating the likelihood that the sample belongs to each relation and revealing the subtle differences between them. Our proposed method improves upon this by generating a training label distribution that considers semantic similarities and differences between relations. This method achieves balanced performance across both high-frequency and low-frequency relations in the model.

Label smoothing and label confusion

Label smoothing (Szegedy et al., 2016) is a regularization technique designed to prevent overly confident predictions on training examples. It achieves this by mixing one-hot label vectors with a uniform noise distribution. However, this method of generating soft labels, primarily by introducing noise, fails to capture the semantic ambiguity within samples. Label confusion learning (Guo et al., 2021) was proposed for text classification tasks, introducing a label confusion model that calculates the similarity between instances and labels during training. This model generates a probability distribution, superseding the traditional one-hot label vectors. In addition, label semantic knowledge distillation (LS-KD) (Li et al., 2023) dynamically generates soft labels for each subject-object pair by merging the model’s relation label prediction distribution with the original one-hot labels. However, the prevailing long-tail distribution skews the model’s predictions towards more frequent relations, making it challenging to generate soft labels that accurately reflect the differences between relations. In contrast to these methods, we measure the similarity and differences between relations by calculating the amount of information for each relation. This method ensures that the generated soft labels more accurately reflect the similarities and differences between relations, leading to improved model performance and better handling of low-frequency relations.

Problem definition

The task of SGG is to construct a scene graph G for a given image $I\in {\mathbb{R}}^{H\times W\times 3}$. This graph G comprises a set of objects $O=\{(b_i,c_i){\}}_{i=1}^{N_o}$ and a set of relation triplets $E=\{(s_i,r_i,o_i){\}}_{i=1}^{N_e}$, collectively denoted as $G=(O,E)$. Each object in O, represented by $(b_i,c_i)$, includes an object bounding box $b_i\in {\mathbb{R}}^4$ and an object category $c_i$, which is part of the pre-defined object category set C. Furthermore, each relational triplet $(s_i,r_i,o_i)$ is composed of a subject $s_i\in O$, an object $o_i\in O$, and a relation $r_i$ between them, where $r_i$ is a member of the predefined set of relation categories R.

Mixup relation learning

To tackle semantic ambiguity, the Mixup relation learning (MRL) module enriches the dataset by allocating soft labels to samples of relation triplets that exhibit semantic ambiguities. These soft-labeled samples are subsequently employed in the training of SGG models.

Following Zhang et al. (2022), we first identify confusion pairs as semantically similar relation pairs, since informative relation categories are easily confused with general ones. Specifically, for each relation triplet category $(c_s,r_{\ast },c_o)$, we use a pre-trained baseline model to predict relation labels of all samples belonging to $(c_s,r_{\ast },c_o)$ in the training set, and average their score vectors. Subsequently, relations with a predicted score higher than that of the ground-truth relation are regarded as semantically similar to the ground-truth relation $r_{\ast }$. This is formalized as $R_{sim}=\{r_i|p_{r_i}>p_{r_{\ast }}\}$, where $p_{r_i}$ is the predicted score for the $i$-th relation and $p_{r_{\ast }}$ denotes the predicted score for the ground truth relation $r_{\ast }$. Based on this, we collect all samples in the training set satisfying Eq. (1):

(1) $$T_{sim}=\{(s_j,r_j,o_j)\mid (c_{s_j}=c_s)\wedge (r_j\in R_{sim})\wedge (c_{o_j}=c_o)\}$$where $\wedge$ denotes the logical conjunction operator. We quantify the information contained in $r_{\ast }$ and $r_j$ within the subject-object pair. Soft labels are then assigned to all samples in $T_{sim}$ based on the proportion of information content between $r_j$ and $r_{\ast }$, replacing the original one-hot labels $r_j$.

To achieve this, we use an attraction factor (Zhang et al., 2022) to calculate the amount of information contained in the relation within each relational triplet, as defined in Eq. (2):

(2) $$A(c_s,r_{\ast },c_o)=\frac{N(c_s,r_{\ast },c_o)}{{\sum }_{c_i,c_j\in C}I(c_i,r_{\ast },c_j)\cdot N(c_i,r_{\ast },c_j)}$$where $N(c_s,r_{\ast },c_o)$ denotes the number of samples of the relation triplet $(c_s,r_{\ast },c_o)$ within the training set, and $I(c_i,r_{\ast },c_j)$ indicates whether the triplet category $(c_i,r_{\ast },c_j)$ exists in the training set. $I(c_i,r_{\ast },c_j)$ returns 1 if the relation triplet $(c_i,r_{\ast },c_j)$ is present in the training set, and 0 otherwise. A higher $A(c_s,r_{\ast },c_o)$ indicates that the relation triplet is relatively more unique or carries more information within the entire dataset. This is because it represents a larger proportion among all triplets with relation $r_{\ast }$. Based on this, we assign the relation $r_{\ast }$ from $(c_s,r_{\ast },c_o)$ to each relation triplet in $T_{sim}$. Specifically, for each relation triplet $(s_j,r_j,o_j)$ in $T_{sim}$, we compute its semantic similarity to the target relation $r_{\ast }$ and generate the corresponding soft labels $r_j^{soft}$ and $r_{\ast }^{soft}$ by normalization. These two soft labels represent the similarity between $r_j$ and $r_{\ast }$, as defined in Eqs. (3) and (4):

(3) $$r_j^{soft}=\frac{A(c_s,r_j,c_o)}{A(c_s,r_j,c_o)+A(c_s,r_{\ast },c_o)}$$

(4) $$r_{\ast }^{soft}=\frac{A(c_s,r_{\ast },c_o)}{A(c_s,r_j,c_o)+A(c_s,r_{\ast },c_o)}$$

The denominator represents the total amount of information contained in the two relations, $r_{\ast }$ and $r_j$, within the same subject-object context. The resulting quotient produces a score that falls within the range of 0 to 1, reflecting their semantic similarity and differences. Higher scores indicate greater similarity, while lower scores indicate significant differences. Next, soft labels $r_{\ast }^{soft}$ and $r_j^{soft}$ are assigned to all samples in $T_{sim}$. However, not all samples receive soft labels. As the confusion matrix in Fig. 3 shows, the relation “flying in” is not incorrectly assigned to other categories. This indicates that “flying in” is distinctive enough to be clearly identifiable, thus making soft labeling unnecessary and potentially misleading for such unique cases.

Balanced relation learning

In this module, our objective is to address the challenges presented by the long-tail distribution by modifying the loss weights for each fine-grained and low-frequency relation sample. Fine-grained relations usually offer more specific and detailed information than coarse-grained relations, thus possessing greater informational value in numerous contexts. To effectively differentiate between these two types of relations and utilize this distinction to improve model performance, we set a threshold $\theta$. Soft label scores that exceed $\theta$ are considered fine-grained relations. Upon classifying a relation as fine-grained, we adjust its loss weight by applying the loss balancing hyperparameter $\alpha$, ensuring that these relations receive appropriate attention and emphasis during the model training process. During training, we adjust the cross-entropy loss to accommodate soft label training, as defined in Eq. (5):

(5) $$L_{soft}=-\sum _{i=1}^N{w}_i{r}_i^{soft}\log (p_i)$$where N denotes the total number of relation categories, $r_i^{soft}$ represents the score of the $i$-th relation category in the soft label, $p_i$ indicates the prediction probability of the $i$-th relation category, and $w_i$ is the weight assigned to each relation label. The weight $w_i$ as defined in Eq. (6):

(6) $$w_i=\{\begin{array}{cc}\alpha ,& \mathrm{i}\mathrm{f}{r}_i^{soft}\ge \theta \\ 1,& \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\end{array}.$$

In model training, low-frequency relations that appear in only a small number of samples are often neglected, which can result in these relations receiving less emphasis during the learning process. Nevertheless, these low-frequency relations may carry unique and valuable information that contributes to the model’s overall performance. In the MRL module, not all low-frequency relation samples are assigned soft labels. To ensure that all low-frequency relation samples are given adequate consideration during training, we apply the loss balancing hyperparameter $\alpha$ to adjust the loss weights for these single-label samples, as defined in Eq. (7):

(7) $$L_{single}=-\alpha \sum _{i=1}^N\log (p_i)r_i$$where $r$ adopts a one-hot representation, meaning ${\sum }_{i=1}^N{r}_i=1$ and $r_i=1$ for the correct relation category, which denotes the ground-truth relation.

In order to handle both soft-labeled and single-labeled samples effectively, we compute the total loss by combining the individual losses for each type of sample. The final total loss function as seen in Eq. (8):

(8) $$L_{total}=\frac{-{\sum }_{m_{soft}=1}^{M_{soft}}{\sum }_{i=1}^N{w}_i{r}_i^{soft}\log (p_i^{(m_{soft})})-{\sum }_{m_{single}=1}^{M_{single}}\left(\alpha {\sum }_{i=1}^N\log (p_i^{(m_{single})})r_i^{(m_{single})}\right)}{M_{soft}+M_{single}}.$$

Here, $M_{soft}$ and $M_{single}$ represent the total number of soft-labeled and single-labeled samples, respectively.

Experimental settings

Compared methods

To prove its performance, we compare it with state-of-the-art methods. These include classic feature- and relation-based models like motifs (Zellers et al., 2018) and VTransE (Zhang et al., 2017), more structurally complex approaches like Transformer (Tang et al., 2020) and VCTree (Tang et al., 2019), and knowledge-augmented models such as KERN (Chen et al., 2019). Additionally, we evaluate against recent unbiased SGG methods, including SHA+GCL (Dong et al., 2022), BGNN (Li et al., 2021), and PCPL (Yan et al., 2020), which aim to address data bias challenges and improve generalization. Given the model-agnostic nature of our framework, we further compare it with other model-agnostic methods like group collaborative learning (GCL) (Dong et al., 2022), total direct effect (TDE) (Tang et al., 2020), DLFE (Tang et al., 2020), CogTree (Yu et al., 2020), IETrans (Zhang et al., 2022), and NICE (Li et al., 2022a), to illustrate its seamless integration capability and performance improvements.

Comparison with state-of-the-art methods

Ablation studies

MBRL consists of two components: Mixup relation learning (MRL) and balanced relation learning (BRL). As shown in Table 5, we evaluate the impacts of each component of MBRL, which is based on Motifs, in the PredCls task on the VG150 dataset. From the results, we observe that MRL significantly improves the performance in terms of mR@50/100 metric and mean metrics. This demonstrates the effectiveness of MRL in accurately classifying certain coarse-grained relations into their corresponding fine-grained ones. Furthermore, BRL contributes more significantly to improvements in Tail R@100 metric compared to MRL, indicating that BRL plays a crucial role in predicting diverse tail relations. This method effectively protects the learning of tail relation samples and reduces the impact on head relation samples.

Hyperparameter analysis

Influence of $\theta$

We investigate the impact of different thresholds $\theta$, ranging from 0.75 to 1, on model performance. As shown in Fig. 4, the R@100 metric shows an increasing trend as the value of $\theta$ increases. Before $\theta$ reaches 0.95, the mR@100 metric remains relatively stable, suggesting that the model maintains consistent performance across tail relations. Once $\theta$ exceeds 0.95, the mR@100 metric significantly decreases. Therefore, based on the results of the mean metrics, we select 0.95 as the optimal threshold.

Influence of $\text{α}$

We experiment with different $\alpha$ values from 2 to 9 to assess the effect of the loss balancing hyperparameter on the model’s performance. As shown in Fig. 5, an increase in the value of $\alpha$ results in a decline in the performance of head relations, while simultaneously enhancing the performance of tail relations. Once the hyperparameter $\alpha$ exceeds 5, the model begins to overfit on tail relations, resulting in diminishing performance gains for these categories. Therefore, based on the mean metric results, the optimal value for $\alpha$ is determined to be 5.

Visualization results

To demonstrate the effectiveness of our proposed MBRL in accurately identifying relations, we visualize several PredCls examples generated from motifs (with a purple background) and motifs combined with our proposed MBRL (with a blue background) in Fig. 6. Comparing the results of the Motifs, we find that our method can detect more fine-grained relations, such as “walking on”, “eating”, “growing on”, and “laying on”. MBRL effectively mitigates ambiguity issues and reduces prediction errors in relation recognition by enabling the model to discern subtle differences among relations. Thus, over-confident predictions of head relations under a long-tail distribution can be alleviated to some extent. To illustrate the discriminatory capabilities of MBRL against semantically similar relations, we present the PredCls results of Motifs+MBRL in Fig. 7. Observations indicate that Motifs+MRL leads to enhancements in most relations. However, for challenging predictions, such as “flying in” and “mounted on”, Motifs+MRL is susceptible to errors due to the long-tail distribution. Conversely, BRL significantly bolsters the model’s ability to distinguish between fine-grained and infrequent relations. These results demonstrate that our proposed MBRL can enhance scene graph generation by generating more reasonable relations.