Knowledge enhanced bottom-up affordance grounding for robotic interaction

Language conditioned manipulation grounding

Language conditioned manipulation grounding tasks have attracted extensive attention recently. In contrast to traditional visual perception tasks with predefined object category labels, the language conditioned manipulation grounding task involves more intricate language and visual information. Some works Chen et al. (2020), Mi et al. (2020a), Nguyen et al. (2020), Shridhar et al. (2020) focus on grounding tasks for different objects, while some works Hristov et al. (2017), Ahn et al. (2018), Magassouba et al. (2019), Hatori et al., (2018), Shridhar, Mittal & Hsu (2020), Mi et al. (2020b) focus on the grounding of spatial relationships and attributes of objects. Several studies have focused on human–robot interactions with natural language. Utilizing question asking can increase accuracy and eliminate ambiguity (Hatori et al., 2018; Chen et al., 2020). In order to improve the generalization ability, some end-to-end approaches attempt to scale and broaden the data collected (Mees, Hermann & Burgard, 2022; Jang et al., 2022). According to the data types, the existing datasets in Table 1 can be divided into simulated dataset and realistic dataset. The simulated datasets are primarily employed for tasks that involve extended interactions based on natural language instructions (Shridhar et al., 2020; Mees, Borja-Diaz & Burgard, 2023), which focus on learning robotic control directly from image-instruction pairs. However, the simulated datasets have the problem of domain shifting in practical applications. In addition, most realistic datasets are manually annotated, resulting in higher costs. Automatically constructing datasets faces the challenge of lacking diverse in instructions and fine-granularity annotations for images. To address these challenges, we propose a unified framework that generates language instructions and fine-granularity image annotations simultaneously.

Object affordance detection

The concept of affordance consists of object affordance and environment affordance. In this article, we specifically focus on the visual affordance of objects in images. Myers et al. (2015) propose jointly applying geometric properties and local shapes to identify object affordance. Do, Nguyen & Reid (2018) propose AffordanceNet to simultaneously detect the affordance of multiple objects in RGB images. More recently, Mi et al. (2020a) propose an attention-based architecture to learn the affordance of objects. Zhai et al. (2022) propose OSAD-Net for detecting object affordances in a scene through collaboration learning. Zhao, Cao & Kang (2020) utilize a relationship-aware network to implement affordance segmentation in an end-to-end way. In addition, several researchers have introduced context like the object shape (Chen et al., 2024), boundary (Yin & Zhang, 2022) to improve the performance of affordance segmentation. On one hand, these methods neglect the correlation between natural language and object affordance. In our approach, we segment the corresponding affordance part, which better aligns with realistic scenarios. On the other hand, existing methods lack of the commonsense for affordance learning. In this work, we introduce external knowledge to understand the usability of objects better.

Prompt learning

Prompt learning, a recent trend in the field of few-shot learning of NLP (Jiang et al., 2020; Brown et al., 2020; Zhu et al., 2023), involves leveraging pretrained language models through cloze-style prompts to enhance the performance of downstream tasks. Brown et al. (2020) propose a method for adjusting the behavior of a frozen GPT-3 model through prompt design. In order to efficiently deploy prompt-learning pipelines for researchers and developers, Ding et al. (2021) present OpenPrompt that modularizes the entire framework of prompt-learning. In the latest research, prompt learning is applied to large vision-language models in computer vision (Du et al., 2022; Zhou et al., 2022b; Zhou et al., 2022a). CoOp (Zhou et al., 2022b) turns discrete words in a prompt into continuous prompt learning for adapting pre-trained vision-language models. In order to generalize to unseen classes, CoCoOp (Zhou et al., 2022a) generates an input-conditional prompt instead of the fixed learned vectors for each input image.

In general, to determine the verb associated with an object, the common approach is to establish a mapping between the object’s category and its corresponding verb. However, the method of dictionary mapping is a challenge for unseen objects to generate corresponding verbs. To address this issue, we utilize a large language model as a knowledge base and use prompt-learning to predict the probability distribution across the vocabulary for the <MASK> token’s position, thus obtaining the verb of the object.

Multimodel encoder

Visual encoder. Given an image I ∈ R^H×W×3 with the height H, the width W and three color channels, we employ ResNet101 (He et al., 2016) as the visual backbone, a deep residual network widely recognized for its effectiveness in various computer vision tasks. In our model, we denote res_i as the output features of the i − th block in ResNet101. Specifically, we use res_2 and res_5 to represent low-level visual features $F_v^l$ and high-level visual features $F_v^h$, respectively. The choice of these specific layers is strategic: low-level features are adept at capturing intricate details such as edges and textures, which are essential for detailed image analysis. On the other hand, high-level features excel at capturing advanced semantic information about objects, providing a broader context and understanding of the objects in the images. Our approach aims to fully leverage the strengths of both low-level and high-level visual features.

Knowledge enhanced text encoder. In our approach, we handle an expression $X=\left\{x_1,\dots ,x_m\right\}$ consisting of m words, where x_i represents the i − th word in the sequence. Our initial step involves utilizing an entity recognition algorithm to identify and extract object entities within the expression, which we denoted the entities as $E=\left\{e_1,\dots ,e_n\right\}$ with n < m. Each entity serves as a key to retrieve corresponding knowledge from the collected knowledge. The collection of knowledge will be illustrated in the experiments section.

For every identified entity e_j ∈ E in the expression X, we match it with a corresponding fact k_j in our knowledge base. Each fact k_j is composed of multiple sentences, providing a rich context for the entity. To process and extract features from both the expression X and the knowledge K, we employ BERT (Devlin et al., 2019), an effective language representation model to capture contextual features. The features extracted from the expression and the knowledge are denoted as L_X and L_K, respectively. Then, we design a bimodal language feature interaction module to align and fusion the two features. This process of knowledge selection is pivotal, as it significantly enhances the model’s capability to comprehend and interpret natural language instructions with greater semantic context.

Bimodal language feature interaction

The bimodal language feature interaction (BLFI) module is a cornerstone of our approach, designed to enhance the understanding of instructions by focusing on two key aspects: (a) emphasizing key information within L_X and L_K, and (b) filtering out irrelevant information from these features. This dual focus ensures that our model captures the most significant elements of the language and knowledge features. As depicted in Fig. 4, the BLFI module operates in two distinct stages: knowledge alignment and knowledge fusion.

Bottom-up language-guided multi-modal fusion

Our model employs a bottom-up language-guided multi-model fusion (BULG) module, an innovative approach to integrative visual features with knowledge-enhanced text features. Traditional multimodal methods often rely on attention mechanisms to fuse text and visual features. These methods, as illustrated in Figs. 5A, 5B, 5C, typically focus on object-level alignment using text features that contain limited information. However, in our approach, the knowledge-enhanced text features are rich in semantic content, playing a pivotal role in part-level affordance grounding.

Inspired by structure (c) from Fig. 5 and the work on attention mechanism (Vaswani et al., 2017), our novel BULG structure addresses two critical factors. (1) Enhanced role of text information: We employ multi-layer self-attention mechanisms to iteratively extract information from text features. This method allows for a deeper understanding of the text content, leveraging its rich semantic nature. (2) Fine-grained multimodal alignment: Given the details characteristic of the task, our model requires precise alignments between text and visual features. This alignment is crucial for accurately capturing the nuanced interplay between different modalities.

In the BULG module, illustrated in Fig. 6, we combine a intra-modal interaction module and a cross-modal interaction module. The right part of the module takes text features F_text as input, employing N layers of self-attention to generate refined text features f_text. Simultaneously, in the left part, the input comprises f_text and low-level visual features $F_v^l$. Initially, $F_v^l$ undergoes unimodal interaction to extract critical visual features from $f_v^l$. These features are then combined with f_text in the multimodal interaction module, employing a bottom-up guided attention mechanism to enhance visual features learning. This process results in the generation of multimodal features f_m1.

For the fusion process, we handle text and visual features distinctly based on their level. With low-level visual features $F_v^l$, we apply a bottom-up text-guided approach to form f_m1. High-level visual features $F_v^h$, containing rich object-level semantic information, are directly fused with text features to produce multimodal features f_m2. This process is described mathematically as: (4)${\displaystyle f_{m2}=Concat\left(g\left(F_v^h\cdot W_v\right)\cdot g\left(F_{text}\cdot W_t\right)\right)\text{.}}$ Here, W_v and W_t are two transformation matrices that align the visual and text representations into a uniform feature dimension, facilitating effective fusion. ‘Concat’ is the concatenation operation. g denotes the LeakyRelu activation function.

Part-level affordance grounding

In this subsection, we detail how our model leverages text-guided multimodal features to detect various affordances in images.

Our process begins with the integration of multimodal feature f_m1 and f_m2.f_m1 is the multi-model feature from bottom-up language-guided multi-modal block. The first step involves passing f_m1 through an ASPP module, as described in the DeepLab framework by Chen et al. (2017). ASPP is effective in capturing multi-scale contextual information, which is crucial for affordance understanding in complex images. The output feature from ASPP is denoted as $f_{m1}^A$ and is calculated as follows: (5)${\displaystyle f_{m1}^A=ASPP\left(f_{m1}\right)\text{.}}$

Next, we concatenate $f_{m1}^A$ with f_m2 and pass this combined feature through our segmentation module. This process yields the final segmentation result, represented as: (6)${\displaystyle f_{mask}=unsample\left(Concat\left(unsample\left(f_{m1}^A\right),f_{m2}\right)\right)\text{.}}$

By employing filters with various sampling rates and fields of view, ASPP allows the model to capture both detailed and broader contextual elements in images. To achieve more precise segmentation results, we upsample the feature map by a factor of four, ensuring a finer granularity in the final output.

Loss function

For training our model, we utilize the Sigmoid Binary Cross Entropy (BCE) (Mao, Mohri & Zhong, 2023) loss function. This loss function is particularly suited for binary classification tasks, such as segmenting specific affordances in images. The BCE loss for our task is defined as: (7)${\displaystyle L=\sum _{l=1}^{H\times W}\left[y_{l}log\left(p_l\right)+\left(1-y_l\right)log\left(1-p_l\right)\right]\text{,}}$ where y_l and p_l are the l-th elements of the ground-truth mask and predicted mask f_mask, respectively. Here, W and H denote the width and height of the image respectively.

Notation and formulation

Given an unlabeled image I, our goal is to automatically generate the annotation A for the image. Specifically, each A = {L, B, M} comprises a language instruction L referring to a candidate object, the corresponding object location B and the affordance mask M of the object part. The affordance mask M is a mask of the object part that is related to the manipulation verb in the instruction, such as “contain“ and “cut”. Importantly, a candidate object may have both bounding box and mask annotations within our dataset generation process. The overall framework, as illustrated in Fig. 7, comprises three key components, each contributing to the automatic annotation process:

First, the object detection and re-correction module takes an unlabeled image as input, utilizes an object detector to predict the bounding box coordinate, and object regions accordingly. The cropped regions are then fed into the CLIP (Radford et al., 2021), which generates possible class labels for the cropped regions. The results of the object detector and the CLIP are compared to filter the wrong predictions. Then, the prompt-based instruction generation module automatically generates positional relationships and predicts verbs related to the candidate objects. Finally, the object affordance mask generation module is responsible for producing affordance masks for target objects using DeepLabV3+ (Chen et al., 2018). We provide detailed implementation insights in the following subsections.

Object detection and re-correction module

In this subsection, we delve into the object detection and re-correction module, a critical component of our automatic data annotation process. Its primary objective is to enhance the quality of object class labels obtained from an initial object detector, such as Faster RCNN, and to mitigate issues like incorrect category predictions and the presence of redundant bounding boxes. To achieve this, we leverage the power of the Visual and Language model CLIP (Radford et al., 2021), which has demonstrated state-of-the-art capabilities in understanding visual concepts from a vast corpus of image-text pairs. CLIP comprises two encoders, one for text and another for images. We employ a specific input template, “This is a photo of a {label}”, for the text encoder, where {label} represents the object category outputted by the object detector. For the image encoder, we use the cropped region as input. These two encoders result in feature vectors f_t and f_v, respectively. At last, the CLIP adopts a dot product to compute the similarities between two features (f_t and f_v) from each modality, followed by Softmax-based probability predictions for all object categories.

Let p_i denote the probability of a region proposal being predicted as the ith object category by CLIP, which is computed as Eq. (8). In the equation, cos(⋅, ⋅) denotes cosine similarity and τ is a temperature parameter learned by CLIP during training. max(p_i) is the maximum of the prediction possibilities. (8)${\displaystyle p_i=\frac{exp\left(\frac{cos\left(f_t,f_v\right)}{\tau }\right)}{\sum _{j=1}^{K}exp\left(\frac{cos\left(f_t,f_v\right)}{\tau }\right)}\text{.}}$

When the object detector and CLIP both predict the object as the ith category, we first compute the average score S_i of the object detector and the CLIP follow Eq. (9): (9)${\displaystyle S_i=\frac{r_i+max\left(p_i\right)}{2}\text{.}}$

To enhance the quality of category labels, we introduce a score S_i, computed as the average of the probability score from the object detector (r_i) and the maximum CLIP prediction probability (max(p_i)), as expressed in Eq. (9). Then, the average score S_i is compared with a score threshold S_T. This threshold filtering helps eliminate predictions with low confidence, ensuring that only high-confidence category predictions are considered. Additionally, we address cases where the object detector and CLIP provide conflicting category predictions. In such situations, we prioritize the prediction from CLIP to improve label accuracy. Predictions from the object detector that do not align with CLIP predictions are discarded, further refining the category labels. Figure 7A provides a visual representation of the process.

Prompt-based instruction generation module

This module is responsible for generating manipulation instructions, which can be categorized into two types: explicit and implicit instructions. The explicit instructions contain specific details, including the object category, verb, and positional relationship, providing comprehensive guidance for manipulation tasks. For example, “Hand me the cup on the right to contain”. The implicit instructions focus primarily on the manipulation verb itself, omitting specific object category references. An example of an implicit instruction is “Hand me something to contain”, where the object category is not explicitly mentioned.

Generating explicit instructions requires careful consideration of both positional relationships and appropriate verbs to convey the desired manipulation task. To achieve this, we employ a multi-step approach, as shown in Fig. 7B:

Positional relationship analysis: Distinguishing among multiple objects of the same class is challenging. We address this issue by analyzing spatial relationships in both horizontal (left, middle and right) and vertical (top and bottom) dimensions. By calculating the center coordinates of bounding boxes obtained from the object detection and re-correction module, the position relationship is determined by comparing the center coordinates. These offsets help us construct instructions such as “Bring the cup on the left to contain” and “I want the bowl on the top to contain”.

Prompt construction: Prompt-learning uses pre-trained language models as knowledge bases resulting in great performance on different downstream tasks. Using prompt-learning can automate the process of finding the object-verb pair relationship. Denote x_inp as the original input and x_prompt as the prompt that is fed into the MLM. The mapping from x_inp to x_prompt is performed using a template m. This template defines where each input sequence and the placement will be placed in the prompt. The template used in our work is “{“placeholder”: Text}. In other words, give me something to [MASK]. ”. The Text contains x_inp as shown in Table 2. Feeding the prompt into the MLM will produce a probability distribution p([MASK]|x_prompt), which predicts the token most likely filled in the blank. The x_inp is the output of the object detection and re-correction module. We use GPT-2 (Radford et al., 2019) as a pre-training language model and adopt OpenPrompt (Ding et al., 2021), an open-source and unified easy-to-use toolkit, to implement prompt-learning.

Prompt-based instructions generation: To ensure diversity in our instructions, we generate explicit and implicit instructions simultaneously. As shown in Table 2, we start by randomly selecting an explicit template and construct a Text from x_inp. The Text is then mapped to x_prompt, which is used as input to the MLM. The MLM outputs a probability distribution p([MASK]|x_prompt) describing which verb most likely fills in the mask. The verb with the highest probability will be added to the end of the explicit instruction and also serves as the basis for the implicit instruction.

Object affordance masks generation module

After object detection and prompt-based instruction generation, we take a crucial step towards enhancing the realism and applicability of our generated dataset. Existing datasets typically provide object bounding boxes for each instruction, which may not suffice for tasks requiring part-level instruction grounding. Consider an instruction such as “Use the cup to contain”. In such cases, the robot agent needs to focus on the specific part of the object that provides the “contain” affordance. To address this need, our goal is to provide not only the object bounding box but also the affordance mask for each instruction. This addition enhances the practicality of the generated dataset and its suitability for a wider range of real-world applications. Our approach involves the following key steps, as shown in Fig. 7C.

Object region cropping: For each instruction, we begin by cropping the object region I_o from the original image. This isolated region serves as the basis for affordance mask prediction.

Affordance mask prediction: We employ the DeepLabV3+ (Chen et al., 2018) segmentation model, fine-tuned on an affordance dataset, to predict the affordance category for each pixel within the object region. This prediction results in an object affordance mask M_o. Importantly, since an object may possess multiple affordance classes, the pixel values in the affordance mask M_o are assigned as n unique labels {0, 1, …, n − 1}. If the pixel values correspond to the affordance class specified in the instruction, they are kept and set as ones. Otherwise, they are set to zeros.

Remapping of the affordance mask: To align the object affordance mask M_o with the coordinates of the object region, we remapped it to an initial affordance mask that is initialized as zeros and shares the same size as the original image. The outcome of this process is the pixel-wise segmentation of the object part M corresponding to the instruction. This segmentation provides crucial information about which part of the object is relevant to the instruction, facilitating precise instruction grounding.

Experiments setup for affordance grounding

Before evaluating the semi-automatic data construction, it is crucial to first validate the effectiveness of KBAG-Net using manually annotated datasets. This initial step ensures that the performance measurements are not adversely influenced by any noise potentially introduced in the semi-automatically generated data. For this purpose, we utilized two well-established datasets: IIT-AFF VL and UMD VL (Qu et al., 2024). We first introduce the datasets in details, then present the evaluation metrics for the affordance grounding task. At last, the knowledge collection process is illustrated.

Datasets For a comprehensive evaluation of our method, we utilized two visual language datasets IIT-AFF VL and UMD VL dataset (https://github.com/WenQu-NEU/Affordance-Grounding-Dataset). The IIT-AFF VL and UM VL datasets comprises a total of 15, 905 language instruction expressions, each paired with a corresponding RGB image (images within original IIT-AFF (Nguyen et al., 2017) (https://sites.google.com/site/iitaffdataset/) and UMD (Myers et al., 2015) (https://users.umiacs.umd.edu/ fer/affordance/part-affordance-dataset/index.html), thereby ensuring a one-to-one mapping between images and instructions. Following the IIT-AFF dataset, the IIT-AFF V-L dataset includes ten object categories and nine affordance classes. The dataset contains 24, 677 affordance regions at the pixel level. The UMD VL dataset comprise of seven affordance classes and 17 object categories. The two datasets are split into training (80%), testing (10%) and validation (10%). The experiments following the dataset split ensuring a comprehensive and fair evaluation for the compared methods.

Evaluation metric To assess the performance of our method, we employed two evaluation metrics: structural similarity (denoted as $F_{\beta }^{\omega }$) (Margolin, Zelnik-Manor & Tal, 2014) and Mean Intersection-over-Union (mIoU). The $F_{\beta }^{\omega }$ evaluates the agreement between model’s predictions and ground truth and provides insights into the precision in delineating object boundaries. The mIoU calculates the intersection regions over union regions of the predicted segmentation mask and the ground truth, providing a quantitative measure of the accuracy in terms of area overlap.

Knowledge collection Addressing the challenge of enabling models to comprehend natural language instructions, particularly in understanding common sense and contextual information, is pivotal in our approach. For instance, interpreting an instruction like ‘Use the knife to cut some apple’ requires not just recognizing objects in images but also understanding their affordance parts based on common knowledge, a capability that existing methods in affordance detection and visual grounding often lack.

To bridge this gap, we collect object and affordance-related knowledge from three comprehensive knowledge bases: ConceptNet (Speer, Chin & Havasi, 2017) is instrumental in providing relational knowledge in a triadic form. This structure is particularly useful for understanding relationships between objects and their affordances. WebChild (Tandon, De Melo & Weikum, 2017) offers detailed commonsense knowledge extracted from language usage across major networks. It enriches our model with nuanced insights into everyday object functions and uses. With its vast and diverse content, Wikipedia serves as a rich resource for knowledge. It provides extensive information on a wide range of object types and their associated affordances.

Our collection process involves several steps: (a) We began by gathering facts for each object category. This involved matching the starting or ending nodes in ConceptNet and WebChild with corresponding category labels and Wikipedia articles. (b) To standardize the information, we converted data from ConceptNet and WebChild into sentences, treating each sentence in a Wikipedia article as a standalone fact.(c) We adopted a key-value pair format for the knowledge base, where the key represents an object and the value details facts about the object, including its function, shape, and other relevant attributes. Figure 8 showcases a subset of the knowledge we have accumulated.

In our model, we opted for unstructured knowledge due to its flexibility and ease of updating and expansion, as opposed to the rigid structure of conventional databases. Furthermore, unstructured knowledge more closely mirrors the natural language expression, making it more suitable for our purposes. To ensure the quality and accuracy of the collected knowledge, we conducted manual checks to eliminate biases and redundant information. The final composition of our knowledge base includes contributions from Wikipedia (59.87%), WebChild (27.88%), and ConceptNet (12.25%).

Quantitative and qualitative results (RQ1)

Compared method. In order to validate the effectiveness of our model, we conducted comparisons with eight established methods. These comparisons are detailed in Tables 3 and 4. To ensure a fair and unbiased comparison, all models were trained from scratch with the same sample size and the number of classes. This approach guarantees that any observed performance differences are attributable solely to the distinct model architectures and training methodologies.

• DeepLabV3+ (Chen et al., 2018) employs dilate convolutions and an encoder–decoder structure to achieve fine-grained segmentation results through multi-scale feature fusion and spatial pyramid pooling.

• OCRNet (Yuan, Chen & Wang, 2020) introduces object-contextual representations in conjunction with attention mechanisms. Since affordance segmentation is closely related to object classes, we set OCRNet an appropriate image-only baseline for comparison.

• AffordanceNet (Do, Nguyen & Reid, 2018) is a classical affordance detection method. It segments images based on affordance categories without the integration of language instructions, providing a language-agnostic baseline.

• BPN (Yin & Zhang, 2022) proposes a boundary-preserving network which considers the relationship between object categories and object affordances. It is an image-only method for affordance detection. Because this dataset is a closed-source dataset, we employed the results of the IIT-AFF dataset reported in the original articles.

• GSE (Zhang et al., 2022) utilizes a repeated multi-scale feature-map-fusion network to produce category-relevant feature maps. Because this dataset is a closed-source dataset, we employed the results of the IIT-AFF dataset reported in the original articles.

• ADOSMNet (Chen et al., 2024) is the state-of-the-art affordance detection method. We considered it an upper bound in our comparisons since it is an image-only method which eliminates the need to interpret language. We employed the results (with ResNet101) of the IIT-AFF dataset reported in the original articles. Unfortunately, BPN, GSE and ADOSMNet’s performance on the UMD VL dataset could not be compared because the UMD vision language dataset delete the repeated images in the UMD dataset to avoid data bias.

• Two-stage method: We design a two-stage approach to integrate Fast R-CNN (Ren et al., 2015) and DeepLabV3+ as a baseline method. Firstly, Fast R-CNN is used to detect and crop the target objects. Subsequently, the cropped regions are input into DeepLabV3+ to generate affordance segmentation masks for the targets. This approach can provide deep insights into the impact of language composition in the affordance grounding task.

• BKINet (Ding et al., 2023) represents the state-of-the-art multi-modal method with knowledge incorporation. It is designed for the referring image segmentation task, incorporating specific knowledge of the target object in the image.

The image-only methods provide an access to the impact of language instructions. The two-stage method and multi-modal methods evaluate how different integration of text and image data affect the performance for affordance grounding. Different kinds of methods provide comprehensively evaluate of our method for language-following affordance segmentation and multimodal understanding.

Quantitative evaluation. Table 3 summarizes the results of the IIT-AFF VL dataset. The results clearly show that KBAG-Net has achieved substantial improvements. Compared to DeepLabV3+, our approach exhibits a notable increase in performance, demonstrating a gain of +2.9% in mIoU and +2.35% in $F_{\beta }^{\omega }$, respectively. Against OCRNet, our method shows superior results with increases of +1.72% in mIoU and +1.64% in $F_{\beta }^{\omega }$. It is noteworthy that KBAG-Net achieves these results using an end-to-end architecture, and no further post-processing step like object detection and crop as the two-stage method. Our approach displays a significant enhancement over two-stage method with improvements of +12.98% in mIoU and +8.17% in $F_{\beta }^{\omega }$. While our $F_{\beta }^{\omega }$ scores are slightly lower than those of KBAG-Net, our model surpasses KBAG-Net in mIoU by a significant margin of +17.26%. The higher mIoU with lower F-score could be due to imbalanced class distributions. The discrepancy of $F_{\beta }^{\omega }$ scores may be attributed to our method’s handling of certain affordance classes, notably “grasp” and “hit”. These classes often present significant challenges in accurately delineating edge regions, which is crucial for achieving high $F_{\beta }^{\omega }$ scores. Therefore, the model has high precision but low recall for certain affordance classes, leading to a lower average F-score. Overall, our method has a stronger performance in terms of overall segmentation accuracy.

Table 4 provides a comprehensive summary of our model’s performance on the UMD VL dataset. In these experiments, KBAG-Net consistently outperforms other comparative methods, affirming its robustness and effectiveness. Specifically, our model surpasses the second best (OCRNet) +1.02% in mIoU and +1.14% $F_{\beta }^{\omega }$. It is worth noting that the UMD dataset only contains clutter-free scenes. Therefore, the improvement of KBAG-Net over compared methods is not as significant as those observed on the IIT-AFF dataset.

In conclusion, the results from our extensive experiments, as detailed in Table 3 and Table 4, clearly demonstrate that our KBAG-Net method marks a significant advancement over existing technologies in the realm of fine-grained part-level affordance segmentation. A key strength of our approach is its ability to operate in an end-to-end manner, effectively eliminating the need for any additional post-processing or data augmentation steps. This streamlined process not only simplifies the workflow but also contributes to the robustness of the results. The performance across different affordance categories, as reflected in the mIoU and $F_{\beta }^{\omega }$ scores, reveals insightful trends. The “grasp” category presents a more challenging scenario. This is primarily due to the complexity of scenes and the irregular shapes of objects requiring grasping, which inherently lead to greater difficulties in achieving precise segmentation.

Qualitative results. To further validate the effectiveness of our proposed module in identifying the object’s graspable region and generating accurate mask predictions, we showcase a selection of results in Fig. 9. These visual representations show that our method can effectively detect part-level regions of masks according to the language, and the predicted results are close to the ground truth.

Ablation studies (RQ2)

In order to comprehensively understand the contribution of each component within our framework, we conducted a series of ablation studies. These studies are essential for isolating and evaluating the impact of individual elements of our model on its overall performance.

The components of the network. In validate the effectiveness of the BULG module, Knowledge, and BLFI module, we conducted ablation studies on IIT-AFF VL dataset, with the results presented in Table 5. The notation ‘w/KB’ indicates the incorporation of the knowledge information. The notations ‘w/BLFI’ and ‘w/BULG’ denote the addition of the Bimodal Language Feature Interaction and Bottom-up Language-guided Multi-modal Fusion, respectively. The ‘Attention’ column refers to the three types of typical multimodal attention mechanism displayed in Fig. 5. The baseline method (first row of Table 5) combines image features with text features via simple concatenation. The ablation studies reveal significant findings:

(1) Knowledge based only (second row of Table 5): The introduction of external knowledge alone enhances the model’s understanding of natural language instructions, evidenced by increases in mIoU and $F_{\beta }^{\omega }$ by 1.24% and 0.37%, respectively, compared to the baseline.

(2) Knowledge with BLFI module (fourth row of Table 5): The incorporation of the BLFI module to filtering text information noise results in further improvements in text features, with subsequent increases in mIoU and $F_{\beta }^{\omega }$.

(3) Knowledge with BULG module (fifth row of Table 5): The fusion of knowledge information with textual feature, combined with the BULG module’s dense multimodal information interaction, significantly aligns visual and text features. This configuration shows a substantial improvement over the baseline, with mIoU and $F_{\beta }^{\omega }$ increasing by 2.33% and 1.32%, respectively.

(4) Full integration (last line of Table 5): Utilizing all three components simultaneously results in the most significant performance enhancements, with increase of 6.51% in mIoU and 1.99% in $F_{\beta }^{\omega }$ compared to the baseline.

Further evaluations were conducted to assess different attention mechanisms for multimodal fusion, with results from the sixth to ninth rows of the Table 4 illustrating the effectiveness of replacing the attention mechanism in multimodal fusion with three alternative mechanisms as shown in Fig. 5. The outcomes confirm the effectiveness of the proposed BULG module.

Model efficiency analysis. As shown in Table 6, we calculate the number of parameters and FPS for comparison methods. Our method achieves the best performance in terms of mIoU while utilizing fewer parameters compared to two other multimodal methods. Compared to DeepLabV3+, our model has demonstrated a significant improvement in terms of computational overhead, processing speed, and performance, with enhancements of +2.9% and +2.35% on IIT-AFF VL dataset, respectively. Compared to the two-stage methods, we achieved better performance and processing speed at a relatively lower computational overhead, resulting in increases of +12.98% and +8.17% on IIT-AFF. Although our computational overhead and processing speed do not match the levels of BKINet, our performance has shown improvements of +17.26% on the mIoU metric. The total number of parameters in our CNN model is less than half that of the BKINet.

Data quality and efficiency analysis (RQ3 and RQ4)

To evaluate the proposed semi-automatic method whether generate high-quality dataset, we used the framework to generate two datasets, which is used to validate the quality. We conducted experiments on two publicly available datasets: IIT-AFF dataset (Nguyen et al., 2017) and UMD dataset (Myers et al., 2015), which have ground truths of object detection and affordance segmentation. We add manipulation instructions for part of the dataset manually, and compare the generated annotations (instructions, object bounding boxes and affordance masks) with the ground-truth to evaluate the quality of the generated data by our method.

Generated data analysis. Some examples of generated data are shown in Fig. 10. The left examples are generated from the IIT-AFF dataset and the right examples are generated from the UMD dataset. For the instructions, we mixed the generated instructions and manual annotations together, and selected five volunteers to choose the generated instructions from the mixed instructions. The test shows that it is difficult to distinguish the generated instructions from the manual annotations. For the quality of object bounding boxes, we evaluate the performance based on the object detection mean average precision (mAP). The mAP achieved on the IIT-AFF dataset is 80%, and that on the UMD dataset is 99%. The results show that our method can generate both object-level and part-level annotation for implicit and explicit instructions automatically, which improves the data annotation efficiency for the manipulation grounding task. The false annotation in the generated data could be further removed by humans, which is much easier than data labeling.

Data construction efficiency analysis. For the IIT-AFF and UMD datasets, we generated two types of instructions for each object class. Table 7 presents the statistical information of the vision-language dataset generated based on the IIT-AFF and the UMD datasets. We list the explicit and implicit instructions for the corresponding object classes. To generate the object bounding boxes and the instructions, it takes approximately 75 hours for the two datasets. The affordance masks generation takes approximately 9 and a half hours. According to prior study Papadopoulos et al. (2017), skilled annotators may need an average of 79s to label a polygon-based instance mask for an image in MS COCO. Compared with human annotation, our method could save considerable time and reduce manual annotation efforts for dataset construction.

Organization and application of the datasets. We use a structured JSON file format to store these data, ensuring that they are easily accessible and well-organized for various applications. Each entry in the JSON file corresponds to an image and contains a set of key properties related to the detected objects in that image. These properties include box, explicit_sentence, guid, implicit_sentence, label. ’guid’ represents the object id number for each annotation. The specific format of this JSON structure is illustrated in Fig. 11, providing a visual reference for clarity.

Additionally, we employ a naming convention for object part-level masks to ensure consistency and easy retrieval. These mask filenames follow a specific format that includes the RGB name, index references, and object-specific information. For example, “ILSVRC2014_train_00059832_0_0.png” has “ILSVRC2014_train_00059832” as the RGB name, “first-index” corresponds to the index of the value array in the JSON file corresponding to the image name, and “second-index” represents the index of the object’s explicit instruction array. This structured organization and naming convention enhance the usability of our dataset, making it well-suited for a wide range of robotic manipulation and language-grounding tasks.

Using object visual grounding as a test, we evaluate the performance of the TransVG (Deng et al., 2021), which is an end-to-end visual grounding model, on the IIT-AFF and UMD datasets with generated instruction(Using ground-truth image annotation). The TransVG predicts the object bounding box according to the image and language. As shown in Table 8, we assess its accuracy in two scenarios: using explicit and implicit instruction as language expressions for visual grounding. The accuracy percentages refer to the localization accuracy of object bounding boxes. For the TransVG model, we utilize both ResNet50 and ResNet101 as convolutional backbone networks. Under the ResNet-101 backbone, TransVG achieves accuracy rates of 88.00% and 99.86% on the generated IIT-AFF and UMD datasets based on explicit instructions, respectively. However, when it comes to implicit instruction, TransVG achieves 75.14% and 99.76% with the same backbone. The high accuracy achieved on the UMD dataset can be attributed to the dataset’s unique characteristics, where each image contains only one object, captured from various angles, making object localization relatively straightforward.

The results illustrate that the difficulty of the benchmark dataset is determined by at least two aspects: the diversity of the objects appearance and the complexity of the language instructions. Since the object in the UMD dataset is lack of variation, the complexity of instructions has limited influence for the performance(0.11 and 0.1 for two backbones). In contrast, the images in the IIT-AFF dataset have more diversity in the object appearance, the performance drop about 12.32 and 12.86 for the implicit instructions with two different backbones. Compared with existing datasets of visual grounding, our generated language-grounded manipulation dataset (IIT-AFF implicit) demonstrates a level of difficulty similar to that of Flickr30K Entities (Plummer et al., 2015) and presents a more challenging task than RefCOCO (Yu et al., 2016). Besides the REC task, the generated dataset can also be applied to the RIS task. In future work, we plan to assess the suitability of existing RIS methods on our generated language following manipulation datasets. Additionally, we anticipate that our dataset may have broader applications beyond the REC and RIS tasks, which we will explore in subsequent research.