Text-guided RGB-P grasp generation

Grasp detection

In this section, we will focus on the analysis of grasping methods, including 2D planar grasping and 6-DoF grasping, along with recent prominent research in this area. 2D planar grasping simplifies the task by restricting the gripper to move perpendicular to the camera plane. The oriented rectangle becomes a powerful tool to present the grasp in this plane, conveying essential information about the grasp position, orientation, and gripper opening. Jiang, Moseson & Saxena (2011), combining the “grasping rectangle” representation with a two-step learning algorithm, offer a significant advancement in robot grasping capabilities for novel objects. Lenz, Lee & Saxena (2015) demonstrate that deep learning, coupled with a novel multimodal regularization technique, provides a powerful and effective approach for robotic grasp detection in real-world scenarios. Chu, Xu & Vela (2018) proposed combining grasp region proposals with classification-based orientation estimation, leading to robust and efficient grasp detection. Guo et al. (2017) propose a novel hybrid deep learning architecture for robotic grasp detection that integrates visual and tactile sensing data. Some other notable studies include (Watson, Hughes & Iida, 2017; Wang et al., 2016; Morrison, Corke & Leitner, 2018). However, grasp generation is limited to the 2D plane due to the limited degrees of freedom in grasping poses, causing important grasping poses to be overlooked.

On the other hand, Six-Degrees of Freedom (6-DoF) grasping capability demonstrates the robot’s finesse and flexibility in manipulating and grasping objects, similar to the human hand. It allows grasping objects from a variety of angles and directions, providing maximum flexibility in complex situations (Chen et al., 2023; Zhai et al., 2023; Mousavian, Eppner & Fox, 2019; Sundermeyer et al., 2021; Hoang, Stork & Stoyanov, 2022). Chen et al. (2023) present a highly efficient and accurate solution for 6-DoF grasp pose detection. By leveraging heatmaps for guidance and incorporating semantic information, it effectively addresses the challenges of grasping in cluttered scenes. Mousavian, Eppner & Fox (2019) proposed a new method called 6-DoF GraspNet to generate diverse and successful grasps for unknown objects. This method consists of two approaches including using a variational autoencoder (VAE) to generate diverse grasp poses from the object’s point cloud, and using a neural network to evaluate the grasp ability, assigning a probability of success to each pose. Contact-GraspNet (Sundermeyer et al., 2021) is an end-to-end neural network to generate 6-DoF grasp poses for unknown objects in cluttered scenes, directly from a depth image. Hoang, Stork & Stoyanov (2022) took advantage of a voting mechanism and contextual information to directly generate grasp configurations from 3D point clouds, solving common occlusion challenges in manipulation.

Although the above studies have achieved good results, there are still some problems. The accurate generation of 6-DoF grasps often requires geometric information, making many existing methods dependent on 3D point cloud data. In addition, RGB information can provide additional contextual information about the texture, color, and shape of the object, which can improve the robustness and reliability of grasp detection algorithms. Contributing to the solution of the 6-DoF problem, our previous work (Xuan Tan et al., 2024) proposes a novel deep learning framework named Attention-based Grasp Detection with Monocular Depth Estimation for 6-DoF grasp generation using only an RGB image as input. Zhai et al. (2023) propose MonoGraspNet, the first deep learning pipeline for 6-DoF robotic grasping using only a single RGB image. This is in contrast to previous methods that heavily rely on depth sensors, which perform poorly on objects with challenging photometric properties such as transparency or reflectivity. However, some limitations, such as relying solely on visual data, lead to overlooking an important aspect of human-robot interaction. New dataset such as Vuong et al. (2024), which combine visual and textual data, show that combining text can help to create better grasps. Our method seamlessly integrates 3D point cloud data (shape), RGB images (color and texture), and text descriptions of objects into a unified “RGB-P”. Specifically, we use a state-of-the-art LLM to capture and model the grasp generation process, enhancing human-robot interactions. This method enables the generation of object-specific, object-centered grasps based on text descriptions, providing greater accuracy and flexibility in real-world scenarios.

LLMs-large language models

Natural language processing (NLP) was born as an ambitious human effort to bridge the gap between two seemingly separate worlds, the world of natural language and the digital world. Currently, its application in everyday life is increasing due to its practical benefits in life thanks to its large language model. Studies that have had a major impact on development, such as Vaswani et al. (2017) introduced the Transformer architecture, which has become the backbone of numerous state-of-the-art models in NLP and beyond, including BERT and GPT-3. Peters et al. (2018) proposed ELMo (Embeddings from Language Models), an innovative method to generate deep, contextualized word representations. ELMo employs a bidirectional long short-term memory language model (biLSTM), pre-trained on an extensive text corpus, to capture intricate word characteristics (syntax and semantics) and their variations across different contexts (polysemy). Radford & Narasimhan (2018) developed a semi-supervised approach to language understanding tasks, focus on the use of a generative pre-trained language model for various downstream applications. Devlin et al. (2019b) introduced Bidirectional Encoder Representations from Transformers (BERT), a groundbreaking technique for pre-training language representations that addresses the limitations of earlier models like ELMo and OpenAI GPT by enabling deep bidirectional training. The emergence of LLMs such as Gemini (Team et al., 2024) developed by Google DeepMind represents a new era in natural language processing. As a next-generation multimodal LLM, Gemini stands out with its capability to process ultra-long contexts of up to 10 million tokens and retrieve information with exceptional accuracy (>99%) across diverse data modalities, including text, video, and audio. Alongside other cutting-edge LLMs (Brown et al., 2020; Touvron et al., 2023; Chowdhery et al., 2022; Achiam et al., 2023), these advances have marked a major leap forward in the field, allowing the generation of creative and high-quality content across a wide range of genres, from poetry, short stories, and screenplays to articles, emails, and social media post, with a level of fluency and coherence that closely mirrors human writing.

LLM-based robotic grasping

Recent advancements in LLMs have significantly influenced robotic grasping by enabling semantic understanding and task-oriented manipulation. In which Jin et al. (2024) proposed a new task called reasoning grasping, where a robot performs reasoning to determine grasping poses based on implicit instructions. To accomplish this task, they developed a system that integrates a multimodal large language model—specifically LLaVA—that enables simultaneous processing of text and RGB image data to extract information for object recognition and manipulation. However, using only RGB and text data as model inputs have certain limitations. Specifically, in scenarios with multiple overlapping objects, RGB images may not provide sufficient information because the target object is occluded. This leads to conflicts between the image and text description, causing the linguistic information to no longer accurately reflect the characteristics of the object to be manipulated. Wang et al. (2024) proposed the Polaris framework, using the power of GPT-4 to analyze natural language instructions and extract target queries. These queries are then integrated with the RGB-D image data through a vision module, enabling the system to perform robot control tasks, such as object grasping, effectively. Meanwhile, Tang et al. (2023) introduced an innovative modular architecture for task-oriented grasping (TOG). This framework uses a LLM to generate open-ended semantic descriptions, allowing the model to generalize in a zero-shot manner to objects and tasks not present in the training set. GraspGPT serves as a grasp pose evaluator, receiving grasp candidates from Contact-GraspNet, a pre-trained sampling module. However, a notable limitation of both approaches is their complete reliance on linguistic input without using visual information during the query semantics determination phase. This leads to difficulties in handling referentially ambiguous instructions. For example, in a scenario with two cups of different colors (one blue and one white) on a table, a simple instruction like “pick up the cup” may prevent the model from identifying the target object. This ambiguity reduces the accuracy and automation level of the system, particularly in real-world environments with multiple similar objects. Xu et al. (2023) presented a method integrating vision, language, and action to perform target object grasping in cluttered environments, using deep reinforcement learning with the Soft Actor-Critic algorithm and pre-trained models like CLIP and GraspNet. However, this method has limitations in handling complex linguistic instructions. The system relies on fixed language templates such as “Give me the {keyword}” or “Grasp a {keyword} object,” reducing flexibility when processing diverse sentence structures or free-form language in real-world settings. Our work addresses these limitations by using Gemini 2.0 to generate detailed 3D object descriptions and the BERT encoder to extract robust features from the object descriptions.

Stable diffusion

Stable Diffusion is a powerful text-to-image generation model (Rombach et al., 2022) that forms the core component of our automated dataset generation process. We leverage the SDXL model (Rombach et al., 2022) as the foundation, enhancing it with low-rank adaptation (LoRA) (Hu et al., 2021) to fine-tune its performance for our specific application. This adaptation process allows us to tailor the model’s capabilities to generate images that are particularly relevant and useful for training our grasp generation model. The text prompts generated by Gemini (Gemini 2.0) (Team et al., 2024) serve as the input to Stable Diffusion, guiding the model in creating high-quality, realistic images that depict the specified scene. These images capture a wide range of variations in object arrangements, lighting conditions, and perspectives, ensuring that our training dataset is rich and diverse. This diversity is essential for our grasp generation model to learn to handle real-world scenarios where objects may be positioned in various ways, under different lighting conditions, and from different viewpoints. The emergence of text-to-image generation models has revolutionized computer vision and artificial intelligence. Models like DALL-E2 (Ramesh et al., 2022) and Imagen (Saharia et al., 2022) have also demonstrated remarkable abilities in creating photorealistic images from textual descriptions. However, Stable Diffusion stands out for its open-source nature and flexibility, making it particularly suitable for our application. The open-source nature of Stable Diffusion allows researchers and developers to freely experiment with the model and adapt it to their specific needs. Its flexibility enables us to customize the generation process by controlling various parameters, including the image resolution, the number of steps in the diffusion process, and the specific prompt engineering techniques employed. In contrast to other text-to-image models that rely on proprietary systems, Stable Diffusion allows us to fine-tune and control the dataset generation process to ensure that it aligns perfectly with our requirements for training a robust grasp generation model. This control over the dataset is crucial for enabling our robot to learn to perform grasping tasks with high accuracy and reliability in diverse real-world scenarios.

Fusion mechanism

The fusion of textual and visual features poses a unique challenge in multimodal learning for different tasks. Lu et al. (2022) used different late fusion strategies for image-text multimodal classification. However, the drawback of late fusion is that it may miss capturing the complex relationships and dependencies between textual and visual features. Yu et al. (2023) proposed DFM, which aims to facilitate the information exchange between linguistic and visual features, helping the model learn richer and more discrete representations, ultimately leading to better component-free learning. Although DFM aims to enhance visual feature learning through guidance from language, it still relies heavily on pre-trained linguistic features. The ability of the model to infer unseen components may be limited by the expressiveness and scope of the language embedding space. Liang et al. (2022) introduced cosine similarity to measure how well the generated text matches the input image. CLIP encodes the image and text into separate vectors, and the cosine similarity is calculated between these vectors. A higher cosine similarity indicates that the text is more relevant to the image, which serves as a reward signal for the reinforcement learning algorithm to learn better alignments. In this study, we merge text and image features using cosine similarity to focus on each relationship between text and image.

Depth estimation

Accurate depth information is crucial for our language-driven RGB-D grasp generation system, as it provides the necessary three-dimensional context for effective grasp planning. Depth estimation aims to infer the distance of objects in a scene from a single or multiple images (Yang et al., 2024a). This information is essential for robots to accurately perceive the environment and plan their actions, especially when grasping objects. To generate depth maps for our dataset, we leverage the Depth Anything V2 algorithm (Yang et al., 2024b), an advanced monocular depth estimation model known for its superior accuracy and robustness.

Depth estimation is a rapidly evolving field, with numerous methods and models being developed. Other notable approaches include stereo vision (Scharstein & Szeliski, 2003), structured light (Zhang, 2000), and time-of-flight sensors (Jähne, 2002). However, monocular depth estimation methods like Depth Anything V2 are particularly valuable for robotics applications, as they offer a cost-effective and flexible solution for depth perception, with improved generalization to diverse real-world scenarios.

Materials

This section details the dataset and codes employed in this study.

1. Code repository: The code utilized for data processing, model training, and evaluation is publicly available on Zenodo (DOI: 10.5281/zenodo.14038077). This repository includes the code itself, comprehensive installation instructions, and detailed documentation, ensuring the reproducibility of our experiments. Access the code here: https://zenodo.org/records/14038077.

2. 3rd-party dataset DOI/URL: The publicly available GraspNet dataset (https://github.com/graspnet) provided a comprehensive collection of data for object grasping tasks.

Training data: The GraspNet training data consist of four compressed files (train_1.zip to train_4.zip) downloadable directly from the following Zenodo link:

- https://zenodo.org/records/16006850

Testing data: The testing data encompasses three separate zipped archives: test_seen.zip, test_similar.zip, and test_novel.zip downloadable directly from the following Zenodo link:

- https://zenodo.org/records/16006850

Additionally, separate label files (grasp_label.zip and collision_label.zip) provide the corresponding ground-truth labels. These files are available at the following Zenodo link:

- https://zenodo.org/records/16006850

3. Data extraction steps

To prepare the data for model training and testing, you need to carefully follow the steps below to ensure that the data are properly organized and ready for use. First, download the GraspNet dataset directly from the following Google Drive links above to download the following files: train_1.zip, train_2.zip, train_3.zip, train_4.zip, grasp_label.zip, collision_grasp.zip, test_seen.zip, test_similar.zip, and test_novel.zip. After downloading, create a root directory named data to store all the data. Within the data directory, create four subdirectories with the following organization and processing:

• •

  Directories grasp_label and collision_grasp:

• Extract the files grasp_label.zip and collision_grasp.zip. After extraction, you will have two folders named grasp_label and collision_grasp.

• •

  Directory scenes:

• Extract all the files train_1.zip, train_2.zip, train_3.zip, train_4.zip, test_seen.zip, test_similar.zip, and test_novel.zip. From the extracted files, take all the subdirectories and consolidate them into a single folder named scenes.

• •

  Directory text data:

• This research utilizes text data provided in Supplemental File 1. After downloading and extracting, you will have a folder named text_data inside the data directory.

Next, you need to generate tolerance labels by using the script generate_tolerance_label.py. This script requires the path to the data folder as input. Make sure to provide the correct path and execute the script. Finally, you can begin the training or testing process by running the files train.py or test.py. Before running these scripts, ensure that all the paths within the scripts are configured correctly to point to the data directory with the structure outlined above.

To streamline data handling and ensure efficient model training, the downloaded data have been meticulously extracted and organized into a structured directory as follows:

• root:

  • data

    • grasp_label

      • folder containing label data from grasp_label.zip

    • collision_label

      • folder containing label data from collision_label.zip

    • scenes

      • folder containing extracted scene data (train_1.zip to train_4.zip, test_seen.zip, test_similar.zip, and test_novel.zip) from GraspNet

    • text_data

      • folder containing the text dataset, including data.json and vocab.json.

Methods

The raw data was considered sufficiently clean and suitable for the intended analysis. Therefore, no data pre-processing steps such as cleaning, transformation, or feature engineering were applied.

Synthetic data generation details

The generated image is processed using a depth estimate model, such as Depth Anything V2 (Yang et al., 2024b), to extract depth information. For converting these relative depth maps into 3D point clouds suitable for GraspNet (Fang et al., 2020), we utilize standard intrinsic parameters from the Kinect depth sensor. Our automated pipeline generates a large volume of synthetic RGB images using Stable Diffusion XL. From this pool, we select images where the projection of the Depth Anything V2 output, enhanced by its teacher-student framework for improved metric accuracy, results in coherent 3D geometry and yields high-confidence grasp proposals from the subsequent GraspNet annotation stage. This selection ensures that the generated 3D data are well-formed and aligns closely with real-world sensor characteristics.

Grasp generation pipeline

Our grasp generation pipeline is a multistage process that leverages a combination of language processing, image generation, and depth estimation to create a robust dataset to train a grasp generation model. This pipeline efficiently automates the process of creating a diverse and realistic dataset for robust grasp generation. See Fig. 1 for a visual representation of the pipeline.

Text generation on available GraspNet data

The availability of comprehensive textual descriptions is crucial for effectively utilizing GraspNet data in language-driven grasp generation. However, for existing GraspNet datasets lacking such descriptions, the manual process of creating them can be incredibly time consuming and laborious. To address this challenge, we propose an efficient and scalable solution: taking advantage of LLMs (Radford et al., 2019; Brown et al., 2020) to automatically generate text descriptions for the scene and objects depicted in the data. LLMs are capable of analyzing the available GraspNet data, understanding the objects present, their attributes, and their spatial relationships within the scene. This understanding enables them to construct comprehensive text prompts that not only describe the objects themselves but also incorporate details such as their relative positions, potential lighting conditions, and other relevant attributes. These attributes are critical for generating accurate and contextually relevant grasps. By automating the process of text description generation, we significantly reduce the manual effort required to enrich existing GraspNet datasets. This enables us to expand the scope of language-driven grasp generation, allowing us to train more robust and versatile grasping systems using a wider range of data (See Fig. 2 for examples of text descriptions.).

Grasp generation and training

The generated image and its corresponding depth map are transformed into a 3D point cloud and then fed into GraspNet (Fang et al., 2020). This model analyzes the scene, using both visual and depth information, to identify potential grasp points for a robotic manipulator. The model outputs a set of candidate grasp poses, providing the robot with the information needed to safely and effectively grasp the desired objects.

Language-driven RGB-P grasp detection

The aim of our proposal is to estimate grasps for objects described in text combined with RGB images and point clouds. The proposed method is illustrated in Fig. 3. The features are first extracted from text, RGB image, and point cloud using the corresponding backbone. Attention weights are then calculated between the features from different sources using cosine similarity. Finally, the attended features are passed through a text-guided grasp generation module, refined based on GraspNet (Fang et al., 2020), a baseline model, to estimate grasps for the object.

Attention mechanism with cosine similarity

Taking as input textual features $F_{text}\in {\mathbb{R}}^{L\times 256}$ and spatial, appearance features $F_{rgbp}\in {\mathbb{R}}^{N\times 256}$, this module implements an attention mechanism based on cosine similarity (Lee et al., 2018), to measure the similarity between textual features and each point in the point cloud features, is illustrated in Fig. 4.

$F_{rgbp}$ and $F_{text}$ will calculate the cosine similarity matrix $\in {\mathbb{R}}^{N\times L}$. This matrix measures the similarity between each pair of features. The AEM ( $AttentionElement\text{-}wiseMask$) is generated from the cosine similarity matrix, high similarity represents high attention and vice versa. This mask indicates which features in $F_{text}$ should be attended to for each feature in $F_{rgbp}$, helps to focus on important features in $F_{text}$.

$F_{text}$ is multiplied by AEM. This produces the AM ( $AttentionMask$) representation $AM=\{a{m}_1,\dots ,a{m}_n\}\in {\mathbb{R}}^{256}$, shows the correlation between each feature in $F_{rgbp}$ with all features in $F_{text}$, helps determine which features in $F_{rgbp}$ are important to determine the location of the object mentioned in the text. The attention element-wise mask is computed as follows:

(1) $$AEM=Softmax\left[\frac{\delta (F_{rgbp}\cdot (F_{text}{)}^T)}{\parallel F_{rgbp}\parallel \parallel F_{text}\parallel }\right]$$where $\delta$ denotes for $ReLU$ activation function. And, the attention mask AM is computed as follows:

(2) $$AM=\frac{\delta (AEM\cdot (F_{text}{)}^T)}{\parallel AEM\parallel \parallel F_{text}\parallel }.$$

The weighted features, denoted as $F_{weighted}$, are calculated by element-wise multiplication between $F_{rgbp}$ and AM. This produces the final attention weights, which are used to focus the network’s attention on certain parts of $F_{rgbp}$.

(3) $$F_{weighted}=F_{rgbp}\odot AM$$where $\odot$ denotes element-wise multiplication.

Evaluation metrics

To judge how well our system predicts grasp poses, especially the most promising ones, we use a metric called $Precision@k$ (Fang et al., 2020). This metric considers how many of the top k grasp predictions are actually correct for grabbing a specific object. We figure this out by checking the point cloud data within the gripper for each predicted pose and using a force-closure analysis (considering different levels of friction coefficients $\mu$) to see if it would work. In the context of our text-guided approach, $Precision@k$ indirectly evaluates vision-language grounding accuracy, as successful grasps require the model to correctly align the textual description with the visual object before generating appropriate grasp poses. A direct vision-language grounding metric, such as text-image cosine similarity or CLIP-based alignment scores, could provide additional insight and is considered for future work.

Since good robot grasp often relies on the very first suggested options, $Precision@k$ focuses on the accuracy of those top predictions. We calculate it by ranking all the predicted poses based on their confidence scores, then picking the top k ones, and finally checking how many of those k actually work (true positives).

To get a more complete picture, we calculate $AveragePrecision@\mu$ ( $A{P}_{\mu }$) in a range of k values (typically 1 to 50) for different levels of friction coefficients. This $A{P}_{\mu }$ metric is especially important for cluttered environments where robots need to prioritize the best grasp options first. In our experiments, we will report $A{P}_{\mu }$ results under various conditions, including different settings and friction coefficients $\mu$. This will help demonstrate how well our system finds good 6-DoF grasps in various situations.

Evaluation on GraspNet-1Billion dataset

The main purpose in this section is to evaluate the overall performance of the model, and we compared it with the current state-of-the-art methods. In general, these models achieve impressive results and make significant contributions to the field. We used the Precision@k metric to provide a comprehensive evaluation of the methods. To ensure fairness, we reimplemented the methods using the source code provided by the authors. The results in Table 2 present the performance on “Seen” objects using the Kinect sensor, showing our improvement with an average precision (AP) of 53.2%, $A{P}_{0.8}$, $A{P}_{0.6}$, and $A{P}_{0.4}$ of 63.1%, 55.6%, and 44.1%, respectively. These results demonstrate that our method improves upon other methods. Even when compared with different thresholds ( $A{P}_{0.8}$, $A{P}_{0.6}$, and $A{P}_{0.4}$), our method retains superior performance, demonstrating accurate and reliable capture. This performance highlights the advantage of our multimodal approach, which leverages semantic information from text to disambiguate objects, combined with RGB and point cloud data for robust grasp detection. This targeted prioritization is crucial for robots operating in cluttered environments where choosing the best grasp attempt is critical. Additionally, the system operates with exceptional efficiency, processing each grasp prediction in just 260 milliseconds. This rapid processing time makes it suitable for real-time applications where time-sensitive decisions are essential.

The Table 3 aims to demonstrate two important points. First, we show that improving the model’s input by incorporating color information in addition to point cloud data is beneficial. Instead of only using point cloud data to extract features, we leverage the geometric information from the point cloud and add color information. This significantly enhances object detection performance. Specifically, from the results table, for the cosine similarity fusion mechanism, the AP scores without color are 45.4%, 52.1%, 46.2%, and 36.3%, while with color the AP scores are 53.2%, 63.1%, 55.6%, and 44.1%. Similarly, other fusion mechanisms also show an approximate 1% improvement when color information is added. Second, we compare three different fusion methods. The first method is multiplication. Although this method allows the model to learn nonlinear relationships between variables, it encounters issues when features have very different ranges. This leads to features with larger values dominating, reducing the influence of other features and causing imbalance in the model. Consequently, the multiplication method may decrease the model’s effectiveness in accurately representing object characteristics. The second method is concatenation. This method is simple and flexible, and does not require complex operations to combine features. However, it does not handle nonlinear relationships between features of the text encoder and visual encoder. The final method is cosine similarity. This method measures the similarity between two vectors without being affected by differences in scale or magnitude of the features. Experimental results indicate that using cosine similarity for fusion between features from the text encoder and visual encoder is the most suitable approach for our study.

Grasping robot simulation experiment

To assess our method’s ability to generate diverse and effective grasps, we conducted simulation experiments in CoppeliaSim (Rohmer, Singh & Freese, 2013) using a robotic arm with a multi-fingered gripper. For each experiment, we fed the same object point cloud and different text prompts into the system. The generated grasps were evaluated based on quality, stability, and feasibility before simulation. As Fig. 5 demonstrates, varying text prompts led to distinct grasp configurations for the same object, highlighting the system’s capacity to produce task-specific grasps. These results indicate the potential for our approach to enable robots to perform complex manipulation tasks in real world environments. However, extensive validation on physical robotic platforms is necessary to confirm real-world applicability, as discussed in “Limitations”.