Novel cross-dimensional coarse-fine-grained complementary network for image-text matching

Coarse-grained global matching

The process involves independently encoding images and texts into a unified embedding space using two separate branches, obtaining coarse-grained global representations through specific aggregators, and performing similarity-matching calculations. Existing work typically utilizes GCN (Li et al., 2019, 2024b; Wang et al., 2020b) or transformer (Wang et al., 2020a; Wen, Gu & Cheng, 2020; Wu et al., 2019) to enhance the contextual semantics of local features. For instance, Li et al. (2019) introduced a simple semantic reasoning network (VSRN) to learn local image features with key scene concepts. Wang et al. (2020a) introduced a consensus-aware module (CVSE) that incorporates common sense knowledge into local features of two patterns. An important research direction in coarse-grained global matching focuses on the work of clustering pooling strategies (Chen et al., 2021; Li et al., 2019; Zeng et al., 2021; Zhu et al., 2023). For instance, Li et al. (2019) leveraged gates and memory mechanisms to conduct global semantic inference on features, selectively identify discriminative information, and derive representations of the entire scene. Chen et al. (2021) introduced the optimal pooling method (GPO) that automatically adapts to different characteristics without manual tuning, down sampling intra-modal features into global features. Zhu et al. (2023) introduced a flexible ESA strategy, facilitating element-level fusion of image and text features in specific spatial dimensions. Kim, Kim & Kwak (2023) proposed the idea of utilizing a set that transforms an image or text sample into a collection of unique embedding vectors, designed to encompass diverse semantic elements of the samples. However, previous works still follow the existing paradigm, this work considers that the same feature may mean different things in different dimensions and have different importance in different dimensions. Aggregating vector weights in both different and same dimensions can maximize the acquisition of potentially rich dimensional semantics.

Fine-grained local matching

The goal is to identify all potential alignments between textual words and image regions, thereby inferring the comprehensive correlation between the image and text, better performance can usually be achieved (Chen et al., 2020; Diao et al., 2021; Fu et al., 2023; Lee et al., 2018; Liu et al., 2020; Wang et al., 2023b; Zhang et al., 2022a, 2022b, 2024b). Most work in this field is based on attention mechanisms. Lee et al. (2018) introduced the SCAN technique as a prominent approach for dynamically targeting the common semantics across different modalities in response to each query content. On this basis, many variants were proposed. The focus of a sequence of works is to devise precise and complex matches (Chen et al., 2020; Diao et al., 2021; Lee et al., 2018; Liu et al., 2020; Zhang et al., 2022b). For instance, Chen et al. (2020) introduced Recurrent Cross Attention, which iteratively extracts and enhances shared semantics across multiple levels. Zhang et al. (2022b) proposed measuring the degree of similarity and dissimilarity through two matches (NAAF). Pan, Wu & Zhang (2023) proposed a new encoding strategy (CHAN) to find the most relevant region-word pairs to address redundant alignment and cross-attention mechanisms. Diao et al. (2021) first used the vectorised representation method (SGRAF), which is able to calculate the similarity and inference between two modal at a finer granularity. Simultaneously, various techniques concentrates on leveraging external and other information to construct relationships within the same modal and between different modal of sentences and images (Fu et al., 2023; Zhang et al., 2022a). For example, Zhang et al. (2022a) proposed the region position (CMCAN) to reflect the surrounding visual scene, while Fu et al. (2023) emphasized and enriched the interaction dynamics between visual and texts through semantic relationships between instances. Nevertheless, the emphasis of these methods remains on purely calculating the localized similarity between image segments and textual terms to gather the overall image-text correlation, which can easily lead to excessive attention to the the connections between image areas and textual terms, thereby causing a discrepancy between the global and local information. our proposed method focuses on using coarse-grained global matching as a complement to fine-grained image segments and textual terms matching to solve the problem of semantic inconsistency between global and local information, and adopting cross-dimensional semantic dependency in the process of fine-grained image segments and textual terms matching to mine richer image and text features.

Vision-language pre-trained model matching

Early Vision-Language Pre-trained (VLP) models followed a detector-based roadmap (Chen et al., 2019). The CLIP model (Radford et al., 2021), ALIGN (Jia et al., 2021), and unicoder-VL (Li et al., 2020) respectively utilized 400 million image-text pairs, 1.8 billion noisy image-text pairs, and 3.8 million web image-text data for contrastive training. These models achieved efficient cross-modal retrieval and cross-modal understanding and generation tasks by integrating multimodal information. Although this approach demonstrated strong performance, it also posed challenges of expensive computation and unstable detector training. Subsequently, methods based on Vision Transformers (ViTs) adopted a pure Transformer architecture for image encoding, eliminating the need for object detectors and enabling an end-to-end VLP framework. For instance, Huang et al. (2021) proposed an “out-of-the-box” solution that does not rely on complex preprocessing steps; Kim, Son & Kim (2021) utilized the Transformer architecture to directly take unprocessed image segments and textual word embeddings as inputs, avoiding complex visual feature extraction processes. Li et al. (2021) introduced momentum distillation techniques, using a momentum model to enhance visual and language feature alignment, resulting in significant performance improvements across multiple vision-language tasks. However, these methods had limitations in fine-grained cross-modal alignment, increased computational costs, and introduced noisy visual information for cross-modal fusion. More recent works have made further advancements in enhancing the effectiveness of the pre-trained process for vision-language models. For instance, Fu et al. (2024) introduced a patch fusion method designed to obtain a compact representation of visual token sequences and strengthen the integration across different modes. Additionally, the BLIP-2 model series (Li et al., 2023) adopted a guided language-image pre-trained method utilizing pre-fixed image encoders and substantial language models, achieving effective fusion and precise matching of visual and language information.

Recent studies have produced works similar to this research. For instance, Xu et al. (2020) introduced the CASC joint framework that performs general local alignment and global semantic consistency multi-label prediction. Similarly, Li et al. (2024b) proposed the memory networks to achieve complementarity between independent and interactive embedding approaches, thereby improving retrieval efficiency. However, these methods do not consider address fine-grained cross-dimensional semantic matching, nor did it consider the relationship between different dimensions of the same feature and different features of the same dimension in coarse-grained matching. Zhang et al. (2023) conducted precise mining and modeling of dimensional semantic dependencies (x-dim), but still only performed fine-grained cross-dimensional semantic matching. In contrast, the approach presented in this work explicitly constructs a coarse-grained complementary cross-dimensional semantic aggregation network, performs fine-grained cross-dimensional semantic dependency matching, and further aggregates cross-dimensional image and text semantic to ensure consistency between global and local information.

Feature extraction

Image representation. For an image v, this work uses a bottom-up attention network (Anderson et al., 2018) to extract salient regions, which employs ResNet-101 (He et al., 2016) as the backbone and is implemented by Faster-RCNN (Ren et al., 2015). Then, this work adds a fully connected (FC) layer to map each region feature to a d-dimensional local feature, represented as $\mathrm{V}=\left\{{\mathrm{v}}_1,...{\mathrm{v}}_{\mathrm{m}}\right\}\in {\mathrm{R}}^{\mathrm{m}\times \mathrm{d}}$, which is the visual fragment and local feature of image v, and the $\mathrm{m}$ is the number of image region features.

Text representation. For a sentence u, this work uses a sequence model, bidirectional gated recursive unit (BiGRU) (Schuster & Paliwal, 1997), or pre-trained BERT model (Devlin et al., 2018) to extract the word feature set. This work also adds a fully connected (FC) layer to maintain the same dimension as the image, represented as $\mathrm{U}=\left\{{\mathrm{u}}_1,...{\mathrm{u}}_{\mathrm{n}}\right\}\in {\mathrm{R}}^{\mathrm{n}\times \mathrm{d}}$, which is the text fragment and local feature of the text u, $\mathrm{n}$ is the number of word features.

Fine-grained cross-dimensional semantic dependency branch

For the semantic similarity between any word feature $\mathrm{u}={\left\{{\mathrm{u}}_{\mathrm{i}}\right\}}_{\mathrm{i}=1}^{\mathrm{d}}$ and region feature $\mathrm{v}={\left\{{\mathrm{v}}_{\mathrm{i}}\right\}}_{\mathrm{i}=1}^{\mathrm{d}}$, according to Fig. 1B, typical approaches often employ implicit independent aggregation to reflect it for all dimension correspondences, i.e., ${\sum }_{\mathrm{i}=1}^{\mathrm{d}}{\mathrm{s}}_{\mathrm{i}}$, where ${\mathrm{s}}_{\mathrm{i}}$ can be determined by the product of scalars ${\mathrm{u}}_{\mathrm{i}}$ and ${\mathrm{v}}_{\mathrm{i}}$ in internal product operations. That is to say, The aggregated process of cross-modal correspondence vectors ${\mathrm{s}}_{\mathrm{i}}={\left\{{\mathrm{s}}_{\mathrm{i}}\right\}}_{\mathrm{i}=1}^{\mathrm{d}}$ in the similarity sharing-space is based on the unconscious assumption that each dimension is perceived as a distinct isolated component.

Inspired by Zhang et al. (2023), not all dimensions in a similarity shared-space are independent, and some local dimensions have potential relationships between them, defined as cross-dimensional semantic dependencies. This work introduces the conditional probability of each dimension to represent joint semantics. It selects dimensions with significant correlation for sparse correlation probability learning based on adaptive adjustment of learned weights to sparsity. Firstly, as shown in the dimension-dependent perception aggregation in Eq. (1):

(1) $$\mathrm{A}\mathrm{g}{\mathrm{g}}_{\mathrm{w}}\left(\mathrm{s}\right)=\sum _{\mathrm{i}=1}^{\mathrm{d}}{\left||{\left[\begin{array}{ccc}{\mathrm{w}}_{11}& \cdots & {\mathrm{w}}_{1\mathrm{d}}\\ ⋮& \ddots & ⋮\\ {\mathrm{w}}_{\mathrm{d}1}& \cdots & {\mathrm{w}}_{\mathrm{d}\mathrm{d}}\end{array}\right]}_{\mathrm{W}}\times {\left[{\mathrm{s}}_1,...,{\mathrm{s}}_{\mathrm{i}},....\right]}^{\mathrm{T}}|\right|}_{{\mathrm{l}}_2}$$where $\mathrm{W}\in {\mathrm{R}}^{\mathrm{d}\times \mathrm{d}}$represents a learning weight matrix and each row of which represents a semantic dependency between various dimensions. Specifically, for row i of the weight matrix, The formula for $\widehat{{\mathrm{s}}_{\mathrm{i}}}$ is as follows:

(2) $${\hat{\mathrm{s}}}_{\mathrm{i}}={\mathrm{w}}_{\mathrm{i}1}.{\mathrm{s}}_1+...+{\mathrm{w}}_{\mathrm{i}\mathrm{j}}.{\mathrm{s}}_{\mathrm{j}}+...+{\mathrm{w}}_{\mathrm{i}\mathrm{d}}.{\mathrm{s}}_{\mathrm{d}},\mathrm{j}\in \left[1,\mathrm{d}\right].$$

It models the dependency relationships among all dimensions within the shared representation space of the i-th joint representation, collectively representing latent semantics. To explicitly quantify, the conditional probabilities for each dimension are introduced as follows:

(3) $$\mathrm{p}({\hat{\mathrm{s}}}_{\mathrm{i}}|{\mathrm{s}}_{\mathrm{j}})=\mathrm{S}\mathrm{i}\mathrm{g}\mathrm{m}\mathrm{o}\mathrm{i}\mathrm{d}\left({\mathrm{w}}_{\mathrm{i}\mathrm{j}}\right),\mathrm{j}\in \left[1,\mathrm{d}\right],$$where $\mathrm{p}(\widehat{{\mathrm{s}}_{\mathrm{i}}}|{\mathrm{s}}_{\mathrm{j}})\in \left[0,1\right]$ reflects the extent of dependency of the j-th dimension for the jointly represented ${\hat{\mathrm{s}}}_{\mathrm{i}}$. The likelihood that this dimension in the shared space participates in the joint representation is positively correlated with the value $\mathrm{p}({\hat{\mathrm{s}}}_{\mathrm{i}}|{\mathrm{s}}_{\mathrm{j}})$. The learned conditional probability $\{\mathrm{p}({\hat{\mathrm{s}}}_{\mathrm{i}}|{\mathrm{s}}_{\mathrm{j}}){\}}_{\mathrm{j}=1}^{\mathrm{d}}$ represents the dependence of all dimensions on latent semantics ${\hat{\mathrm{s}}}_{\mathrm{i}}$, Zhang et al. (2023) proved in the article that the probability density is approximately a normal distribution. Drawing from the statistical characteristics of conditional probability, this work initially allows the model to independently acquire a soft threshold for distinguishing if the dimensions exhibit joint dependencies. The significance is as follows:

(4) $${\mathrm{t}}_{\mathrm{i}}={\mathrm{u}}_{\mathrm{i}}+{\mathrm{\alpha }}_{\mathrm{i}}\cdot {\mathrm{\sigma }}_{\mathrm{i}}$$where ${\mathrm{u}}_{\mathrm{i}}$ and ${\mathrm{\sigma }}_{\mathrm{i}}$ denote the mean and standard deviation of the sampling probability values $\{\mathrm{p}({\hat{\mathrm{s}}}_{\mathrm{i}}|{\mathrm{s}}_{\mathrm{j}}){\}}_{\mathrm{j}=1}^{\mathrm{d}}$. ${\mathrm{\alpha }}_{\mathrm{i}}$ is a learnable parameter for adaptively adjusting the soft threshold to regulate the ratio of selected dimensions. Further scaling operations obtain the modified sparse probability as:

(5) $$\hat{\mathrm{p}}({\hat{\mathrm{s}}}_{\mathrm{i}}|{\mathrm{s}}_{\mathrm{j}})=\mathrm{\delta }\left(l^{\mathrm{\gamma }(\mathrm{p}(\widehat{{\mathrm{s}}_{\mathrm{i}}}|{\mathrm{s}}_{\mathrm{j}})-{\mathrm{t}}_{\mathrm{i}})}\right)$$where $\mathrm{\gamma }$ represents a trainable scaling parameter, $\mathrm{\delta }(\cdot )$ is the activation function.

In other words, based on the learning of conditional probability $\mathrm{p}({\hat{\mathrm{s}}}_{\mathrm{i}}|{\mathrm{s}}_{\mathrm{j}})$, ${\mathrm{t}}_{\mathrm{i}}$ as a critical point, dimensions with dependency less than the threshold ${\mathrm{t}}_{\mathrm{i}}$ will be discarded, while dimensions with dependency greater than the threshold ${\mathrm{t}}_{\mathrm{i}}$ will continue to be used. With the operations mentioned above, Eq. (1) will be modified to Eq. (6):

(6) $$\widehat{{\mathrm{s}}_{\mathrm{i}}}=\sum _{\mathrm{i}=1}^{\mathrm{d}}\hat{\mathrm{p}}(\widehat{{\mathrm{s}}_{\mathrm{i}}}|{\mathrm{s}}_{\mathrm{j}})\cdot {\mathrm{w}}_{\mathrm{i}\mathrm{j}}\cdot {\mathrm{s}}_{\mathrm{j}},\mathrm{j}\in \left[1,\mathrm{d}\right].$$

Finally, adaptive dimensional semantic dependency perception can be formulated as:

(7) $$\mathrm{A}\mathrm{g}{\mathrm{g}}_{\mathrm{w}}\left(\mathrm{s}\right)=\sum _{\mathrm{i}=1}^{\mathrm{d}}||\left(\hat{\mathrm{P}}\odot \mathrm{W}\right)\times {\mathrm{s}}^{\mathrm{T}}|{|}_{{\mathrm{l}}_2}$$where $\hat{\mathrm{P}}=\hat{\mathrm{p}}({\mathrm{s}}_{\mathrm{i}}|{\hat{\mathrm{s}}}_{\mathrm{j}})$, $\mathrm{i},\mathrm{j}\in \left[1,\mathrm{d}\right]$ can be regarded as an adaptive regularizer to achieve fine-grained dimensional dependency modeling.

Coarse-grained cross-dimensional semantic aggregation module

The fine-grained cross-dimensional semantic dependency branch has completed the fine-grained cross-dimensional region-word similarity computation, and in order to avoid excessive focus on the local image region and text word information to maintain the consistency of global and local semantic information, coarse-grained global image text information needs to be introduced as an auxiliary balance. However, current global features generated by aggregation of commonly used pooling operations, such as average pooling and maximum pooling, for example, contain a lack of semantic features. Inspired by Wu et al. (2023), his work found that different dimensions may have different meanings for the same feature. Taking the term “match” as an illustration, certain dimensions may connote “light” while others represent “competition”. Similarly, some dimensions of “doctor” may represent “physician” while others represent “doctor of philosophy”. It means that when consolidating global features, the significance of various dimensions within the same feature vector should vary.

Based on previous research, this work introduces a novel aggregation module called CGDSA. Unlike common pooling operations, as shown in Fig. 4, CGDSA module considers the relationships between different vectors on the same dimension and the differences of the same feature between different dimensions. Specifically, taking an image as an example, this work first enhances the obtained features:

(8) $$\mathrm{F}={\mathrm{W}}_2\left(\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{u}\left({\mathrm{W}}_1\mathrm{U}+{\mathrm{b}}_1\right)\right)+{\mathrm{b}}_2$$where ${\mathrm{W}}_1$, ${\mathrm{b}}_1$ and ${\mathrm{W}}_2$, ${\mathrm{b}}_2$ are the parameters of the two fully connected layers, the fully connected layer transforms u into the d-dimensional common space to standardize the representation dimension, ReLu activation function for nonlinear mapping to obtain higher semantic information.

Then, this work draws on an additional external d * d space (denoted by M) to keep the image and text mapped within a fixed dimensional attention space, which allows for relationships between different vectors in the same dimension. The expression formula for this matrix is as follows:

(9) $${\mathrm{F}}_1=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\mathrm{F}\ast \mathrm{M}\right).$$

Meanwhile, under the influence of the combination of vertical vectors, discriminative information in the horizontal dimension cannot be flexibly integrated in a fine-grained manner. Therefore, this work aggregates features from the direction of column elements to obtain aggregation weights to obtain richer features, and the specific formula is as follows:

(10) $${\mathrm{F}}_2=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\mathrm{F}\right).$$

Note that the softmax function is executed separately in the region dimension column direction. Column-wise softmax involves applying the softmax function to each column individually, ensuring that the sum of N elements in each column equals 1.

Additionally, this work introduces vectors as aggregation weights, which can achieve finer granularity aggregation compared to scalars as aggregation weights. Lastly, this work amalgamates features of the identical dimension mapped to the external space and features of diverse dimensions to derive the ultimate aggregated feature. The computation formula is shown as Eq. (11):

(11) $$\hat{\mathrm{u}}=\sum _{\mathrm{i}=1}^{\mathrm{N}}{\mathrm{F}}_1\odot \left({\mathrm{u}}_{\mathrm{i}}\right)+\sum _{\mathrm{i}=1}^{\mathrm{N}}{\mathrm{F}}_2\odot \left({\mathrm{u}}_{\mathrm{i}}\right).$$

Note that ${\mathrm{F}}_1$, ${\mathrm{F}}_2$ are the vectors, where each element stands for the combined weight of each dimension of ${\mathrm{u}}_{\mathrm{i}}$, of course, the aggregation of textual features can also be accomplished by the operation of this module, notated as $\hat{\mathrm{v}}$.

Coarse-fine-grained semantic matching

The goal of the CDGCN method is the combination of fine-grained cross-dimensional semantic dependency match branch and coarse-grained cross-dimensional semantic aggregation match branch. The CDGCN is included into the cross attention inference to establish cross-modal fine-grained matching. Taken out an image-text pair ( $\mathrm{U},\mathrm{V})$ from the dataset which comprises n text-words ${\left\{{\mathrm{u}}_{\mathrm{q}}\right\}}_{\mathrm{q}=1}^{\mathrm{n}}$ and m salient image-regions ${\left\{{\mathrm{v}}_{\mathrm{p}}\right\}}_{\mathrm{p}=1}^{\mathrm{m}}$, the semantic similarity of all text-word image-region pairs is obtained as follows (Lee et al., 2018):

(12) $${\mathrm{r}}_{\mathrm{q}\mathrm{p}}=\mathrm{A}\mathrm{g}\mathrm{g}\left({\mathrm{s}}_{\mathrm{q}\mathrm{p}}\right),\mathrm{s}.\mathrm{t}.{\mathrm{s}}_{\mathrm{q}\mathrm{p}}={\mathrm{u}}_{\mathrm{q}}\odot {\mathrm{v}}_{\mathrm{p}}.$$

For each text word query ${\mathrm{u}}_{\mathrm{q}}$, its relevant area can be represented as:

(13) $${\hat{\mathrm{v}}}_{\mathrm{q}}=\sum _{\mathrm{p}=1}^{\mathrm{m}}{\mathrm{\beta }}_{\mathrm{q}\mathrm{p}}{\mathrm{v}}_{\mathrm{p}},\mathrm{s}.\mathrm{t}.{\mathrm{\beta }}_{\mathrm{q}\mathrm{p}}={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(\lambda \odot \mathrm{\sigma }\left({\mathrm{r}}_{\mathrm{q}\mathrm{p}}\right)\right)}{{\sum }_{\mathrm{p}=1}^{\mathrm{m}}\mathrm{e}\mathrm{x}\mathrm{p}\left(\lambda \odot \mathrm{\sigma }\left({\mathrm{r}}_{\mathrm{q}\mathrm{p}}\right)\right)}}$$where ${\mathrm{\beta }}_{\mathrm{q}\mathrm{p}}$ is weight, $\lambda$ denotes a trainable scaling parameter. Following the same logic, this work can obtain the correlation score between the q-th text-word and the image as: ${\hat{\mathrm{r}}}_{\mathrm{q}}=\mathrm{A}\mathrm{g}\mathrm{g}\left({\mathrm{s}}_{\mathrm{q}}\right)$, ${\hat{\mathrm{s}}}_{\mathrm{q}}={\mathrm{u}}_{\mathrm{q}}\odot {\hat{\mathrm{v}}}_{\mathrm{q}}$. Finally, the overall correlation between image and text is inferred as:

(14) $$\mathrm{S}{\left(\mathrm{U},\mathrm{V}\right)}_{\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{e}}={\displaystyle \frac{1}{\mathrm{n}}\sum _{\mathrm{q}=1}^{\mathrm{n}}{\hat{\mathrm{r}}}_{\mathrm{q}}.}$$

Similarly, this work maps the aggregated image features $\hat{\mathrm{v}}$ and text features $\hat{\mathrm{u}}$ into a shared space to compute cosine similarity for coarse-grained global matching: $\mathrm{s}{\left(\hat{\mathrm{v}},\hat{\mathrm{u}}\right)}_{\mathrm{c}\mathrm{o}\mathrm{a}\mathrm{r}\mathrm{s}\mathrm{e}}={\displaystyle \frac{{\hat{\mathrm{v}}}^{\mathrm{\top }}\cdot \hat{\mathrm{u}}}{||\hat{\mathrm{v}}||\cdot ||\hat{\mathrm{u}}||}}$.

This work trains and computes bi-directional ranking losses (Faghri et al., 2017) for matching image-text pairs:

(15) $$\mathcal{L}\left(\mathrm{U},\mathrm{V}\right)={\left[\mathrm{\gamma }-\mathrm{S}\left(\mathrm{U},\mathrm{V}\right)+\mathrm{S}\left(\mathrm{U},{\mathrm{V}}^{-}\right)\right]}^{+}+{\left[\mathrm{\gamma }-\mathrm{S}\left(\mathrm{U},\mathrm{V}\right)+\mathrm{S}\left({\mathrm{U}}^{-},\mathrm{V}\right)\right]}^{+}$$

The final objective function of the CDGCN method derived is:

(16) $$\mathcal{L}=\mathcal{L}{\left(\mathrm{U},\mathrm{V}\right)}_{\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{e}}+\mathrm{\alpha }\mathcal{L}{\left(\mathrm{U},\mathrm{V}\right)}_{\mathrm{c}\mathrm{o}\mathrm{a}\mathrm{r}\mathrm{s}\mathrm{e}}$$where ${\mathrm{V}}^{-}$denotes hardest negative image and ${\mathrm{U}}^{-}$denotes hardest negative text, $\mathrm{\alpha }$ is a hyper-parameter that balances the loss functions between fine-grained cross-dimensional semantic matching and coarse-grained cross-dimensional semantic aggregation matching.

Datasets and evaluation metrics

Datasets: This work uses Flickr30K dataset (Young et al., 2014) and the MS-COCO dataset (Lin et al., 2014), where each image is paired associated with five texts descriptions. The Flickr30k dataset comprises 29,000 training images, 1,000 testing images and 1,014 validation images. The MS-COCO contains 82,738 training images, 5,000 testing images, and 5,000 validation images. Results for MS-COCO were evaluated by averaging outcomes across five runs for the 1 K test images and for the entire set of 5K test images. The Flickr30K dataset can be accessed at: https://shannon.cs.illinois.edu/DenotationGraph/; and the MSCOCO dataset is accessed through URL: https://cocodataset.org/#download. Codes are accessible at https://github.com/meizhenliu/CDGCN-code.

Evaluation metrics: The proposed method’s effectiveness is assessed using metrics R @ K (where K = 1, 5, 10) and rSum (Chen et al., 2021). Specifically, R@K quantifies the proportion of accurately retrieved true results among the top K items returned, whereas rSum represents the sum of all R@K scores. Particularly, R@K indicates the percentage of correct ground truth results retrieved within the first K search items, and rSum signifies the total of all R@K values.

Implementation details

All experiments have been conducted using an NVIDIA GeForce RTX 3090 GPU and the proposed CDGCN is implemented using PyTorch. In the visual branch, this work uses the Bottom-Up and Top-Down Attention (BUTD) network (Anderson et al., 2018; Li et al., 2019) pre-trained on the Visual Genome (VG) dataset to extract N = 36 region features from images, each with 2,048 dimensions. These features are then sent to a linear fully connected layer to map them into a shared space of 1,024 dimensions. For text representation, this work employs BiGRU model with a word embedding size of 300, along with the default pre-trained BERT (Devlin et al., 2018) with 768 dimensions. When using the BiGRU text encoder, this work initialize the word vectors with 300-dimensional global vectors (GLoVe) embeddings (Pennington, Socher & Manning, 2014), applying a dropout rate of 0.1 during training to enhance the generalization of text representations. In the case of using the BERT text encoder, our work makes use of the standard pre-trained BERT model, consisting of 12 transformer layers, each equipped with 12 attention heads and a hidden layer dimension of 768 units. This work also uniformly applys text augmentation strategies to train BERT. Additionally, after encoding both the images and sentences, this work incorporates L2 normalization to process the features, ensuring the stability and consistency of the representations.

During the training process, the visual encoder’s parameters remain unchanged, whereas the parameters of BERT undergo fine-tuning. This work uses the Adam optimizer, with an initial learning rate of 0.0002 for the Flickr30k dataset and 0.0005 for the MSCOCO dataset. The training process spans 30 epochs across both datasets, with a batch size of 128 for each dataset. The learning rate is reduced by 10% every 15 epochs. The margin parameter λ is set to 0.2. The hyperparameter α, which balances the coarse-grained cross-dimensional semantic aggregation matching branch and the fine-grained cross-dimensional semantic dependency matching branch in the loss function, is set to 0.06 for the Flickr30k dataset and 0.1 for the MSCOCO dataset.

During testing, this work evaluates the single model obtained from training, denoted as-single. For model ensembling, this work averages the similarity matrices of any two models obtained from multiple training runs, the ensembled results are denoted as-ensembled*.

Benchmark with state-of-the-art methods

The performance of the CDGCN is compared with state-of-the-art models using individual and integrated models. In this section, the CDGCN which is a relatively low-cost small model is compared with previous low-cost small models. In addition, this work also presents the retrieval results of the state-of-the-art vision-language pre-training large models (Li et al. 2023, 2021) in Table 1. It can be seen that the vision-language pre-training large models (such as BLIP-2) demonstrate superior ability with significant improvement over existing methods. To ensure an unbiased comparison of experimental results, this work categorizes existing methods (focusing only for the low-cost small models) into two groups based on their distinct approaches to extracting visual and textual backbone features. Tables 1 and 2 respectively showcase the quantitative findings of the CDGCN and existing methods on the Flickr30K and MSCOCO 1 K datasets. Even when employing a standalone model rather than an integrated one, the proposed CGDSN has shown significant improvement compared to the state-of-the-art methods (Fu et al., 2023; Pan, Wu & Zhang, 2023; Yao et al., 2024; Zhang et al., 2023; Zhu et al., 2023). The proposed CDGCN has impressive advantages in R @ 1,5,10 and rSum. Notably, compared to the existing SOTA X-dim method (Zhang et al., 2023), the proposed CDGCN achieved a rSum improvement rate of 4.6–7.7% relative to the two datasets. For larger MSCOCO 5K test sets, the proposed method surpasses the SOTA model in all evaluation metrics and has a significant performance gap in Table 3. The proposed CDGCN achieved a rSum improvement rate of 5.0–8.8% for the two datasets, which more convincingly demonstrates the superiority of the proposed CDGCN.

This research achievement substantially advances the acquisition and understanding of image and text features, enhancing the robustness and the accuracy of image-text matching. It is well suited for application across various multimodal domains, including e-commerce, intelligent assistants, medical image analysis, and social media. This technology enables more accurate image retrieval and recommendation systems, improves user interaction experiences, optimizes content generation and editing tools, and supports intelligent diagnosis and public opinion analysis systems. Hence, this work improves the operational efficiency and user satisfaction of various industries and promotes innovation and development of multimodal data processing technology.

Ablation studies

As shown in Tables 4 and 5, this work conducts comprehensive ablation studies on the Flick30k dataset to validate the efficacy of the proposed CDGCN components. By default, experiments are performed on the fusion mechanism framework using Region+BERT.

Firstly, this work conducts ablation studies to explore the efficacy of two branches of the CDGCN. The without (W/O) fine-grained branch represents the removal of fine-grained cross-dimensional semantic dependency matching branches, the W/O fine-grained cross-dimension branch represents the retention of fine-grained semantic matching branches without performing similarity cross-dimensional dependency calculations, and the W/O coarse-grained cross-dimensional semantic matching branch represents the removal of coarse-grained cross-dimensional semantic matching branches. The comparative outcomes are depicted in Table 4. The performance of rSum without fine-grained cross-dimension module decreased by 4.8%, indicating the importance of using fine-grained cross-dimension semantic matching branches. In addition, the performance of rSum W/O fine-grained cross-dimension shows a significant decrease, which also indirectly verifies the powerful effect of similarity cross-dimension dependency. The performance of the rSum W/O coarse-grained branch module decreased by 1.9%, indicating that using coarse-grained cross-dimensional semantic matching branches can effectively obtain rich image text features while maintaining consistency between overall and local features, thereby enhancing the performance of image-text retrieval.

Secondly, to verify the effectiveness of the CGDSA module, this work conducts ablation experiments on each component in the image-text retrieval task. It uses three standard pooling methods to aggregate local features and Table 5 presents the comparative outcomes. This work shows that only same dimension attention aggregation (OSDAA) represents only external spatial attention aggregation in the same dimension, meanwhile, only different dimension attention aggregation (ODDA) represents only cross-dimensional aggregation in different dimensions with scalar representing scalar aggregation weight, and row-wise softmax representing row element direction aggregation feature. As the data displayed in Table 5, it is clear that the importance of each component in CGDSA module is significant. If only the DDA module is used, without the SDAA module and cross-dimensional aggregation features of different dimensions, the performance of rSum decreases by 4.8%. If only the SDAA module is used, without the DDA module, and without features that interact with different vectors of the same dimension, the performance of rSum decreases by 2.5%. Using scalar aggregation weights instead of vectors cannot obtain richer element-level aggregation, significantly reducing rSum’s performance. Furthermore, the CGDSA method performs better than other methods. For example, using mean pooling resulted in a 12.5% and 11.4% decrease in R @ 1 for text retrieval and image retrieval, individually. GPO resulted in a 2.1% and 1.8% decrease in R @ 1 for text retrieval and image retrieval, particularly, resulting in a 6.4% decrease in rSum performance. Different from simple pooling methods, this work emphasizes the differences in feature dimensions and the mapping of identical dimensions and vectorizes the aggregation weights, thereby achieving a finer-grained aggregation of features across identical and distinct dimensions.

In addition, as a plug-and-play CGDSA module, the CGDSA method can also be directly used in visual semantic embedding (VSE) tasks. Compared with other pooling strategies, all indicators of the proposed CGDSA method have been significantly improved. Table 6 illustrates that the CGDSA module enhances the effectiveness of visual semantic embedding. The CGDSA method improves R @ 1 by 12.5% and 11.4% respectively when compared to average pooling text retrieval and image retrieval. In addition, the improvement of R@1 using the CGDSA method has improved by 2.1% and 1.8%, particularly, resulting in a 6.4% performance improvement when compared to the GPO method. In addition, this work also compared the fusion methods of only SDAA and only DDA, and the results showed that the sum of the elements in the SDAA and DDA had the best effect. The CGDSA (concatenation) indicates that the work performed subsequent operations on elements; meanwhile, CGDSA (element added) suggests that the work directly added elements to operate. The CGDSA (element added) obtains the most encouraging results compared to other methods since the proposed method managed to extract richer information.

Sensitivity to parameters

Hyperparameter α: After computing the coarse-grained cross-dimensional semantic aggregation matching branch and the fine-grained cross-dimensional semantic dependency matching branch, it is imperative to combine the loss functions of these two branches to ensure the consistency of local and global semantic information. The hyperparameter α serves as a weighting factor to balance these loss functions. Specifically, α adjusts the relative contribution of the two branches to the overall loss, as represented in Eq. (16). Typically α is a scalar value, ranging between 0 and 1. Within the CDGCN framework, α plays a crucial role in balancing the loss of the coarse and fine-grained matching branches. By defining α and dynamically adjusting it during the training process, the optimal parameter value that maximizes the model’s performance can be discovered. Other parameters of the CDGCN remain unchanged during the adjustment of α.

The findings of the ablation experiments are presented in Tables 7, 8, and Fig. 5. When α is set to 0, it indicates that the coarse-grained cross-dimensional global matching branch is not engaged in the calculation, resulting in relatively low performance of CDGCN. Notably, setting α to 0.1 on Flick30k and 0.05 on MSCOCO yields the optimal performance of CDGCN. Conversely, as α rises, the retrieval performance of CDGCN progressively deteriorates. The performance degradation is attributed to an excessive emphasis on global semantic information from images and text, which undermines the effective matching of detailed features in images regions and text words. This imbalance affects the performance of the model and compromises the model’s efficacy. Hence, selecting an appropriate α is crucial for CDGCN. Ultimately, when all parameter values are varied within appropriate ranges, the performance remains relatively stable. This demonstrates the robustness of the CDGCN and its insensitivity to the selection of hyper-parameters.

Generalization study

In cross-modal image-text matching tasks, evaluating a model’s generalization capabilities is critical for ensuring its practicality in real-world applications. The significance of cross-dataset generalization research lies in its potential to enhance the robustness of models across disparate datasets, thereby improving the efficiency of real-world deployments. Assessing model’s performance across multiple datasets offers significant insights into the model’s adaptability to varying environmental conditions–a key factor in scenarios where data diversity is unavoidable. Strengthening this robustness minimizes dependence on specific datasets and significantly enhances the model’s generalization ability, ensuring reliable performance on unseen data and increasing its practical applicability.

This work conducts cross-validation experiments to verify the proposed CDGCN’s generalization ability for overall coarse-fine-grained image-text matching. The experiment involves training baseline models and CDGCN on the MSCOCO dataset, followed by cross-validation on the Flickr30K dataset to test the CDGCN’s zero-shot ability. The detailed outcomes are depicted in Table 9, whether it’s the “Region+BiGRU” version or the “Region+Bert” version, the CDGCN model exhibits stronger generalization capabilities than the baseline model, demonstrating more robust performance. The findings validate the efficacy of the proposed CDGCN in cross-dimensional semantic aggregation and coarse-grained semantic consistency.

The implications of such research extend to numerous practical applications, with models exhibiting robust cross-dataset generalization capabilities being more likely to achieve successful implementation. These models demonstrate consistent efficacy across diverse data environments, mitigating the challenges posed by the variability of data characteristics and thereby accelerating the processes of deployment and optimization. For instance, integrating the CDGCN model into e-commerce recommendation systems allows for user-uploaded images and textual descriptions that may originate from disparate data distributions. By harnessing the insights garnered from cross-dataset generalization research, we can enhance and modify the model according to specific needs, improving its performance in recommendation tasks and enriching the user experience. Furthermore, the pursuit of advancements in cross-dataset generalization has catalyzed innovations in technology and methodology.

Visualization and qualitative result analysis

Visualization of visual and textual features: Utilizing a sample of 1,000 images from the Flicker30K test set, this work utilizes t-SNE (Van der Maaten & Hinton, 2008) to transform the acquired final visual features into a two-dimensional space. Following this, the K-means clustering technique is employed to group these features into 40 clusters. Each cluster is distinguished by a unique color, and the centroid of each cluster is annotated with its respective corresponding group number. The outcomes of visualization are illustrated in Fig. 6A, where this work highlights examples from certain clusters. As demonstrated in Fig. 6A, it is evident that images from different clusters exhibit distinct semantic signs, whereas those from the same cluster display consistent semantic symbols, such as young boys, adorable dogs, baseball and soccer, basketball sports, skiing, hats, all visually similar. For Group 20, featuring adorable dogs, and Group 37, focusing on skiing, there is a significant disparity in the two-dimensional feature map, indicating substantial differences in the features extracted by the proposed model for these two groups. Furthermore, despite Group 2, encompassing baseball and soccer, and Group 10, representing basketball and extreme sports, falling under the category of sports, they are still segregated into two classes due to minor distinctions, shown to be relatively close in the two-dimensional feature map. These visualizations underscore the proposed CDGCN model’s efficacy in effectively extracting global and distinctive local visual features, demonstrating strong global semantic coherence and highlighting its effectiveness.

Concurrently, this work projects the 1,000 images and their corresponding top-ranking sentences into a shared two-dimensional space, as illustrated in Fig. 6B. The distribution of the dimensionality-reduced image vectors and text vectors appears largely similar. However, the visualization also reveals that a minority of image and text features are not mapped to the same distribution positions, reflecting potential matching errors during the model’s retrieval process.

Qualitative retrieval results visualization and analysis: This work conducted qualitative visualization and analysis of retrieval results on the Flickr30K dataset (detailed qualitative visualization and analysis figures available at https://doi.org/10.5281/zenodo.14930469). Due to the requirement of the journal that the retrieval visualization of the Flickr30K dataset must be accompanied by the permission signature of the person in the image, the author of the Flickr30k dataset is also unable to achieve it. Therefore, based on the Flickr30K dataset, the authors used their own portraits to create images (similar text descriptions) that were similar to the scene in the dataset, presenting the qualitative visualization of the model. CGDSA represents the Coarse-Grained Cross-Dimensional Semantic Dependency branch, FGDSD represents the Fine-Grained Cross-dimensional Semantic Aggregation branch, and CDGCN represents the Cross-Dimensional Coarse-Fine-Grained Complementary Network.

Qualitative retrieval results: Fig. 7 illustrates the top-5 text results retrieved by CGDSA, FGDSD, and CDGCN models for three query images, ranked by similarity scores. The proposed CDGCN model has retrieved almost all matching texts correctly (e.g., two out of three images matching perfectly, with only one text mismatch in the last image). Even in cases of incorrect matches, the semantic content of the text (e.g., words like “girls,” “lawn,” and “playing” in the incorrectly matched text for the third image) aligns well with the visual semantics of the image. In comparison, CGDSA and FGDSD models exhibit a higher incidence of mismatched texts. FGDSD method has incorrect matches in the second and three query images (e.g., failing to capture the image information “men and women” and incorrectly matching it with the text “several girls” in the second image). CGDSA method has mismatches in the first and third query images (e.g., not capturing the color of the hat and whether there are earrings in the first image, leading to incorrect matches with texts “brown and white hat” and “beige hat”). These findings indicate that the sentences retrieved by the CDGCN method provide detailed descriptions of visual scenes, suggesting that the image representations obtained by introduced model effectively capture rich dimensional feature information, avoiding mismatching issue by the complementary of coarse-fine-grained information.

As illustrated in Fig. 8, three sets of query texts are provided, each of which retrieves the top three images through CGDSA, FGDSD, and CDGCN models. These images are displayed from left to right by their degree of matching. The images enclosed in red boxes signify the correct retrievals derived from the given texts. This illustration demonstrates that the CDGCN model can largely retrieve the correct images (correctly retrieving two out of three texts, with one incorrect retrieval). For straightforward text-to-image retrieval tasks, all three models perform satisfactorily. However, for images that strongly emphasize the overall layout and comprehensive information (such as the first query text), both CGDSA and CDGCN adeptly match the correct images. In contrast, FGDSD overemphasizes minute details like “cake hat” and “candles are inserted into the cake,” thereby overlooking other contextual elements, leading to erroneous matches. In cases with extensive, intricate details (the second query text), CDGCN and FGDSD exhibit their strengths by accurately matching these details. Although CGDSA can also capture cross-dimensional feature information for broad-scale matching, it fails to achieve precise matching for certain detailed elements, resulting in failed matches. The analysis above underscores that the CDGCN method achieves a high matching rate in text-to-image retrieval, adeptly synthesizing local and global information to complement each other, thereby retrieving the ground truth images with greater accuracy.

Error analysis of retrieval results: As depicted in Fig. 7, the third given image resulted in errors across all three models: CGDSA, FGDSD, and CDGCN. Specifically, the text retrieved by the CDGCN method (Sentence 3) was mismatched, failing to align with the provided image. Despite this, the words “girls,” “lawn,” and “playing” within the sentence retained a high semantic relevance to the image content, carrying substantial semantic information closely related to the image. However, the actual content did not match due to discrepancies in minute details. The image showed three girls are lying on the lawn, while the retrieved text, though containing “girls,” “lawn,” and “playing,” omitted the crucial detail “setting.” For the missing correct text, while it included semantically relevant terms like “girls” and “lawn,” it failed to accurately capture verbs such as “huddled together” and “listening”, resulting in an inability to match the image. Similarly, as shown in Fig. 8 for text-to-image retrieval, given Text 3, all three models—CGDSA, FGDSD, and CDGCN—produced erroneous results. Specifically, the first two images retrieved by the CDGCN method are mismatched, while the third image is correctly matched. All three images were visually highly related, depicting a girl riding a bike, with different dressing details, movements, and categories of bikes. Figure 9 visualized examples of text-to-image retrieval errors, where words such as “girl,” “bike,” and “road” aligned well with corresponding visual regions in the images (though interpretations of “bike” varied). However, as seen in Fig. 9, third image (c), the verb “pedaling” lacked detailed semantic capture in the image, reducing the text-to-image match accuracy. Comparing the corresponding texts of the first two images (Text in black font in Fig. 9) with the query text revealed that the query sentence is relatively simple, encompassing basic semantic concepts found in all three images. The key verbs or actions in the first two images, “stepped,” “braking,” and “exploring,” “cruising,” are crucial semantic details missing from the query sentence, which the CDGCN failed to identify accurately.

Based on the above analysis, the errors in retrieval results may stem from the CDGCN model’s issues in capturing verbs, actions, and anthropomorphized vocabulary in both images and texts, overlooking minor details that diminish matching accuracy. Enhancing feature extraction performance, strengthening the relationships between words and verbs or prepositions to capture more local features, and improving the model’s architecture to enhance global contextual understanding (including anthropomorphized sentences) could potentially elevate the accuracy of image-text retrieval.