Adversarial feature learning for semantic communication in human 3D reconstruction

Introduction

With the rapid development of virtual reality (VR), augmented reality (AR), and autonomous driving technology, 3D reconstruction technology has gradually become the focus of scientific research and industrial applications. These technologies have a wide range of application prospects in the fields of digital media, online education, remote sensing monitoring, medical imaging, etc., and can provide a more immersive and interactive user experience (Zhou & Tulsiani, 2023; Anciukevicius et al., 2023). For example, in the field of medical imaging, 3D reconstruction can help doctors analyze and diagnose more accurately (Cai et al., 2013; Wen et al., 2014). In urban planning and disaster assessment, 3D map reconstruction through remote sensing technology can provide more effective decision support. However, 3D reconstruction usually requires processing and transmitting large amounts of data, which is particularly difficult in environments with limited bandwidth or high network latency. In addition, high-quality 3D reconstruction often relies on powerful computational resources, which limits its usefulness in mobile devices or real-time application scenarios (Wu et al., 2023; Melas-Kyriazi, Rupprecht & Vedaldi, 2023).

In order to overcome these limitations, this article proposes a semantic communication approach for 3D reconstruction of the human body. Semantic communication is an innovative communication paradigm (Xu et al., 2023b; Weng et al., 2023; Barbarossa et al., 2023) that focuses on transmitting information that is critical for accomplishing a specific task rather than simply transmitting large amounts of raw data. By transmitting only the most critical semantic information, the method can significantly reduce the amount of data required, thereby increasing transmission efficiency and reducing bandwidth dependency (Nguyen et al., 2011; Zhang et al., 2023; Xu et al., 2023a). The use of semantic communication for human 3D reconstruction not only guarantees the quality of the reconstruction but also significantly reduces the time required for data transmission and processing, which is particularly important for application environments that have restricted network conditions or limited computational resources. Potential applications of this technology include online gaming, telemedicine diagnosis, and holographic projection technology, which all require processing and transmitting large amounts of 3D data while ensuring real-time and high interactivity. By optimizing the data flow, semantic communication brings new development opportunities to these fields, making high-quality 3D reconstruction possible in a wider range of application scenarios.

How to accurately recover detailed human 3D data from limited semantic information and how to design efficient algorithms to achieve stability and reliability under different network and device conditions. In addition, how to balance the efficiency and accuracy between semantic information extraction, transmission, and reconstruction is also a key issue to be addressed in current research (Kang et al., 2023; Yao et al., 2023; Nam et al., 2024). The method in this article accurately extracts the key semantic information that can be used for human 3D reconstruction at the sending end and efficiently transmits it to the receiving end through an adaptive compression algorithm, and then accurately reconstructs the human 3D model at the receiving end by utilizing semantic decoding and 3D reconstruction methods. This strategy not only improves the speed of data processing but also ensures the accuracy and low latency of the reconstruction process. The method in this article can provide a new solution for real-time human 3D reconstruction technology, especially in application environments where bandwidth is limited and real-time response is required, demonstrating its unique advantages and potential prospects for a wide range of applications.

Human 3D reconstruction typically requires processing and transmitting large amounts of data, which is particularly challenging in environments with limited bandwidth or high network latency. In addition, huge computational resources are often required to achieve high-quality reconstruction results, which limits its wide application in mobile devices or application scenarios that require real-time responses. To address the above challenges, our proposed AFLSC method not only optimizes the data flow but also ensures the integrity of the information and the high quality of the reconstruction by accurately extracting the semantic information that is crucial for human 3D reconstruction and efficiently transmitting it through adaptive compression techniques, and the main contributions are summarized as follows:

• Innovative semantic encoding mechanism: This study develops a semantic encoding strategy optimized for 3D reconstruction of the human body, which is able to convert 2D images into key semantic data oriented to 3D reconstruction at the sending end. This approach reduces the transmission of irrelevant data and ensures efficient encoding and transmission of key information.

• Dynamic compression rate adjustment: this method adaptively adjusts the data compression rate according to the importance of the semantic data and the current network status (bandwidth and latency) by means of dynamic compression technology, which optimizes the data transmission process and significantly reduces the latency while ensuring the integrity of the data and the reconstruction quality.

• Efficient semantic decoding and human 3D reconstruction: at the receiver side, our method utilizes the received semantic data, decodes the semantic data into key feature information, and reconstructs the 3D model using these key feature information. Such an approach reduces the computational burden at the receiver side and at the same time ensures the reconstruction quality of the 3D model.

Related work

System modeling

In this study, we design a semantic communication system for 3D reconstruction of the human body to realize the process starting from the input of the original 2D image I at the sender’s end to the final generation of the 3D model at the receiver’s end. The system covers feature extraction and semantic coding of images, compression and transmission of semantic data, and 3D reconstruction based on the received semantic data.

On the sender side, 2D human body images I are used as input and these images are first processed through Vision Transformer (ViT). At this stage, the images are divided into multiple blocks (patches), and each block is processed by ViT to extract key features related to 3D reconstruction of the human body, such as shape, spatial layout, pose, and depth information, to obtain a high-dimensional feature vector X. These features are further semantically encoded by Conditional Variational Auto-Encoders (CVAEs) (Sohn, Lee & Yan, 2015), which, in combination with an adversarial learning approach, generates a labeled dataset $y^{\mathrm{\prime }}$ with rich semantic information. In order to be transmitted from the sender to the receiver, this semantic data $y^{\mathrm{\prime }}$ needs to be compressed. We employ a dynamic compression strategy that dynamically adjusts the compression rate based on the current network conditions (e.g., bandwidth and latency) and the importance of the data. This strategy ensures efficient and stable transmission of semantic data in various network environments.

At the receiver side, the system first decodes the received semantic data $y^{\mathrm{\prime }}$ to obtain potential features $z^{\mathrm{\prime }}$ and then converts these potential features back to image features $s_i$ through a multi-level decoding process. This process involves multiple decoding layers, each of which gradually reconstructs the detailed features of the image by utilizing the output of the previous layer and the potential state of the current layer. After obtaining the image features, we use a modified ViT-diffusion model to generate a 3D point cloud, a process that progressively refines the point cloud data through a diffusion and inverse diffusion process, and finally converts the point cloud into a fine 3D mesh model using the Marching Cubes algorithm.

The overall system structure is shown in Fig. 1.

Feature extraction and semantic encoding

In order to enable 3D reconstruction of the human body at the receiver side, it is crucial to efficiently extract and encode the 2D human image features at the transmitter side. These features need to contain not only information about the shape and spatial layout of the object but also capture task-specific details such as the pose and relative depth of the object. Therefore, this chapter aims to explore how to extract key features from complex data and transform them into useful semantic information through advanced deep learning models. We first parse how the ViT model can be adapted and optimized for different visual tasks based on multi-task learning feature extraction techniques. Next, we delve into semantic encoding methods based on adversarial feature learning.

Semantic encoding based on adversarial feature learning

For the feature X extracted by our multi-task based learning approach, in order to efficiently encode deep semantic information from X, we propose an adversarial feature learning approach for semantic encoding that integrates a conditional variational auto-encoder (CVAE) and generative adversarial networks (GANs), where the CVAE part is responsible for extracting meaningful latent representations from the input features and the GAN part is used to optimize these representations through adversarial learning to generate more realistic and semantically consistent outputs.

The approach consists of two main components: the CVAE and an auxiliary adversarial discriminator, working together to accurately generate and validate semantic labels. The encoder is designed to capture the joint probability distribution between the input features X and the target semantic labels $y$ (used only in the training phase) and efficiently maps this information into a latent space $z$. The process learns the mean ${\mu }_{\varphi }\left(X,y\right)$ and the variance ${\sigma }_{\varphi }{\left(X,y\right)}^2$ of the input data over the network and is characterized as:

(8) $$q_{\varphi }\left(z|X,y\right)=\mathcal{N}\left(z;{\mu }_{\varphi }\left(X,y\right),{\sigma }_{\varphi }{\left(X,y\right)}^{2}I\right),$$where ${\mu }_{\varphi }\left(X,y\right)$, ${\sigma }_{\varphi }\left(X,y\right)$ represent the centrality and dispersion of potential features, respectively, thus describing the conditional distribution of the latent variable $z$.

The task of the decoder is to reconstruct the input features X from the latent variables $z$ and predict as accurately as possible the semantic labels $y$. The decoder combines continuous data reconstruction with discrete label prediction in the expression:

(9) $$p_{\theta }(X,y\mid z)=\mathcal{N}(X;{\mu }_{\theta }(z),{\sigma }_{\theta }(z{)}^{2}I)\times \mathrm{C}\mathrm{a}\mathrm{t}(y;{\pi }_{\theta }(z)),$$where $\mathcal{N}(X;{\mu }_{\theta }(z),{\sigma }_{\theta }(z{)}^{2}I)$ is a Gaussian distribution that describes the process of reconstructing the input feature X from the latent variable $z$. ${\mu }_{\theta }(z)$ and ${\sigma }_{\theta }(z)$ define the distribution of the reconstructed data, while ${\pi }_{\theta }(z)$ provides the probability of generating a specific semantic label from the latent variable $z$. $\mathrm{C}\mathrm{a}\mathrm{t}(y;{\pi }_{\theta }(z))$ is a categorical distribution for predicting discrete semantic labels $y$ based on the latent variable $z$’s current state. $z$’s current state to predict the discrete semantic label $y$.

The role of the discriminator is to distinguish the differences between the samples generated by the decoder and the real samples, and to evaluate whether these generated samples are consistent in content with the input features X. The discriminator is a tool that can be used to evaluate the content of the samples generated by the decoder and the real samples. The operation of the discriminator is based on the following equation.

(10) $$D(\tilde{X},\tilde{y})=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{m}\mathrm{o}\mathrm{i}\mathrm{d}(f(X,\tilde{X},y,\tilde{y})),$$where $\tilde{X}$ and $\tilde{y}$ are the samples and labels generated by the decoder, respectively, and $f$ is the discriminative network.

The training objective of this semantic coding model for adversarial feature learning is optimized through the composition of two main loss functions, a CVAE loss and an adversarial loss, and we combine the CVAE loss and the adversarial loss into a single comprehensive loss function, $L_{\mathrm{c}\mathrm{o}\mathrm{m}}$, which can optimize the model’s reconstruction ability and the realism of the generated samples in a more effective and simultaneous way. This loss function $L_{\mathrm{c}\mathrm{o}\mathrm{m}}$ aims to optimize the performance of the entire model by balancing the reconstruction error, $\mathrm{K}\mathrm{L}$ scatter, and adversarial evaluation with a single optimization objective. The adversarial loss consists of two parts, one that encourages the discriminator to accurately recognize real samples, and the other that motivates the discriminator to recognize the samples generated by the generator, driving the authenticity and semantic consistency of the generated samples. The integrative loss function is defined as follows:

(11) $$\begin{array}{rl}{L}_{\mathrm{c}\mathrm{o}\mathrm{m}}=& \mathrm{M}\mathrm{S}\mathrm{E}(X,{\mu }_{\theta }(z))+\beta \mathrm{K}\mathrm{L}(q_{\varphi }(z\mid X,y)||p(z))+\gamma (\log D(X,y)\\ & +\log (1-D({\mu }_{\theta }(z),{\pi }_{\theta }(z)))).\end{array}$$

Here $\mathrm{M}\mathrm{S}\mathrm{E}$ is used to quantify the accuracy of the reconstruction, while $\mathrm{K}\mathrm{L}$ scatter helps to regularize the latent space. $\beta$ and $\gamma$ are moderating terms used to balance the influence of the different parts of the loss function to suit specific training needs. $\beta$ controls the influence of $\mathrm{K}\mathrm{L}$ scatter, which is typically used to regulate the encoding constraints in the latent space. $\gamma$ adjusts the weight of the adversarial loss to ensure that the generated samples are not only true, but also match the expected label semantics.

In the inference phase of the model, we generate the corresponding semantic labels $y^{\mathrm{\prime }}$ directly from the input features X based on the trained encoder and decoder, without the need for external label input:

(12) $$z={\mu }_{\varphi }(X),$$

(13) $$y^{\mathrm{\prime }}=\arg max({\pi }_{\theta }(z)).$$

This adversarial feature learning method effectively integrates the deep data encoding capability of CVAE and the generation quality optimization capability of GAN through its innovative architecture, forming a powerful semantic encoding system. Through the design of the integrated loss function $L_{\mathrm{c}\mathrm{o}\mathrm{m}}$, the model is able to ensure the reconstruction accuracy while further improving the authenticity and semantic consistency of the generated samples through adversarial training.

The schematic diagram of the feature extraction and semantic coding process is shown in Fig. 2, and Algorithm 1 presents the corresponding algorithmic pseudo-code.

Algorithm 1:

Feature extraction and semantic encoding.

1: Input: 2D human image I

2: Output: Semantic labels y′

3: procedure FEATURE EXTRACTION (I)

4:      Divide I into patches $\{P_i\}$

5:      $X\leftarrow$ Apply ViT on $\{P_i\}$ to obtain feature vectors

6:      Enhance position encoding: $E_{pos}^{task}=E_{pos}+\mathrm{\Delta }{E}_{task}$

7:     for each task $t\in \{\mathrm{k}\mathrm{e}\mathrm{y}\mathrm{p}\mathrm{o}\mathrm{i}\mathrm{n}\mathrm{t},\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{e},\mathrm{d}\mathrm{e}\mathrm{p}\mathrm{t}\mathrm{h}\}$ do

8:          Integrate task-specific heads into ViT

9:          $X_L\leftarrow$ Output from layer L of ViT

10:          Apply task-specific normalization and layers

11:     end for

12:      $Y_k,Y_{pose},Y_{depth}\leftarrow$ Compute outputs for each task

13: end procedure

14: procedure SEMANTIC ENCODING ( $Y_k,Y_{pose},Y_{depth}$)

15:      Encode features using CVAE:

16:      $z\sim q_{\varphi }(z|X,y)=\mathcal{N}(z;{\mu }_{\varphi }(X,y),{\sigma }_{\varphi }^2(X,y)I)$

17:      Decode features and predict labels:

18:      $p_{\theta }(X,y|z)=\mathcal{N}(X;{\mu }_{\theta }(z),{\sigma }_{\theta }^2(z)I)\times \mathrm{C}\mathrm{a}\mathrm{t}(y;{\pi }_{\theta }(z))$

19:      Optimize using GAN:

20:      Train discriminator D to distinguish between generated and real samples

21:      minimize ${\mathcal{L}}_{com}=\mathrm{M}\mathrm{S}\mathrm{E}(X,{\mu }_{\theta }(z))+\beta \mathrm{K}\mathrm{L}(q_{\varphi }(z|X,y)\parallel p(z))+\gamma (\log D(X,y)+\log (1-D({\mu }_{\theta }(z),{\pi }_{\theta }(z))))$

22: end procedure

23: $y^{\mathrm{\prime }}\leftarrow \arg max({\pi }_{\theta }(z))$         ▹ Generate semantic labels from features

DOI: 10.7717/peerj-cs.2731/table-1

Semantic transfer based on dynamic compression rate adjustment

Once the semantically labeled data $y^{\mathrm{\prime }}$ is extracted from the 2D human body image at the sender, it is particularly important to transmit the data efficiently and reliably to the receiver. Considering the dynamic nature of network environments, especially in mobile or international communication scenarios, transmission efficiency and stability are often challenged by bandwidth fluctuations and delay variations. In order to optimize the transmission of semantic data, this article proposes a dynamic compression rate adjustment method, which is specifically adapted and optimized for semantically tagged data $y^{\mathrm{\prime }}$ to ensure that high efficiency and data integrity are maintained under different network conditions.

Considering that the semantic label $y^{\mathrm{\prime }}$ may be closely related to the criminality of a particular task, we define a formula to quantify the importance of each label, with attributes of the label containing type, frequency, and contextual relevance.

Suppose that each tag is associated with an importance weight ${\omega }_i$ that reflects its importance in a particular task. We define this weight using the following formula:

(14) $${\omega }_i=\tau \cdot \exp \left(-\frac{d\left(y_i^{\prime },Y_{contex}\right)}{\sigma }\right),$$where $\tau$ is a task-dependent adjustment factor with respect to the type of the label. $d\left(y_i^{\mathrm{\prime }},Y_{context}\right)$ denotes the average distance between the label $y_i^{\mathrm{\prime }}$ and its contextual label set $Y_{contex}$, and $\sigma$ is a normalization factor to adjust for the effect of distance.

We can get instant feedback on current network conditions by periodically sending control packets and measuring their response times. This data helps us evaluate the network congestion level and transmission speed in real time, which provides the basis for the next compression rate adjustment. Based on the real-time monitored bandwidth and latency data, we optimize the compression rate of semantically tagged data $y_i^{\mathrm{\prime }}$ using a dynamic tuning method that uses an exponentially weighted moving average (EWMA) approach to smooth the bandwidth and latency measurements, thereby reducing the impact of abrupt changes on the compression rate tuning. The expressions for bandwidth and delay monitored by the method are given below:

(15) $$B(t)=\text{ϕ}B(t-1)+(1-\text{ϕ})\frac{{\sum }_{i=1}^n{S}_i}{{\sum }_{i=1}^n\mathrm{\Delta }{t}_i},$$

(16) $$D(t)=\frac{{\sum }_{i=1}^m{w}_i\cdot t_{RTT,i}}{{\sum }_{i=1}^m{w}_i},$$where $S_i$ and $\mathrm{\Delta }{t}_i$ represent the packet size and time interval of the $i$-th transmission within the time window, respectively, $\text{ϕ}$ is the attenuation factor of the bandwidth measurement, and $w_i$ is the weight of the $i$-th response time in the delay measurement. In this way, we are able to obtain smoother and more reliable estimates of the network state, providing accurate inputs for compression rate tuning. Using the importance weights $w_i$ obtained above, we design a dynamic compression rate adjustment formula that is not only based on the importance of labels but also considers the real-time network condition:

(17) $$\rho (y_i^{\mathrm{\prime }})={\rho }_{min}+({\rho }_{max}-{\rho }_{min})\cdot \left(1-\frac{{\omega }_i}{max(W)}\right)\cdot f\left(B(t),D(t)\right),$$where ${\omega }_i$ is the importance weight of semantic label $y_i^{\mathrm{\prime }}$ and W is the set of importance weights of all labels, and $f\left(B(t),D(t)\right)$ is a network bandwidth and latency based adjustment function defined as follows:

(18) $$f\left(B(t),D(t)\right)=a\left(\frac{B_{max}-B(t)}{B_{max}-B_{min}}\right)+b\left(\frac{D(t)-D_{min}}{D_{max}-D_{min}}\right),$$where $a$ and $b$ are coefficients to balance the effects of bandwidth and delay on the compression rate; $B_{max}$, $B_{min}$, $D_{max}$, $D_{min}$ denote the maximum and minimum values of bandwidth and delay, respectively, which are used to normalize these measurements.

This dynamic adjustment strategy ensures the flexibility and adaptability of semantic data transmission, which can intelligently optimize the data flow according to the changing network conditions and the importance of the data. With this dynamic compression technique, the transmission of semantically labeled data $y^{\mathrm{\prime }}$ is more efficient and stable.

3D Reconstruction based on Semantic Data

Semantic transfer allows us to use the semantic data received at the receiver side for 3D human reconstruction. This process involves two key techniques: firstly, multilevel semantic feature decoding, and secondly, improved ViT-diffusion modeling for 3D reconstruction. We will describe in this chapter how to recover image features from semantic tags and utilize these features to guide the generation of 3D models.

Experiments and results

Results and discussion

In Experimental Scenario A, we demonstrate the advantages of the AFLSC method in terms of dealing with latency and improving transmission efficiency by comparing the performance of different 3D reconstruction methods in terms of total task time. As can be seen from the experimental data in Fig. 3, the total task time of all methods grows gradually as the number of 3D models increases. However, AFLSC consistently shows lower latency than AFLSC-without SC, NeRF, and PIFuHD. Specifically, AFLSC exhibits lower latency at all model number levels compared to AFLSC without SC, which uses traditional data transfer methods. This advantage suggests that AFLSC is able to process and compress data more efficiently through an optimized semantic transfer approach, thereby reducing the transfer time. In particular, AFLSC shows significantly lower latency in comparison with PIFuHD and NeRF, which is in part due to the fact that PIFuHD and NeRF may need to process more data details, resulting in larger data volumes and longer processing times.

The strength of AFLSC stems from its use of a semantic communication strategy that transmits only the most critical information for accomplishing a given task, greatly reducing the size of the data that needs to be transmitted. This strategy is particularly effective in environments with bandwidth constraints or unstable network conditions. In addition, although AFLSC and AFLSC-without SC use the same 3D reconstruction technique, the optimized transmission method makes AFLSC more efficient in the actual data transmission process.

In Experimental Scenario A, we compare the performance of AFLSC with four methods, AFLSC-without SC, NeRF, and PIFuHD, under different network bandwidth conditions. The experimental results in Fig. 4 show that the AFLSC method has the shortest total task time for all bandwidth settings, thus clearly demonstrating its significant advantages in improving data transfer efficiency and reducing latency. The total task time of AFLSC at 1 Mbps bandwidth is 15.7% faster than that of AFLSC-without SC, 21.0% faster than NeRF, and PIFuHD by 31.0%. This significant performance difference indicates that AFLSC significantly improves data processing efficiency through effective semantic compression and transmission optimization. In addition, the total task time decreases for all methods as the bandwidth increases. At the high bandwidth setting of 10 Mbps, AFLSC reduces the time by 3.9%, 22.4% and 28.9% compared to AFLSC-without SC, NeRF and PIFuHD, respectively. This shows that even under the favorable conditions of high bandwidth, the optimized transmission strategy of AFLSC still provides additional performance gains.

The superiority of AFLSC is mainly attributed to its ability to efficiently encode 2D image data into the required semantics suitable for 3D reconstruction, and to effectively compress the amount of semantic data, making it possible to rapidly transmit critical data even under low bandwidth conditions, instead of unoptimized transmission of the complete data in traditional approaches. This optimization is particularly suitable for bandwidth-constrained or application scenarios requiring high real-time performance, such as online gaming and telemedicine diagnostics, where reduced latency directly improves user experience and system responsiveness.

In Experimental Scenario A, we compared the quality metrics, including MSE and SSIM, of AFLSC with AFLSC-without SC, NeRF, and PIFuHD in 3D reconstruction of different human movement types. From the perspective of the MSE data in Fig. 5, AFLSC presents lower error values in the reconstruction of each movement type, especially in the “Running” and “Sitting” movements, with MSE values of 0.0650 and 0.050, respectively, compared with the results of NeRF and PIFuHD, which shows that AFLSC is more capable of accurately capturing important details when dealing with dynamic movements and complex poses. For example, in the “Running” action, AFLSC has about 25.7% lower error than PIFuHD, a difference that emphasizes the efficiency of AFLSC in preserving the dynamic features of the action. In terms of the SSIM data results in Fig. 6, AFLSC also shows high model similarity, especially in the “Running” and “Standing” movements, with SSIM values of 0.82, which are about 17.1% and 17.7% higher than that of PIFuHD, respectively. The SSIM values were 0.82 for the “Running” and “Standing” movements, which were about 17.1% and 15.5% higher than those of PIFuHD. This shows that AFLSC not only can accurately reconstruct the model, but also can be visually closer to the original model, providing a more natural and coherent visual effect. In addition, even when other methods perform better, e.g., the SSIM of NeRF in the “Walking” action is 0.79, the AFLSC’s 0.77 is still competitive, which proves its stable performance in different scenarios.

The experimental results show that AFLSC demonstrates superior reconstruction quality in most cases, especially in MSE and SSIM. The AFLSC method is able to encode 2D images into key semantic data suitable for 3D reconstruction at the sender’s end, so that the receiver’s end, after receiving this semantic information, is able to use the semantic data that contains the key information of the original image to perform 3D reconstruction, ensuring the quality of 3D reconstruction.

The purpose of Experimental Scenario B is to compare the performance of our AFLSC approach with other semantic communication technologies such as ISS and DeepMA, especially for applications in 3D reconstruction tasks. Through these comparisons, we can demonstrate the advantages of AFLSC in terms of data transfer efficiency and reconstruction quality preservation in bandwidth-constrained environments. As can be seen from the experimental data in Fig. 7, the total task time for all methods grows as the number of 3D model reconstruction tasks increases. However, AFLSC consistently outperforms ISS and DeepMA in task completion time for all model counts, highlighting its efficiency in processing 3D reconstruction semantic data. For example, in the 10-model reconstruction task, AFLSC has a completion time of 176.0 s, while ISS and DeepMA take 194.5 and 190.5 s, respectively. This indicates that AFLSC is 9.5% and 7.6% faster than ISS and DeepMA, respectively, highlighting the superiority of AFLSC in terms of data processing and processing efficiency.

A key advantage of AFLSC is its ability to efficiently encode semantic data optimized for 3D reconstruction at the transmitter side, thus allowing the reconstruction process to proceed rapidly at the receiver side. In contrast, ISS and DeepMA, while adept at transmitting compressed image data, do not focus on semantic data optimized specifically for 3D tasks. This leads to the fact that they need to reconstruct the image before reconstructing the 3D model, thus relatively prolonging the reconstruction time.

We compare the total 3D model reconstruction task time of our proposed AFLSC method with two semantic communication techniques, ISS and DeepMA, under different network bandwidth conditions in experimental scenario B conditions. As shown in Fig. 8, the comparison of the experimental data demonstrates the significant advantages of AFLSC in terms of data transmission efficiency and reconstruction quality maintenance in bandwidth-constrained environments. From the experimental data, it can be seen that the task completion time of all methods decreases as the bandwidth increases, but AFLSC consistently outperforms ISS and DeepMA at all bandwidth levels. For example, at a bandwidth of 1 Mbps, AFLSC completes the task in 142.8 s, which is 16.0% faster than ISS and 14.5% faster than DeepMA. At the highest bandwidth of 10 Mbps, AFLSC’s time is 38.9 s, which is 17.4% faster than ISS and 15.6% faster than DeepMA. In addition, AFLSC demonstrates greater performance gains in lower bandwidth settings, suggesting that it is more efficient at compressing and transferring critical semantic information. This is particularly important for application scenarios that require real-time or near real-time 3D reconstruction, such as teleoperation and interactive media.

The reduction in the total task time also shows the advantages of AFLSC in terms of compression and coding efficiency of semantic data, enabling it to perform the 3D reconstruction task faster in bandwidth-constrained situations. In contrast, ISS and DeepMA are inferior to AFLSC in semantic communication specifically for 3D reconstruction, although they also employ semantic communication strategies.

In Experimental Scenario B, we evaluate the ability of each method in maintaining the quality of 3D model reconstruction by comparing SSIM. From the experimental results in Fig. 9, it is clear that AFLSC outperforms ISS and DeepMA in different movement categories, e.g., in the “Walking” category, the SSIM of AFLSC is 0.80, which is significantly higher than that of ISS (0.66) and DeepMA (0.72), and in the “Jumping” movement, the SSIM of AFLSC is 0.78, compared with that of ISS and DeepMA (0.78). Similarly, in “Jumping”, the SSIM of AFLSC was 0.78, compared to 0.61 for ISS and 0.65 for DeepMA. This advantage is demonstrated across the action types tested. AFLSC achieves its advantage mainly due to its efficient semantic encoding mechanism that accurately captures and transmits the visual and geometric information that is essential for 3D reconstruction. In contrast, ISS and DeepMA, although they also employ semantic transmission of images and are able to reconstruct images at the receiver side and use NeRF for 3D reconstruction, are not as efficient as AFLSC exhibits in terms of data transmission efficiency and reconstruction quality maintenance. In addition, AFLSC outperforms ISS and DeepMA in adapting to different action types and maintaining the reconstruction quality, which is mainly attributed to its fine-grained processing of semantic information during model training and optimization. By precisely controlling the content of the transmitted information, AFLSC not only improves the utilization efficiency of network resources, but also optimizes the detailed performance during the reconstruction process, making the final 3D model more accurate and natural.

Conclusions

In this study, we propose an adversarial feature learning-based semantic communication method for human 3D reconstruction, aiming to address the challenges faced by traditional 3D reconstruction techniques in bandwidth-constrained and high-latency environments. Our approach significantly improves the efficiency of data transmission and the accuracy of the reconstruction process by efficiently extracting and transmitting semantic information critical for 3D reconstruction. We develop an innovative semantic encoding mechanism that converts human 2D images into compressed semantic data containing the necessary 3D reconstruction-oriented data at the sender’s end. This step not only optimizes the data flow, but also reduces bandwidth dependency, enabling high-quality 3D reconstruction even in bandwidth-constrained environments. By introducing a dynamic compression rate adjustment strategy, our system intelligently adjusts the compression rate according to network conditions and data importance, which not only reduces transmission latency but also ensures the integrity of the transmitted data. This feature is lacking in traditional 3D reconstruction methods and provides technical support for efficient and stable remote 3D modeling. In addition, an efficient semantic decoding and reconstruction process at the receiver side enables fast and accurate recovery of detailed 3D models from compressed semantic data. Our method reduces the computational burden at the receiver side while ensuring accurate reconstruction of 3D models, which is particularly suitable for mobile devices or real-time applications with limited computational resources. Through a series of experiments, we demonstrate that our method outperforms the prior art in several aspects. The experimental results show that this study demonstrates significant advantages in both data transfer efficiency and reconstruction quality. The semantic communication strategy and 3D reconstruction method proposed in this article provide a new possibility to realize high-quality 3D reconstruction under various network and device conditions in the future. Future work will focus on further optimizing the performance of the algorithm, extending it to more application scenarios, and solving more kinds of 3D reconstruction tasks. We will also explore applying this approach to other types of data and tasks to validate its generalizability and scalability.

Supplemental Information