3D reconstruction of toys based on adaptive scaled neural radiation field

Model architecture

The adaptive scale neural radiance fields (AS-NeRF) model proposed in this study aims to comprehensively improve the accuracy and quality of 3D reconstruction of toys by integrating multi-task learning feature extraction, adaptive scale adjustment with spectral sensing, and conditional diffusion model. The overall architecture is shown in Fig. 1. AS-NeRF consists of three main components, each of which plays a key role in the overall reconstruction process and enables efficient 3D reconstruction through close collaboration.

The multi-task learning feature extraction network (MTL-FeatureNet) is responsible for extracting multidimensional features, including texture, shape, color, and depth, from the input single 2D toy view. These rich features provide the necessary contextual information for subsequent 3D reconstruction. MTL-FeatureNet is based on an improved Vision Transformer (ViT) architecture, which optimizes the feature extraction process by introducing a task-dynamic modulation layer, allowing it to better adapt to multi-task learning environments and enhancing the ability to extract high-quality features from a single view.

The extracted multidimensional features are then passed to an improved NeRF model that integrates an adaptive scaling mechanism and spectral sensing techniques. The adaptive scaling mechanism dynamically optimizes the rendering accuracy according to the complexity of toy shapes and details, ensuring accurate reconstruction of details at different scales, thus improving the accuracy and computational efficiency of 3D reconstruction. Spectral awareness technology introduces a spectral decoder, which uses multi-spectral information to enhance the color realism and material details, so that the generated 3D model has higher realism and details under various lighting conditions.

To further enhance the quality and consistency of the generated images, AS-NeRF integrates a conditional diffusion model. The model utilizes multidimensional features from MTL-FeatureNet and the improved NeRF model as conditional information to guide the backward diffusion process, ensuring consistency and accuracy of the generated viewpoint images in terms of geometric and physical properties. The conditional diffusion model not only optimizes the details of image generation, but also enhances the reconstruction ability of the model under few viewpoint conditions, enabling the generation of high-quality 3D models even with limited viewpoint data.

Figure 1 shows the overall architecture of the AS-NeRF method. In the AS-NeRF framework, a single-view image is extracted by MTL-FeatureNet to produce a local feature map and global semantic vectors. The local feature maps are sent to NeRF with ray sampling to guide the density and color prediction of each 3D sampling point. The global vectors are involved in adaptive scale selection and spectral decoding in NeRF, and on the other hand, they are compressed into conditional vectors and input into the conditional diffusion model, which are used to constrain the diffusion process and synthesize the unobserved viewpoint images quickly. In the training stage, the new viewpoints generated by the diffusion model are projected back to NeRF as pseudo-supervision, while in the inference stage, only one feature extraction is needed to render the real viewpoints of NeRF and synthesize the arbitrary viewpoints of the diffusion model at the same time, and the two paths share the features without blocking each other, which ensures that geometrically and spectrally consistent and high-quality 3D reconstruction results can be obtained under the condition of sparse viewpoints.

Feature extraction network based on multi-task learning approach

In this section, we describe how to extract multilevel features such as texture, shape, color, and depth from a single 2D toy view by MTL-FeatureNet to provide the necessary information for complex 3D reconstruction tasks.

MTL-FeatureNet is mainly based on the ViT architecture (Han et al., 2022; Liu et al., 2021; Arnab et al., 2021), but in MTL-FeatureNet we improve the traditional multi-head self-attention mechanism to optimize the extraction of features from a single 2D toy view, especially in a multi-task learning environment. The improvements include the introduction of a task dynamic modulation layer, which is a design that adjusts the self-attention weights according to the task demands. The purpose of the task dynamic modulation layer is to dynamically adjust the attention allocation according to the specific demands of different tasks. This layer optimizes the weight distribution of each head in the attention mechanism by analyzing task-related contextual information. The specific implementation is as follows.

We assume that each task is computed by a neural network module to obtain a task relevance score $T{R}_i$, which is used to reflect the importance of the current feature for a particular task:

(1) $$T{R}_i=ML{P}_{task}\left(C_i\right).$$

$ML{P}_{task}$ refers to a multilayer perceptron that receives high-level features from a specific part of the image and outputs a real value that indicates the importance of the current feature for performing a specific task. $C_i$ refers to regional or global features in the image after preprocessing (e.g., feature extraction network layer processing).

The above scores are then used to adjust the self-attention weights so that the model can focus on the most relevant information according to the specific needs of each task:

(2) $$A{t}_i\left(Q,K,V\right)=softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}\cdot exp\left(T{R}_i\right)}\right)V.$$

Based on the task relevance score $T{R}_i$, we act on the attention weights in the form of $exp\left(T{R}_i\right)$ to enhance or diminish the influence of specific parts to ensure that the model’s attention is more in line with the current task.

Since positional information is extremely important for understanding image content, we adopt an improved positional coding method, which not only considers pixel positions, but also incorporates task-related dynamic adjustments:

(3) $$PE\left(pos,2i\right)=sin\left(pos/{10\text{,}000}^{2i/d_{model}}\right)\cdot TR,$$

(4) $$PE\left(pos,2i+1\right)=cos\left(pos/{10\text{,}000}^{2i/d_{model}}\right)\cdot TR.$$

Since $TR$ is a task relevance score from the task dynamic modulation layer, by multiplying it with the conventional computation of position encoding, a way of dynamically modulating the position encoding can be realized in such a way that it reflects the specific needs of different tasks. With this enhanced multi-head self-attention mechanism and the task dynamic modulation layer, MTL-FeatureNet is able to handle the demands of multi-task learning more efficiently by improving the accuracy and relevance of features extracted from a single 2D toy view.

For the extracted features, we also introduce feature fusion and dynamic adapter layer (DAL) in the MTL-FeatureNet architecture, which aims to further optimize and synthesize these features to ensure that they can be adapted to multiple different task requirements. The DAL works after the multi-head self-attention mechanism, receiving the feature $F_{global}$ output from the self-attention layer and transforming it with a set of specially designed parameters to better serve a particular task:

(5) $$F_{task}=\sigma \left({\gamma }_{task}\cdot W_{task}\cdot F_{global}+b_{task}\right).$$

In this formulation ${\gamma }_{task}$ is a scaling factor customized for each task, which is automatically adjusted through the learning process to optimize the transformation efficiency of the features, $W_{task}$ is the task-specific weight matrix, and $b_{task}$ is the bias term. This setup allows the network to not only respond to different task requirements, but also maintain its versatility and adaptability in handling various visual tasks.

To further enhance the network’s ability to handle multi-tasks, we design a composite loss function that not only considers the performance of individual tasks, but also reinforces the synergistic effect between tasks through cross-task consistency loss:

(6) $$L=\sum _k{\lambda }_k\cdot L_k+\mu \cdot L_{cross}\left(F_{task1},F_{task2},\dots \right).$$

$L_{cross}$ denotes the loss of cross-task consistency, which facilitates knowledge sharing and complementarity between tasks by comparing similarities or differences between feature representations generated by different tasks:

(7) $$L_{cross}\left(F_{task1},F_{task2},\dots \right)=\sum _{i\ne j}{v}_{ij}\cdot \Vert F_{taski}-F_{taskj}{\Vert }^2$$where $v_{ij}$ is a dynamically adjusted weight that adjusts the effect of feature differences between each pair of tasks according to the correlation between different tasks. The schematic diagram of the feature extraction process for multi-task learning network is shown in Fig. 2.

The pseudo-code of the algorithm for MTL-FeatureNet is shown below.

Adaptive scaling

The next step in extracting key visual features from a single 2D toy view via MTL-FeatureNet is to utilize these features for efficient 3D reconstruction. In this section, we introduce the adaptive-scale NeRF technique, an improvement on the standard NeRF model designed to increase the accuracy and efficiency of 3D reconstruction, which is particularly suitable for processing features obtained from MTL-FeatureNet.

NeRF achieves impressive 3D reconstruction results by modeling the scene’s continuous bulk density and color in the line-of-sight direction, using multiple stacked fully connected networks (MLPs) to predict the bulk density and color. However, standard NeRFs often require a large amount of training time and computational resources when dealing with complex scenes, and do not capture enough details at different scales. Adaptive Scale NeRF allows the model to dynamically adjust the accuracy of rendering according to the complexity of the scene and the detail requirements of a specific region by introducing a scale-adaptive mechanism. This improvement is mainly to optimize the rendering efficiency and increase the model’s ability to capture details, especially when combined with advanced features from MTL-FeatureNet.

We use MTL-FeatureNet to extract image global features $G$. These features include not only texture, shape, and color, but also depth and illumination information through specific tasks. In order to dynamically adjust the rendering accuracy according to the extracted features during the rendering process, we define a composite adaptive scaling function:

(8) $$S\left(FA\left(G,x\left(t\right)\right),G\right)=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{m}\mathrm{o}\mathrm{i}\mathrm{d}\left(\beta \cdot D_{complex}\left(CT\left(FA\left(G,x\left(t\right)\right),G\right)\right)\right).$$

$\beta$ is a learnable scaling factor that regulates the sensitivity of the scale-adaptive function, allowing the model to find a balance between preserving detail and computational efficiency. $D_{complex}$ denotes the deep neural network used to analyze the integrated local and global features, whose output determines the degree of scale tuning. $CT(\cdot )$ is a co-feature merging function that synthesizes the local feature $Q$ and the global feature $G$ in order to generate a comprehensive scene description. $FA\left(G,x\left(t\right)\right)$ is an estimation function that determines the local features at a 3D location $x\left(t\right)$ based on that location and the global feature $G$.

Combined with an adaptive scale function, this allows the NeRF rendering process to adaptively adjust the contribution of each sample point:

(9) $$C\left(r\right)={\int }_{t_n}^{t_f}T\left(t\right)\cdot \sigma \left(p\left(x\left(t\right)\right)\right)\cdot c\left(x\left(t\right),S\left(FA\left(G,x\left(t\right)\right),G\right)\right)dt.$$

$c\left(x\left(t\right),S\left(FA\left(G,x\left(t\right)\right),G\right)\right)$ is the adjusted color output, where the adaptive scale function $S$ dynamically adjusts the color rendering at the location based on the global and estimated local features.

Spectrum-aware rendering

Spectrally aware rendering techniques (Zhang et al., 2024; Afanasyev, Voloboy & Ignatenko, 2015; Meng & Yuan, 2021) play a key role in further enhancing the realism and detailed representation of toy 3D models. This technique utilizes data acquired from multispectral images based not only on conventional red, green, blue (RGB) information, but also on a wider range of spectral information to enhance the color realism and detailed representation of materials in toy 3D models. Spectrally-aware rendering is able to simulate the physical interactions of light at different wavelengths, thus providing more accurate color reproduction under various lighting conditions, especially at the edges of the spectrum and on special materials. In this article, we model the spectrally-aware NeRF by introducing a spectral decoder that not only responds to spectra in the visible range, but also processes spectral information beyond this range. This allows the model to take into account more complex spectral data, such as infrared and ultraviolet information, in the rendering process, thus improving the color fidelity and visual effects of the scene. The spectral decoder we used is formulated as follows:

(10) $$c\left(x,\epsilon \right)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\sum _{n=1}^N{G}_n\cdot e^{-\frac{{\left(\epsilon -{\mu }_n\right)}^2}{2{\delta }_n^2}}\right)\cdot ML{P}_{color}\left(F\left(x,\epsilon \right)\right).$$

$c\left(x,\epsilon \right)$ denotes the color output at position $x$ and optical wavelength $\epsilon$. ${\mu }_n,{\delta }_n^2,G_n$ are the center wavelength, variance and weight of the $n$-th spectral component, respectively, which help the model to precisely control the spectral response at different wavelengths. The $ML{P}_{color}$ is a specially designed multilayer perceptron for decoding the color based on the spectral features $F\left(x,\epsilon \right)$, which takes into account the effects from different wavelengths.

Combined with the spectral perception technique, the rendering formula of NeRF is further improved as:

(11) $$C\left(r\right)={\int }_{t_n}^{t_f}T\left(t\right)\cdot \sigma \left(p\left(x\left(t\right),\epsilon \right)\right)\cdot c\left(x\left(t\right),\epsilon ,S\left(FA\left(G,x\left(t\right)\right),G\right)\right)dt.$$

$p(\cdot ),c(\cdot )$ are the bulk density and color, respectively, which are now dependent on the wavelength of light $\epsilon$. This allows the model to dynamically adjust the rendering under different lighting conditions, which not only takes into account the effect of the wavelength $\epsilon$ on the color and the bulk density, but also integrates an adaptive scale adjustment function $S\left(FA\left(G,x\left(t\right)\right),G\right)$ to optimize the rendering efficiency and quality. With this approach, the NeRF model is not only able to adjust the rendering accuracy to adapt to the scene complexity, but also able to reproduce the colors more realistically under various lighting conditions. With this approach of combining adaptive scaling and spectral perception techniques into NeRF models, we can realize higher quality 3D visual content. The implementation process of the NeRF method combining adaptive scaling and spectral perception is shown in Fig. 3.

The pseudo-code of the algorithm that improves on the NeRF method is shown below.

Experimental environment

For the hardware of this experiment, a server equipped with NVIDIA GeForce RTX 4090 GPU with 24GB of video memory was used, the processor was an Intel Core i9-10900K with 10 cores and 20 threads, paired with 64GB of DDR4 RAM, and the storage was equipped with a 2TB NVMe SSD. For the software environment, Ubuntu 20.04 LTS operating system was used, Python 3.8 was chosen as the programming language, and PyTorch 1.8 was used as the deep learning framework. The experimental dataset is the Co3D Toys dataset, which is divided into 80% for the training set and 20% for the testing set. The Co3D Toys subset is derived from the Canonical 3D dataset published by Meta AI. The subset contains 51 types of toy objects, 1,645 individual scenes, and about 88,000 images. The original resolution is 960 × 540 px, and in order to ensure that the algorithm can adapt to the high resolution, we randomly crop and scale the images to 512 × 512 px in the training phase, and keep 768 × 768 px in the testing phase. Co3D provides synchronized internal and external parametric calibration, and the camera trajectory is distributed as an ellipsoidal cone around a single object: elevation angle 10°–70°, and azimuth angle 0°–360°, We follow the official few view classification, where 80% of the scenes (≈1,316 scenes) in each category are used as the training set, and 20% of the scenes (≈329 scenes) are used as the test set, and 1–5 views are uniformly sampled by trajectory as inputs, and the rest of the views are used as inputs. Five views as inputs and the rest of the views are used for evaluation to ensure the comparability of the methods under sparse view and multiview conditions.

Experimental program

In order to comprehensively evaluate the performance of the AS-NeRF method in 3D reconstruction tasks, we carried out detailed experimental parameter settings for the key components of this method: the number of layers of the Transformer encoder was set to 12, the embedding dimensions were set to 768 dimensions, and 12 attentional heads were used, each with a dimensionality of 64, for each task (texture, shape, color, and depth), we introduced an independent task relevance scoring module, whose multilayer perceptron contains two layers, the hidden layer size is 256, and the activation function adopts ReLU. The initial value of the scaling factor ${\gamma }_{task}$ is set to 1.0, which is automatically learned and optimized through the training process. ${\lambda }_k$ is set for each task, and the weights of texture, shape, color, and depth are 1.0, 1.0, 0.8, and 1.0, respectively. $\mu$ is set to 0.1, in order to balance the individual task performance and inter-task synergy. $v_{ij}$ is dynamically adjusted according to task relevance, with an initial value of 0.5. $\beta$ is initialized to 1.0 and automatically adjusted by backpropagation during the training process to optimize the sensitivity of the scale-adaptive function. $N$ is set to 31, which covers major wavelengths in the range of 400 to 700 nm, with one component at 10 nm intervals. ${\mu }_n$ is set to 1.0 for the range from 400 to 700 nm at 10 nm intervals. ${\delta }_n^2$ is uniformly set to 15 to ensure the smoothness of the spectral response. $G_n$ is initialized to 1.0 to learn the contribution of each wavelength through the training process.

In order to comprehensively evaluate the performance of AS-NeRF, we select two comparison methods, SparseFusion and PixelNeRF, respectively. PixelNeRF is typical in single-view or few-view conditions, while SparseFusion combines view-conditional diffusion and distillation strategies, which is currently recognized as an important benchmark for sparse-view reconstruction performance. Both of them cover the feature reprojection route and the diffusion-NeRF route respectively, which are the closest to the technical path of the sparse view toy reconstruction task focused in this article. The experimental standard evaluation metrics cover a variety of aspects such as reconstruction accuracy, visual quality and geometric consistency. The specific evaluation indexes are as follows:

• (1)

  Peak signal-to-noise ratio (PSNR): PSNR is used to measure the similarity between the reconstructed image and the real image at the pixel level. Its calculation formula is: (15) $$PSNR=10\cdot lo{g}_{10}\left({\displaystyle \frac{MA{X}^2}{MSE}}\right).$$ $MAX$ is the maximum pixel value of the image and $MSE$ is the mean square error. Higher PSNR value indicates better quality of the reconstructed image.

• (2)

  Structural similarity index (SSIM): the SSIM is used to evaluate the similarity between the reconstructed image and the real image in terms of structure, brightness and contrast, which is calculated by the formula: (16) $$SSIM\left(x,y\right)={\displaystyle \frac{\left(2{\mu }_x{\mu }_y+C_1\right)\left(2{\sigma }_{xy}+C_2\right)}{\left({\mu }_x^2+{\mu }_y^2+C_1\right)\left({\sigma }_x^2+{\sigma }_y^2+C_2\right)}.}$$ ${\mu }_x$, ${\mu }_y$ are the mean values of image $x$ and $y$, ${\sigma }_x^2$, ${\sigma }_y^2$ are the variance, ${\sigma }_{xy}$ is the covariance, and $C_1$, $C_2$ are the stabilization coefficients. ssim value ranges from [0, 1], and the closer the value is to 1, it means the more similar the structure of the two images is.

• (3)

  Perceptual loss: Perceptual loss is based on the pre-trained deep neural network feature layer, which measures the difference between the high-level semantic features to ensure that the reconstructed image is consistent with the real image at the perceptual level: (17) $${\mathcal{L}}_{Perceptual}=\sum _i\Vert {\mathrm{\Phi }}_i\left(x\right)-{\mathrm{\Phi }}_i\left(y\right){\Vert }^2.$$ ${\mathrm{\Phi }}_i$ denotes the $i$-th layer of features of the pre-trained network, and $x$ and $y$ are the reconstructed and real images, respectively.

• (4)

  Chamfer distance: It is used to evaluate the geometric consistency between the reconstructed 3D point cloud and the real 3D point cloud, which is calculated as (18) $$Chamfer\left(S_1,S_2\right)=\sum _{p\in S_1}\underset{q\in S_2}{min}\Vert p-q{\Vert }^2+\sum _{q\in S_2}\underset{p\in S_1}{min}\Vert p-q{\Vert }^2$$where $S_1$ and $S_2$ are the reconstructed and real point clouds, respectively. The smaller the Chamfer distance, the higher the agreement between the two in terms of geometry.

• (5)

  LPIPS: LPIPS evaluates the perceptual similarity of an image by means of a pre-trained deep neural network, which measures the differences between high-level features and provides sensitivity to changes in image details and textures. Its calculation formula is: (19) $$LPIPS\left(x,y\right)=\sum _i{w}_i\Vert {\varphi }_i\left(x\right)-{\varphi }_i\left(y\right){\Vert }_2^2$$where $w_i$ is the pre-training weights and ${\varphi }_i$ is the ith layer feature of the network. Smaller LPIPS values indicate that the two images are perceptually more similar.

Experimental analysis

To test the model’s generalization ability, we randomly selected a 2D picture of a toy for 3D model generation and obtained the results shown in Fig. 4. By comparing the results with those generated by the SparseFusion method (Zhou & Tulsiani, 2023), it is evident that the AS-NeRF method yields more accurate and detailed models compared to SparseFusion, regardless of whether the analysis is based on the mesh model or the final coloring model.

To evaluate the performance of the AS-NeRF proposed in this study for 3D reconstruction tasks, we use PSNR as the primary evaluation metric and compare it with two current state-of-the-art methods: SparseFusion and PixelNeRF (Yu et al., 2021). The experiments are conducted on the Co3D dataset, and the reconstruction effects of the three methods are evaluated under different viewing angle conditions by adjusting the number of input viewing angles (one to five viewing angles). The experimental results are shown in Fig. 5.

As can be seen from the results in Figure 5, AS-NeRF exhibits high PSNR values for all the number of input viewpoints, and its PSNR values continue to improve with the increase in the number of input viewpoints, but the improvement gradually decreases. This trend indicates that AS-NeRF can significantly enhance the quality of reconstruction when the initial number of viewpoints is increased. At the same time, the performance improvement stabilizes with a further increase in the number of views. With only one input viewpoint, AS-NeRF improves the PSNR by 2.4 and 5.4 dB compared to SparseFusion and PixelNeRF, respectively, showing its better reconstruction capability under extreme conditions. When the number of input viewpoints is increased to two, the PSNR of AS-NeRF is significantly improved to 35.0 dB, which is 1.4 dB higher than that of SparseFusion and 5.3 dB higher than that of PixelNeRF. This larger improvement reflects the advantages of AS-NeRF in fusing information from multiple viewpoints. With a further increase in the number of viewpoints, although the PSNR enhancement gradually decreases, AS-NeRF still maintains a high reconstruction quality, showing its stability and effectiveness in multiview fusion.

For the SSIM metric, the reconstruction effects of the three methods under different view conditions were also evaluated by adjusting the number of input views (one to five views), and the experimental results in Fig. 6 were finally obtained.

AS-NeRF exhibits high SSIM values for all the number of input viewpoints, and its SSIM values continue to improve with the increase in the number of input viewpoints, but the improvement gradually decreases. This trend suggests that AS-NeRF can significantly enhance the structural similarity of the reconstruction when the initial number of viewpoints is increased. At the same time, the performance improvement stabilizes with a further increase in the number of viewpoints. With only one input viewpoint, AS-NeRF improves the SSIM by 0.05 and 0.10 compared to SparseFusion and PixelNeRF, respectively. When the number of input viewpoints is increased to two, the SSIM of AS-NeRF significantly improves to 0.80, representing an increase of 0.08 compared to SparseFusion and 0.10 compared to PixelNeRF, by 0.05. It can be observed that AS-NeRF outperforms SparseFusion and PixelNeRF in terms of SSIM at various input view numbers.

Perceptual loss measures the difference between the reconstructed image and the real image in terms of high-level semantic features, and a lower value indicates that the reconstructed image is closer to the real image at the perceptual level. The experiments are conducted on the Co3D dataset, and the reconstruction effects of the three methods are evaluated under different resolution conditions by adjusting the input image resolution (256 × 256, 512 × 512, 768 × 768, 1,024 × 1,024, 1,280 × 1,280). The specific experimental results are shown in Fig. 7.

At lower resolutions, AS-NeRF has the lowest perceptual loss, which is reduced by 0.05 and 0.10 compared to SparseFusion and PixelNeRF, respectively, showing its accuracy in the base reconstruction task. When the input image resolution is increased to 512 × 512, the Perceptual Loss of AS-NeRF is further reduced by 0.05 and 0.10 compared to SparseFusion and PixelNeRF, respectively, reflecting its stabilizing performance at medium resolution. At a resolution of 768 × 768, AS-NeRF continues to maintain a low Perceptual Loss, which is reduced by 0.08 and 0.12 compared to SparseFusion and PixelNeRF, respectively, indicating that it still possesses excellent reconstruction capability at higher resolutions. Taken together, as the resolution of the input image increases further, AS-NeRF still maintains a low loss value, although the reduction of perceptual loss gradually decreases.

The following experimental test was performed on the Chamfer distance, a metric that measures the geometric agreement between the reconstructed point cloud and the real one, with lower values indicating a closer shape between the two. The experiments were also performed on the Co3D dataset, and the reconstruction results of the three methods were evaluated under different viewing angle conditions by adjusting the number of input viewpoints (one to five viewpoints). Figure 8 shows the experimental results.

In the single-view condition, the Chamfer distance of AS-NeRF is 4.6, which is significantly reduced by 0.4 and 0.8 compared to 5.0 for SparseFusion and 5.4 for PixelNeRF, showing its reconstruction ability when processing single-view information. The Chamfer distance of AS-NeRF continues to decrease as the number of viewpoints increases. For example, its Chamfer distance decreases to 4.2 under the dual-view condition, which is 0.5 and 0.9 lower than that of SparseFusion and PixelNeRF, respectively, indicating that AS-NeRF has a clear advantage in the initial multiview information fusion. AS-NeRF is capable of extracting multidimensional features to provide rich contextual information for the model. This enables the model to capture and restore complex geometric structures more accurately during the reconstruction process and performs exceptionally well in multiview fusion. The use of a composite loss function and balanced training parameter settings enables AS-NeRF to effectively optimize various performance metrics during training, avoid overfitting, and improve generalization capabilities. This is especially helpful in maintaining stable reconstruction performance under multiview and high-view conditions.

LPIPS is used to measure the similarity between the reconstructed image and the real image at the perceptual level, with lower values indicating that the reconstructed image is closer to the real image in terms of perceptual quality. In this experiment, we evaluated the performance of the three methods using the LPIPS metric by conducting experiments on the Co3D dataset. We assessed the reconstruction effect of the three methods under different noise conditions by introducing various levels of image noise (0%, 10%, 20%, 30%, and 40%). The experimental results are shown in Fig. 9.

AS-NeRF exhibits lower LPIPS values at all image noise levels, indicating that it outperforms SparseFusion and PixelNeRF in terms of perceptual similarity. The LPIPS values of all three methods show an increasing trend as the image noise level increases, reflecting the negative impact of noise on the reconstruction quality. However, the LPIPS growth of AS-NeRF is more moderate, especially at higher noise levels, and its performance advantage remains evident. The improved NeRF model employs an adaptive scale adjustment mechanism to dynamically optimize rendering accuracy based on the complexity of image details. The method in this article combines spectral perception techniques to enhance the realism and detail performance of color reproduction, thereby improving the stability of the model under various noise conditions. As the noise level increases, the LPIPS values of SparseFusion and PixelNeRF decrease but fail to reach the level of AS-NeRF, further emphasizing that AS-NeRF has advantages in dealing with noise interference.