FPGA-based systolic deconvolution architecture for upsampling

Overview of 4 × 4 to 8 × 8 deconvolution architecture

To upsample a 4 × 4 input to a 8 × 8 output using FSC, a temporary extended image E of size 10 ×10 is created by inserting zeros between the input elements (shown as white grids in Fig. 1), padding around the boundaries (shown as red grids) and along the right and bottom edges (shown as cyan grids). As the 3 × 3 kernel slides across E, the output is computed from four computational patterns expressed in colours: pink, blue, yellow and green. For example, when the kernel is placed at the top left corner of E, the output O₁ shown as the pink grids, the output image O_8×8 is computed by multiplying the input d₁ with central element k₅ of the kernel, i.e., (4)${\displaystyle {\mathsf{O}}_1={\mathsf{K}}_5\times d_1.}$

Likewise, progressing with a stride of 1 along the row followed by the column, the interpolated elements corresponding to the 8 × 8 output is obtained from the 4 × 4 input. For example, when the kernel is strided along the row and column, the blue and yellow grids of O_8×8 give the interpolated output O₂ and O₃, i.e., (5)${\displaystyle {\mathsf{O}}_2={\mathsf{K}}_4\times {\mathsf{d}}_1+{\mathsf{K}}_6\times {\mathsf{d}}_2}$ (6)${\displaystyle {\mathsf{O}}_3={\mathsf{K}}_2\times {\mathsf{d}}_1+{\mathsf{K}}_8\times {\mathsf{d}}_5.}$

Similarly the green grid denoted by O₄ computes the output (7)${\displaystyle {\mathsf{O}}_4={\mathsf{K}}_1\times {\mathsf{d}}_1+{\mathsf{K}}_3\times {\mathsf{d}}_2+{\mathsf{K}}_7\times {\mathsf{d}}_5+{\mathsf{K}}_9\times {\mathsf{d}}_6.}$

Figures 2A–2D illustrate the four computation patterns, where k₁, k₂, k₃, …, k₉ respectively correspond to the 3 × 3 kernel coefficients 1, 2, 3, …, 9, and d₁, d₂, d₃, …, d₁₆ respectively denote the 4 × 4 input 1, 2, 3, …, 16. Thus, by extending the 4 × 4 input and employing Eqs. (4) to (7) we can compute the required 8 × 8 upsampled outputs.¹

The deconvolution architecture to upsample a 4 × 4 input to a 8 × 8 output by convolving with a 3 × 3 kernel is shown in Fig. 1 and according to Table 3, the architecture requires: (i) SR module of length 5 to allow buffering and enable computations to be performed in parallel; (ii) 4 PEs to compute Eqs. (4) to (7); (iii) 4 FIFOs each of length 16 are used to store the upsampled outputs; and (iv) a DCM comprising of multiplexers and 4 counters (count1, count2, count3, count4) for indexing the row and columns of the input and output, respectively.

The length of the SR module is based on the kernel size and the input resolution. In general the length of the SR module (Num_SR) is given by ${\mathsf{Num}}_{\mathsf{SR}}=\frac{k-1}{2}\times n+\frac{k-1}{2}$. For I_4×4 and K_3×3, the length of SR module is 5. Furthermore, the length each of the FIFO is fixed as n × n. Since the input is 4 × 4, the FIFOs have a length of 16.

The PEs are hardware wired for a particular upsampling interval and kernel size, and execute in parallel to compute one of Eqs. (4) to (7). For example, PE₁ receives input from SR₁ and PE₂ receives inputs from both SR₁ and D₀. The input and output connections of each PEs and their associated kernel coefficients are shown Fig. 3, where SR₁, SR₂, SR₄ and SR₅ are respectively the outputs of the flip flops FF₁, FF₂, FF₄ and FF₅ of the SR module.

To explain the operation of module we use the same inputs and kernel coefficients as shown in Fig. 1, and the timing diagram of the generation of the outputs for the first 24 clock cycles is shown in Fig. 4. Once signal De is enabled, the deconvolution accelerator is active and the input data (signal D₀ in the timing diagram) enters the SR module and propagates forward through FF₁ to FF₅ at the positive edge of the clock.

At time T = t2, both PE₁ and PE₂ simultaneously receive their input from D₀ and SR₁, respectively, which are then multiplied with their corresponding kernel coefficients of the K_3×3 to present the outputs, O₁ and O₂, respectively, i.e., (8)${\displaystyle {\mathsf{PE}}_1:{\mathsf{O}}_1={\mathsf{SR}}_1\times {\mathsf{k}}_5}$ (9)${\displaystyle {\mathsf{PE}}_2:{\mathsf{O}}_2={\mathsf{D}}_0\times {\mathsf{k}}_6+{\mathsf{SR}}_1\times {\mathsf{k}}_4.}$

Subsequently as the input data advances, between clocks T = t3 and T = t6 and employing just PE₁ and PE₂, the upsampled elements of the first row (Row₁) of O_8×8 are computed. Due to zero padding at the rightmost boundary of the extended image, the last computation within PE₂ requires just the multiplication of SR₁ × k₄. This is achieved by employing a counter (count2) to track the column indices and notify the multiplexer as shown in Fig. 3B. The architecture of PE₁ and PE₂ are shown in Figs. 3A and 3B, respectively.

To compute the upsampled elements of Row₂ and Row₃, along with PE₁ and PE₂, PE₃ and PE₄ operate in parallel. At clock T = t6, all the PEs simultaneously receive their input (D₀, SR₁, SR₄ and SR₅) from the SR module which then gets multiplied with the corresponding kernel coefficients and to simultaneously produce the respective outputs. Figures 3C and 3D illustrate the architecture of PE₃ and PE₄ where (10)${\displaystyle {\mathsf{PE}}_3:{\mathsf{O}}_3={\mathsf{SR}}_1\times {\mathsf{k}}_8+{\mathsf{SR}}_5\times {\mathsf{k}}_2}$ (11)${\displaystyle {\mathsf{PE}}_4:{\mathsf{O}}_4={\mathsf{D}}_0\times {\mathsf{k}}_9+{\mathsf{SR}}_1\times {\mathsf{k}}_7+{\mathsf{SR}}_4\times {\mathsf{k}}_3+{\mathsf{SR}}_5\times {\mathsf{k}}_1,}$

Here, O₃ and O₄ represent the outputs of PE₃ and PE₄, respectively. The availability of the input data at every clock cycle and the parallel execution of PEs enable the deconvolution accelerator to compute all 16 interpolated outputs of Row₂ and Row₃ of O_8×8 within 4 clock cycles, i.e., between T = t7 and T = t10. As the input data proceeds into the deconvolution module the elements of Row₄ to Row₇ are computed in the similar fashion. Finally, to compute Row₈ of O_8×8, (row index is traced using count1) only PE₃ and PE₄ execute in parallel and using Eqs. (10) and (11) produces upsampled outputs O₃ and O₄. Again, to compensate for the zero padding at the bottom and right edges, multiplexers and additional controls are provided within PE₃ and PE₄ as shown in Figs. 3C and 3D.

Thus, at each clock instance, the PEs produce simultaneous outputs: O₁, O₂ by PE₁ and PE₂ for Row₁; O₁, O₂ O₃, O₄ by PE₁, PE₂, PE₃ and PE₄ for Row₂ to Row₇; and O₃, O₄ by PE₃ and PE₄ for Row₈ are temporarily stored in 4 separate FIFOs, FIFO₁, FIFO₂, FIFO₃ and FIFO₄ as shown in Fig. 5. The FIFOs write and read commands are synchronised with the input clock of the accelerator module and a series of controls generated by the DCM enables effective writing and streaming of the upsampled outputs from the FIFOs.

DCM of 4 × 4 to 8 × 8 deconvolution architecture

The DCM is shown in Fig. 6 and consists of two control switches CSW1 and CSW2 that assist in the generation of FIFO write and read commands, enabling temporary storage and retrieval of the data. CSW1 and CSW2 are controlled by counters count1 and count3 which track the row indices of the input and the outputs, respectively. The FIFO write cycle is as follows:

1. To store Row₁ of O_8×8: Initially count1 = 0, CSW1 = 0, PE₁ and PE₂ execute in parallel and their corresponding outputs stored in FIFO₁ and FIFO₂, respectively. Also, FIFO₃ and FIFO₄ are write disabled.

2. To store Row₂ to Row₇ of O_8×8: (Beginning T = t7) count1 increments from 1 to 3, CSW1 = 1, PE₁, PE₂, PE₃ and PE₄ execute in parallel, and all the FIFOs are write enabled. PE₃ and PE₄ are connected to FIFO₁ and FIFO₂ where as PE₁ and PE₂ are linked to FIFO₃ and FIFO₄. The FIFO inputs are interchanged to enable easier read of the outputs during the read cycle.

3. Finally for Row₈ of O_8×8: count1 = 4, CSW1 = 1, only PE₃ and PE₄ execute in parallel and their outputs are connected to FIFO₁ and FIFO₂.

The read operation is managed by CSW2 and the Read signal is asserted after a delay of β clocks cycles and after De = 1 where β = θ + FIFO_delay.θ (refer to ‘Computation time of single Deconvolution Accelerator’) represents the delay before a valid sample is available at the output of PEs and normally FIFO_delay = 2 clock cycles. Thus, to upsample 4 × 4 to 8 × 8 using a 3 × 3 kernel we set β to 3 (θ = 2, for details refer to ‘Computation time of single Deconvolution Accelerator’). Once the Read is asserted, count3 and count4 respectively track the number of rows and columns of O_8×8 and the data is read from the FIFOs using separate signals (Fr1, Fr2, Fr3 and Fr4) that are controlled by line control (LC) and transfer control signals (TF), respectively, as shown in Fig. 6. With LC = 1 or 0, and based on the rising edge of the TF, the data is read from the corresponding FIFO in an orderly manner, i.e., (12)${\displaystyle \mathsf{Fr}1=!\mathsf{TF}\&\&\mathsf{LC}.}$ (13)${\displaystyle \mathsf{Fr}2=\mathsf{TF}\&\&\mathsf{LC}.}$ (14)${\displaystyle \mathsf{Fr}3=!\mathsf{TF}\&\&\mathsf{LC}.}$ (15)${\displaystyle \mathsf{Fr}4=\mathsf{TF}\&\&\mathsf{LC}.}$

where ! and && denote the logical NOT and logical AND operations, respectively. The FIFO read cycle is as follows:

1. Initially read Row₁ of O_8×8: count3 = 0, LC = 1 and TF is toggled for every clock cycle. The generated read signals, Fr1 and Fr2, using Eqs. (12) and (13) control the read operations of FIFO₁ and FIFO₂, respectively.

2. To read Row₂ to Row₈ of O_8×8: Starting at T = t13, count3 increments from 1 to 7, LC increments for each update of count3 and TF is toggled for every clock cycle as shown in Fig. 4. If LC is 0, using Eqs. (14) and (15) the computed results are read from FIFO₃ and FIFO₄. When LC is 1, FIFO₁ and FIFO₂ are enabled for reading. Note that count3 is controlled by the column counter count4 which increments for every 0 to 2n − 1.

The read cycle of the DCM introduces a delay (DCM_delay) of 3 clock cycles before the outputs are streamed in a systolic manner regulated by a reference clock. The proposed deconvolution architecture can be extended for various upsampling intervals by just extending the number of FFs within the SR module. The number of the PEs remain the same but their inputs differ. The PE equations for different upsampling internals for different kernel size are given in Table A1.

Kernel bit width

At the positive edge of a clock signal, the deconvolution accelerator receives a stream of pixels 8-bit width which propagates through the shift register and PEs. The inputs are multiplied with the corresponding kernel coefficients with the results stored in FIFOs. For hardware implementations, fixed point is the natural choice of data representation due to simplicity and less usage of hardware resources. Thus, the floating point kernel coefficients are converted to fixed point by using a scaling factor of 2^f and expressing the output as (f + 1)-bit within the FPGA. Here the optimum f is chosen by comparing the metrics such as Root Mean Square Error (RMSE) and the Peak Signal to Noise Ratio (PSNR) for different combinations of 2^f with the corresponding IEEE double-precision output. Table 4 illustrates the results, where the kernel coefficients were selected from the distribution of range between −1 to +1 by invoking Keras tool (He et al., 2015). Initially, when f = 7, 8 and 9, the RMSE is high but with increase in the precision (bit width of the kernel), the PSNR improves and RMSE lowers, suggesting that fixed-point calculations are comparable to those of floating point operations. A scaling factor of 2¹¹ gives acceptable PSNR of 78.52 dB (Rao et al., 1990) with a low RMSE of 0.0303 and indicates that the fixed-point result is close to the IEEE double-precision . Increasing the bit width above 12 resulted in no significant improvement in PSNR and therefore the bit width of the kernels was set to 12-bit (f = 11 and 1 sign bit). Therefore a kernel value of (0.13250548)₁₀ was first multiplied by 2048 (2¹¹) and its result (271.37122304)₁₀ was rounded to (271)₁₀. Its equivalent fixed-point representation in 11-bit along with 1 sign bit (000100001111)₂ was used to represent the filter coefficient.

PEs output bit width

To illustrate that a deconvolution architecture produces upsampled outputs with considerable accuracy, we compare the upsampled outputs at different upsamping intervals (from 32 × 32 to 256 × 256) with those of the corresponding MATLAB outputs. For a realistic comparison, an image with a flat Power Spectral Density (PSD) (e.g., a white noise) was chosen as an input and the metrics, PSNR and RMSE, were used to evaluate the model. Based on the experimental results of the previous section, the input and kernel bit widths were set to 10-bit and 12-bit, respectively. The output the PEs were varied between 8 to 12-bit and the upsampled results of the deconvolution accelerator was compared with the corresponding MATLAB outputs. Table 5 shows the results and it can be inferred that 10-bit output is sufficient since the PSNR averages more than 58 dB across all upsampling intervals. Further increasing the bit widths resulted in no significant increase in the PSNR but resulted in considerable increase in hardware. Therefore, the choice of 10-bit upsampled outputs is reasonable. With the kernel and input width set to 12-bit and 8-bit, the accelerator produces upsampled outputs of 22 maximum bits (computation within the PEs include both multiplication and addition), and therefore the upsampled elements are left shifted 11 times and the 9 most significant bits (MSB) bits in addition to the sign bit are stored in the respective FIFOs. The shift operation compensates the earlier 2¹¹ multiplication of the kernel coefficients.

Comparison of upsampled results of different kernel sizes obtained from a trained U-Net models

We compare the outputs of the deconvolution accelerator with the MATLAB versions for various input sizes on kernel coefficients obtained from a trained U-Net model and natural images obtained from various datasets. First, we upsampled an random image of size 32 × 32 image to resolutions: 64 × 64, 128 × 128 and 256 × 256 using a 3 × 3 kernel with a maximum and minimum values of 0.7219356 and −0.64444816. The kernel coefficients obtained from the corresponding decoder frame work of the U-Net are stored in a register as 12-bit fixed point representation (as explained in ‘Kernel bit width’) and the upsampled results of the previous stage are provided as inputs to the current stage. Figure 7A illustrates the upsampled images at each stage of the pipeline (32 to 256). Tables 6 and 7 respectively show the corresponding performance scores and the resource usage. Furthermore, Table 8 reports resource usage for individual deconvolution units employing 3 × 3 kernels. Next, the camera man and the Natural images are examined with similar interpolation intervals. To illustrate that the proposed model can be extended for different kernel sizes, we also present upsampled results (Figs. 7B and 7C) obtained from 5 × 5 and 7 × 7 kernel sizes with maximum and minimum coefficient values of 0.78133786, −0.7012087, 0.5295713 and −0.46372077, respectively. The 10-bit deconvolution accelerator output is compared with the corresponding IEEE double-precision outputs using the metrics RMSE and PSNR. The outputs across different upsampling intervals show low RMSE and high PSNR of almost 80 dB which are comparatively better than the 40 dB of maximum PSNR reported by Chang, Kang & Kang (2020). Thus the 10-bit deconvolution accelerator indeed produces upsampled outputs comparable to MATLAB results.

Computation time of single Deconvolution Accelerator

The total computation time (T_total) required in terms of clock cycles for upsampling is given by (16)${\displaystyle T_{total}=T_{CT}+\theta ,}$ where T_CT is the time required to obtain 2n × 2n samples from a n × n input, θ denotes the delay before a valid sample is available at the output of the PEs. T_CT is obtained as follows:

1. To compute Row₁ of the 2n × 2n, PE₁ and PE₂ execute in parallel n times.

2. To compute Row_2n of the 2n × 2n, PE₃ and PE₄ execute in parallel n times.

3. To computes rows Row₂ to Row_2n−1 of the 2n × 2n, PE₁, PE₂, PE₃ and PE₄ operate in parallel as batches represented by N with each batch executing n times.

Therefore (17)${\displaystyle T_{CT}=2\times n+N\times n,}$

where n denotes the input size and N is given by (18)${\displaystyle N=\left[\frac{\left(Ro{w}_{2n}-Ro{w}_1\right)-1}{2}\right].}$

The denominator indicates that 2 rows of the 2n ×  2n output are computed when the all the PEs execute in parallel. The initial delay θ depends on k and is given by (19)${\displaystyle \theta =\lceil \frac{k+1}{4}\rceil +1.}$

⌈ ⌉ denotes the ceiling operation. Figure 8 illustrates T_total and Table 9 tabulates θ, T_CT and T_total for different upsampling intervals and kernels. Thus, using the 3 × 3 kernel to upsample 4 × 4 to 8 × 8, (substitute k = 3 in Eq. (19)), the first effective result at the output of the PEs (PE₁ and PE₂) is obtained after a delay of two clock cyles, (i.e., θ =2). Subsequently $〈{\mathsf{PE}}_1,{\mathsf{PE}}_2〉$ execute 4 times in parallel to compute the samples of Row₁. For Row₂ to Row₇, all the PEs independently execute 4 times in parallel but in 3 pipelined batches (N = 3 as computed using Eq. (18)). Finally, for Row₈, $〈{\mathsf{PE}}_3,{\mathsf{PE}}_4〉$ again execute 4 times in parallel. Substituting the execution cycles of PEs required to compute each row of the output along with N in Eq. (17), the computation time T_CT can be found. Thus, to upsample 4 × 4 to 8 × 8; T_CT = 20 (i.e., 2 × 4 + 3 × 4 = 20) clock cycles, and T_total = 22 clock cycles. The upsampled outputs are temporarily stored in the FIFOs and after an initial delay of β +  DCM_delay clock cycles are read simultaneously by initiating the FIFO read signals as in Eqs. (12) to (15). The time-to-read (T_R) the upsampled elements is 2n × 2n for an n × n input since the upsampled elements are streamed in a systolic manner (1 output per clock cycle) in reference to the common clock.

Computation time for the Pipelined architecture

The DCM allows separate read and write controls of the FIFOs and thus the upsampled elements of deconvolution accelerator can be readily streamed to the next stages: 2n × 2n to 4n × 4n, 4n × 4n to 8n × 8n and so on to represent a pipelined architecture that is similar to the decoder module of the U-Net. The computation time for the pipelined (T_P) deconvolution framework is given by (20)${\displaystyle {\mathsf{T}}_{\mathsf{P}}=\mathsf{D}\times \left(\beta +{\mathsf{DCM}}_{\mathsf{delay}}\right)+{\mathsf{T}}_{\mathsf{R}},}$

where D denotes the number of upsampling intervals, T_R (time-to-read) is T_R = (2^D×n)²) and DCM_delay = 3, and β is the delay before the read signal (Read) is asserted (refer to ‘DCM of 4 × 4 to 8 × 8 deconvolution architecture’). To upsample 32 × 32 to 256 × 256 using K_5×5, T_P is computed by substituting D = 3, β + DCM_delay = 8 (β = θ + FIFO_delay; refer to Table 9 for θ and ‘DCM of 4 × 4 to 8 × 8 deconvolution architecture’ for FIFO_delay and DCM_delay, and T_R = 65536 cycles ((2³ × 32)²) in Eq. (20)). Thus, T_p = 65560 clock cycles (3 × 8 + (2³ × 32)²). Furthermore, for example, if a clock frequency of 50 MHz is considered, then the T_P of the three-stage pipelined deconvolution module capable of upsampling 32 × 32 to 256 × 256 is 1310.84 µs (65542 × 0.02 μs), thus achieving a frame rate of 763 fps (frames per second). Figure 8 illustrates T_P for a two stage pipelined deconvolution framework (n × n to 4n × 4n).

Comparison of computation complexity of the proposed architecture with other deconvolution architectures

The total number of operation (multiplications and additions) required to complete the upsampling process represents the computation complexity of the model. For the proposed architecture the number of multipliers OP_mul and adders OP_add required to upsample n × n to 2n × 2n using k × k kernel are given by (21)${\displaystyle {\mathsf{OP}}_{\mathsf{mul}}={\left[n\times k-\frac{1}{8}{\left(k-1\right)}^2-\frac{1}{4}\left(k-1\right)\right]}^2.}$ (22)${\displaystyle {\mathsf{OP}}_{\mathsf{add}}={\left[n\times k-\frac{1}{8}{\left(k-1\right)}^2-\frac{1}{4}\left(k-1\right)\right]}^2-{\left(2n\right)}^2.}$

The total operations OP_total is given by (23)${\displaystyle {\mathsf{OP}}_{\mathsf{total}}=2{\left[n\times k-\frac{1}{8}{\left(k-1\right)}^2-\frac{1}{4}\left(k-1\right)\right]}^2-4n^2.}$

Table 10 shows the OP_mul, OP_add and OP_total for various upsampling intervals and kernel sizes. When compared with existing architectures(refer to Table 10) where the total operations are computed using k²n² + 2k(k − s)(n − s) + (k² − s²)(n − 2)² (for Liu et al. (2018)) and (2 × k² − 1) × n² for (Zhang et al. (2017) and Yan et al. (2018)), the proposed deconvolution architecture reduces the required operations by a maximum of 20%. We attribute this reduction to the pipelined structure of the architecture which executes either 2 or 4 PEs in parallel per clock cycle to produce the interpolated outputs. Also, at any clock instance, the maximum number of multiplier employed by the accelerator using a kernel of size k × k is k², which relates to the parallel execution of all PEs in a batch for rows 2 to 2n − 1. Furthermore from Table 10, we observe a significant reduction in operations when 3 × 3 kernel size is used for up-sampling which directly contributes to resource utilization.

We also compare our proposed architecture with other deconvolution architectures in terms of (i) total operations, (ii) clock cycles required to complete an upsampling interval, (iii) hardware usage, (iv) GOPS and (v) resource efficiency (GOPS/DSP). To have favourable comparison across all architectures, we compare a single deconvolution module based on fixed point representation capable of upsampling a 128 × 128 input to 256 × 256 using a 5 × 5 kernel and Table 11 shows the results. Here GOPS, which denotes the processing performance of the model is computed using Di et al. (2020): (24)${\displaystyle \mathsf{GOPS}=\frac{{\mathsf{OP}}_{\mathsf{total}}}{{\mathsf{T}}_{\mathsf{total}}\times \frac{1}{\mathsf{Freq}}},}$

where Freq denotes the frequency. From Table 11, it is evident that the proposed architecture uses fewer operations and therefore less hardware resources to upsample. Furthermore, the proposed architecture produces the best resource efficiency of 0.309 GOPS/DSP at 200 MHz. The lowest clock cycles are required to upsample a 128 × 128 input to 256 × 256 across all considered architectures. We attribute the improvement to the hardware design which benefits in the reduction of operations and produces a maximum operations saving of 23% (by comparing the OP_total of Di et al. (2020)) which directly relates to lower usage of the hardware resources. Furthermore, the proposed deconvolution accelerator achieves GOPS = 3.641 and GOPS/DSP = 0.135 for the pipelined architecture 32 × 32 to 256 × 256.

Extension of the proposed Deconvolution Accelerator

Although traditional U-Nets are based on 3 × 3 (Shvets & Iglovikov, 2018) kernels, few architectures either employ 5 × 5 (Chang, Kang & Kang, 2020) or 7 × 7 (Badrinarayanan, Kendall & Cipolla, 2017) in their encoder–decoder pipeline. Thus, to allow reusability of the architecture, we present in Table A1, equations for different upsampling intervals for 3 × 3, 5 × 5 and 7 × 7 kernels. The number of PEs are the same, but the length of the SR module and the FIFOs differ(refer to Table 3). Thus, by rewiring the inputs to the PEs, different upsampling intervals using different kernels sizes are obtained.