1. Introduction

2. Realted Work

3. Proposed Method

3.1. Deep Learning for Generating ToF Data

3.1.1. Network Architecture

The network architecture used in this work is a modified version of MCW-Net [11] which is inspired by the encoder-decoder architecture of U-Net [20]. The MCW-Net was originally designed for rain removal, which targets the identification and manipulation of specific image features. The encoder captures the essential features of the image, while the decoder reconstructs the image, applying learned modifications. This structure is equally useful for ToF artifact simulation.

In the encoder phase, the network gradually downsamples the input image to capture lower-resolution representations of the features. These features are then utilized in the decoder phase to generate a new image through an upsampling process. Figure fig:mcw-net provides an overview of the MCW-Net architecture.

Figure 1. MCW-Net Architecture. The levels represent the size of the feature map. The stages consist of multiple blocks on the same level.

Figure 1. MCW-Net Architecture. The levels represent the size of the feature map. The stages consist of multiple blocks on the same level.

MCW-Net consists of four levels, each representing a specific feature map size. Within each level, a set of layers is defined as a stage. Inspired by U-Net, skip connections are integrated between stages of the same level to mitigate information loss in the second half of the network. MCW-Net additionally connects stages at different levels as seen in fig:mcw-net. The multi-level connections allow the decoder to access features of varying sizes and levels, which is particularly beneficial when dealing with region-based errors like MPI.

Within each stage of MCW-Net, feature extraction is achieved by combining two densely connected residual (DCR) blocks and a regional non-local block (RNLB). The DCR block consists of three sequential 3 × 3 convolutional layers, followed by a parametric rectified linear unit (PReLU) activation function [21].

The following RNLB further enhances the spatial awareness of the network. It operates by dividing the feature map into patches and performing non-local convolutions. In each non-local operation, the response at a specific position is computed as a weighted sum of features from all spatial positions within the patch. While the original MCW-Net proposed wide rectangular grids for the RNLB [11], a squared grid is implemented in this work. By employing a squared grid, the number of horizontal and vertical features becomes equal, achieving a balanced distribution of information across both dimensions.

MCW-Net uses direct wavelet transform (DWT) for downsampling and inverse DWT (IDWT) for upsampling processes. This avoids the information loss caused by conventional subsampling, like max-pooling where only strong features are considered. The DWR is simply implemented by the Haar transform which consists of four 2 × 2 filters. Therefore the feature map size is halved and the number of channels quadrupled in the downsampling steps and vice versa in the inverse operation.

The stages in the decoder are designed with a necessary squeeze-and-excitation (SE) block [22] to rescale the feature maps obtained from the multi-level connections, see fig:mcw-net. The number of channels is adjusted by a 1 × 1 convolutional layer.

The scalability of the model is achieved by adjusting the number of channels, as demonstrated in the work of Park et al. [11]. They presented two variants of the model: large and small. The large model contains eight times more channels compared to the small model, resulting in 129.5 million parameters for the large model and 2.2 million parameters for the small model. However, due to the limited amount of training data, experiments revealed that the large model tends to overfit quickly. As a result, subsequent discussions only consider the small version of the model. To properly adapt the network to the depth maps the input and output channels have been set to one, which is another modification from the original MCW-Net architecture.

3.1.2. Training

The RWU3D dataset is split into 40 train scenes and 17 validation scenes. Each scene consists of a single laser depth map which serves as the input image, and a set of n ToF depth maps with a size $n=20$, referred to as labels $\mathbf{Y}=\{{\mathbf{y}}_{\left(1\right)},...,{\mathbf{y}}_{\left(n\right)}\}$. During training, a random patch of 256 × 256 is cropped from the input image, and a batch size of 4 is utilized. Not a Number (NaN) values in the input data are appropriately handled by setting them to zero, while NaNs in the labels do not contribute to the loss function. The Adam optimizer is employed, with an initial learning rate of 0.0005 and a learning rate decay of 1/2 every 100 epochs. The model is trained for a total of 400 epochs.

To harness the potential of the dataset, two distinct loss functions were investigated. The first loss function is characterized by the L1 Loss, computed between the mean of all available labels and the network’s prediction $\widehat{\mathbf{y}}$, as depicted in Equation 1. This approach aims to constrain the network’s prediction to a singular solution instead of considering the 20 different possibilities.

$${\mathcal{L}}_{\mathrm{mean}}={\parallel \left({\displaystyle \frac{1}{n}}\sum _{k=1}^n{\mathbf{y}}_{\left(k\right)}\right)-\widehat{\mathbf{y}}\parallel }_1,$$

(1)

On the other hand, the second loss function is determined by calculating the minimum L1 distance among all individual labels in the dataset, as shown in equation 2. This emphasizes precise predictions, favoring scenarios where multiple outcomes are appreciated.

$${\mathcal{L}}_{\min }=\underset{k=1,\dots ,n}{min}{∥{\mathbf{y}}_{\left(k\right)}-\widehat{\mathbf{y}}∥}_1.$$

(2)

Data augmentation is applied using horizontal and vertical flipping to increase the diversity of the training dataset. This augmentation is especially important due to the presence of floor areas at the bottom of each scene. Furthermore, the impact of noise on data augmentation is explored. Three distinct input types are employed:

• No noise
• Additive Gaussian noise, denoted as [Laser+Noise]
• Gaussian noise introduced on a separate channel, referred to as [Laser, Noise]

In addition to the enhanced robustness achieved through data augmentation, the implementation also has the objective of ensuring that the network’s output predicts realistic noise characteristics. However, results show that all of the above-mentioned models suffer to produce realistic noise.

3.2. ToF Noise Model

The behavior of the utilized model made clear that a noise model is needed. Developing an accurate depth noise model for ToF sensors is still a topic of current calibration research. Thus, similar to [16] a simple noise model is presented. This noise component will be combined with the model’s predictions to simulate realistic noise characteristics.

Various noise sources contribute to the overall noise in ToF cameras. The statistical distribution of ToF cameras can be approximated reasonably well by a Gaussian distribution, as shown in previous studies [16,23]. To quantify the spread of this distribution, three key parameters that affect the noise were examined: distance, angle of incidence, and the presence of edges. These parameters were selected based on the available input data, which is the depth map obtained from the laser scanner. The output functions of the three distinct noise sources are combined to generate a pixel-wise standard deviation.

The standard deviation as a function of distance is approximated by the linear model in equation 3. The coefficients $k_0$ and $k_1$ for the linear function are determined from values specified in the datasheet provided by the manufacturer [24]. The standard deviation ${\sigma }_d$ does not require any pre-processing and only uses the distance d in the laser depth map.

$${\sigma }_d=k_1·d+k_0$$

(3)

To compute the standard deviation on edges, the training data’s edge-containing areas undergo analysis. Utilizing the Canny edge detection algorithm on the depth map identifies these edges. The ensuing step involves calculating the mean standard deviation at the edges, resulting in a mean value of 0.03 m across all training images. This value is then applied to stddist if the pixel location is at an edge in the validation image.

The third parameter which contributes to the total standard deviation is the angle of incidence $\theta$. To determine it from the depth map, first, the normal map is computed by using trigonometric functions. For each pixel, the angle between the normal vector and the incidence vector is calculated. Therefore two assumptions have been made. Since the ToF data is transformed to the RGB sensor this minor change in perspective as well as the co-positioning of the illumination element are not taken into account. Secondly, the light direction is set to a global constant vector which is equal to the normal of the sensor plane.

In order to determine the standard deviation based on the angle of incidence the characteristics of the sensor were investigated. For this purpose, a target plane positioned at a distance of 1 meter, is captured at various angles. To achieve this, a target plane is mounted on a 3D printed platform that allows the plane to be rotated in 5° increments from 0° to 85°. The utilized material of the target plane is the less reflective wood in one series, while in a separate series, the target plane made of rigid foam is captured. Therefore an effective comparison between the two materials with different reflectivity is made.

In line with the experimental setup described by Chiabrando et al. [25], a sequence of 50 frames for each angle set is recorded. Notably, tests using a sequence of 500 frames showed no discernible difference in standard deviation as the number of frames increased.

As the angle increases, the area of the target visible to the sensor decreases while the edges of the target become more present, which can lead to inaccurate measurements. Therefore a thin region of interest (ROI) with 5 × 20 pixels is investigated to calculate the mean standard deviation at each angle. However, since it couldn’t be avoided that even this area is affected by higher values at the edges, the measurement at 85° is discarded.

Inspired by the work of Keller et al. [16] fitting of a polynomial is used to approximate the measured data points. For each sequence, a polynomial function of degree 5 is fitted to the measured values. The results are plotted in fig:angstddata. Considering that the variation among the utilized materials is relatively small in comparison to the overall range of the functions within the interval [0, 90], the final function $P\left(\theta \right)$ includes the data points from both materials.

$${\sigma }_{\theta }=P\left(\theta \right)-b$$

(4)

To compensate the offset of $P\left(\theta \right)$ which ${\sigma }_d$ already incorporates a bias term b is subtracted. Combining the results from all three parameters gives the total standard deviation ${\sigma }_{total}$ by

$${\sigma }_{total}=\sqrt{{\sigma }_d^2+{\sigma }_{\theta }^2+{\sigma }_e^2}.$$

(5)

Furthermore, this leads to the resulting noise distribution:

$$X_{noise}\sim \mathcal{N}(0,{\mathbf{\sigma }}_{total}^2),$$

(6)

where ${\mathbf{\sigma }}_{total}$ is the total standard deviation matrix and $X_{noise}$ is the noise matrix around zero.

Figure 2. Standard Deviation as a function of the Angle of Incidence for wood (blue) and rigid foam (orange). Fitted curves are polynomials of degree 5.

Figure 2. Standard Deviation as a function of the Angle of Incidence for wood (blue) and rigid foam (orange). Fitted curves are polynomials of degree 5.

The resulting noise is finally added to the model’s predicted depth map, enabling a more realistic simulation of ToF camera behavior. It’s important to note that the noise model is not used during the training phase.

4. Evaluation

5. Conclusion

Abbreviations

References

1. Freedman, D.; Smolin, Y.; Krupka, E.; Leichter, I.; Schmidt, M. SRA: Fast removal of general multipath for ToF sensors. Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13. Springer International Publishing, 2014, pp. 234–249.
2. Godbaz, J.P.; Dorrington, A.A.; Cree, M.J. Understanding and ameliorating mixed pixels and multipath interference in amcw lidar. TOF Range-Imaging Cameras 2013, pp. 91–116.
3. Mutny, M.; Nair, R.; Gottfried, J.M. Learning the correction for multi-path deviations in time-of-flight cameras. arXiv preprint arXiv:1512.04077 2015.
4. Guo, Q.; Frosio, I.; Gallo, O.; Zickler, T.; Kautz, J. Tackling 3d tof artifacts through learning and the flat dataset. Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 368–383.
5. Buratto, E.; Simonetto, A.; Agresti, G.; Schäfer, H.; Zanuttigh, P. Deep learning for transient image reconstruction from ToF data. Sensors 2021, 21, 1962. [Google Scholar] [CrossRef] [PubMed]
6. Su, S.; Heide, F.; Wetzstein, G.; Heidrich, W. Deep end-to-end time-of-flight imaging. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 6383–6392.
7. Agresti, G.; Zanuttigh, P. Deep learning for multi-path error removal in ToF sensors. Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018, pp. 0–0.
8. Geiger, A.; Lenz, P.; Stiller, C.; Urtasun, R. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research 2013, 32, 1231–1237. [Google Scholar] [CrossRef]
9. Caesar, H.; Bankiti, V.; Lang, A.H.; Vora, S.; Liong, V.E.; Xu, Q.; Krishnan, A.; Pan, Y.; Baldan, G.; Beijbom, O. nuscenes: A multimodal dataset for autonomous driving. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 11621–11631.
10. Hackel, T.; Savinov, N.; Ladicky, L.; Wegner, J.D.; Schindler, K.; Pollefeys, M. Semantic3d. net: A new large-scale point cloud classification benchmark. arXiv preprint arXiv:1704.03847 2017.
11. Park, Y.; Jeon, M.; Lee, J.; Kang, M. MCW-Net: Single image deraining with multi-level connections and wide regional non-local blocks. Signal Processing: Image Communication 2022, 105, 116701. [Google Scholar] [CrossRef]
12. Agrawal, A.; Müller, T.; Schmähling, T.; Elser, S.; Eberhardt, J. RWU3D: Real World ToF and Stereo Dataset with High Quality Ground Truth. 2023 Twelfth International Conference on Image Processing Theory, Tools and Applications (IPTA). IEEE, 2023, pp. 1–6.
13. Bulczak, D.; Lambers, M.; Kolb, A. Quantified, interactive simulation of AMCW ToF camera including multipath effects. Sensors 2017, 18, 13. [Google Scholar] [CrossRef] [PubMed]
14. Keller, M.; Orthmann, J.; Kolb, A.; Peters, V. A simulation framework for time-of-flight sensors. 2007 International Symposium on Signals, Circuits and Systems. IEEE, 2007, Vol. 1, pp. 1–4.
15. Peters, V.; Loffeld, O.; Hartmann, K.; Knedlik, S. Modeling and bistatic simulation of a high resolution 3D PMD-camera. Proc. Congress on Modelling and Simulation (EUROSIM), 2007.
16. Keller, M.; Kolb, A. Real-time simulation of time-of-flight sensors. Simulation Modelling Practice and Theory 2009, 17, 967–978. [Google Scholar] [CrossRef]
17. Lambers, M.; Hoberg, S.; Kolb, A. Simulation of time-of-flight sensors for evaluation of chip layout variants. IEEE Sensors Journal 2015, 15, 4019–4026. [Google Scholar] [CrossRef]
18. Meister, S.; Nair, R.; Kondermann, D. Simulation of Time-of-Flight Sensors using Global Illumination. VMV, 2013, pp. 33–40.
19. Yan, Z.; Wang, H.; Liu, X.; Ning, Q.; Lu, Y. Physics-Based TOF Imaging Simulation for Space Targets Based on Improved Path Tracing. Remote Sensing 2022, 14, 2868. [Google Scholar] [CrossRef]
20. Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer, 2015, pp. 234–241.
21. Trottier, L.; Giguere, P.; Chaib-Draa, B. Parametric exponential linear unit for deep convolutional neural networks. 2017 16th IEEE international conference on machine learning and applications (ICMLA). IEEE, 2017, pp. 207–214.
22. Hu, J.; Shen, L.; Sun, G. Squeeze-and-excitation networks. Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–7141.
23. Mufti, F.; Mahony, R. Statistical analysis of signal measurement in time-of-flight cameras. ISPRS journal of photogrammetry and remote sensing 2011, 66, 720–731. [Google Scholar] [CrossRef]
24. ifm electronic gmbh. https://www.ifm.com/, 2023. Last accessed in 2023.
25. Chiabrando, F.; Chiabrando, R.; Piatti, D.; Rinaudo, F. Sensors for 3D imaging: Metric evaluation and calibration of a CCD/CMOS time-of-flight camera. Sensors 2009, 9, 10080–10096. [Google Scholar] [CrossRef] [PubMed]
26. Agresti, G.; Schaefer, H.; Sartor, P.; Zanuttigh, P. Unsupervised domain adaptation for tof data denoising with adversarial learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 5584–5593.