1. Introduction

Figure 1. Scan to BIM workflow to create as built model starting from point cloud capturing; after [7].

Figure 1. Scan to BIM workflow to create as built model starting from point cloud capturing; after [7].

Figure 2. Exemplary comparison of acquisition and processing time, as well as resolution, between MLS (Lynx mobile mapper) and TLS (Trimble GX) on 3D point cloud from cultural heritage sites [8].

Figure 2. Exemplary comparison of acquisition and processing time, as well as resolution, between MLS (Lynx mobile mapper) and TLS (Trimble GX) on 3D point cloud from cultural heritage sites [8].

Figure 3. Steps of Scan to BIM showing where our approach fits in the automation of the semantic segmentation stage for 3D point clouds.

Figure 3. Steps of Scan to BIM showing where our approach fits in the automation of the semantic segmentation stage for 3D point clouds.

2. Related Work

2.1. Classical Machine Learning Method

The process of semantic segmentation using classical machine learning methods entails three fundamental stages, as described earlier. In order to obtain a complete comprehension of the workflow, it is essential to provide a concise discussion of each stage, emphasizing the techniques that have been previously applied in related literature. Furthermore, we will review the prior studies that have utilized various methods to tackle the semantic segmentation of distinct objects present in a given scene.

Figure 4. Workflow of point cloud semantic segmentation using classical machine learning method [21].

Figure 4. Workflow of point cloud semantic segmentation using classical machine learning method [21].

2.1.1. Neighborhood Selection

Selecting a respective neighbourhood containing all relevant 3D points around a given point is particularly crucial to define the local features for machine learning algorithms. There are different forms of neighborhood definitions. The most commonly applied ones are spherical neighborhood, cylindrical neighborhood and K-nearest neighborhood. Spherical neighborhood definition refers to the local neighborhood that is formed by all 3D points inside a sphere of a fixed radius [22]. In the definition for cylindrical neighborhoods, the neighborhood is defined by the 2D projections of all 3D points that fall within the sphere of a fixed radius [23]. K-nearest neighborhoods are characterised by a particular number of 3D points within a specific area [24,25]. It is argued that from a purely conceptual stance, radius neighborhood definition should be the appropriate procedure to describe a group of 3D points [26], although it could be less useful when the point density shows strong variations [27]. The three common definitions will be discussed in detail in Section 3.2.1 to show the difference between each of them and which definitions are applied in this work.

To date, studies have examined two approaches for neighborhood scale selection using the above mentioned neighborhood definitions. The first approach is to use a fixed scale parameter (usually either k or the radius) to select the neighborhood around a 3D point. This scale parameter is frequently selected to be identical across all 3D points and relevant to each 3D data set. The main drawbacks of this approach is that in 3D point cloud semantic segmentation most of the outdoor scenes contain objects that vary in size and one can thus argue that using fixed scales inadequately capture this information. Moreover, fixed scale approach is sensitive to the point cloud densities, which is a feature that defines the number of points located in a specific neighborhood. The second approach, which was developed in order to avoid the major drawback of the first approach, considers the point cloud density by identifying an individual neighborhood for each 3D point. It is an optimization for the fixed scale approach, which selects the scale parameters based on local point density, as well as local 3D structure. Besides, this approach offers a solution for the problem of selecting the fixed scale parameter via heuristic or empirical knowledge on the scene, which is the case in the first approach [28,29]. The two most common selection methods for the scale parameter are Eigenentropy based selection [28,30] and dimensionality based selection [29]. The advantages as well as the disadvantages of the two approaches will be mentioned in the methods section showing the ability of the second approach to define a representative neighborhood.

2.1.2. Feature Extraction

Based on the neighborhood defined, a suitable description of a 3D point X is required for the point cloud semantic segmentation. Each neighborhood is characterized by unique features, which can be extracted at a single scale or multi-scale. If a single scale approach is used, one has to make sure that a suitable scale is selected. Extracting features at different scales is a better alternative to a single scale approach, where the behaviour of the local 3D geometry is described through different scales. It has been argued that this approach is significantly more effective, regardless of whether it is used with k-nearest neighborhood [31], with spherical/cylindrical neighborhoods [25,26] or with a combination of all neighborhood types [32]. However, the common drawback between this approach and the first one is the selection of the different scales and the spacing between them, as well as its computational time due to the variation of point densities at each scale. It is suggested that defining a multiscale neighborhood ensures sufficient density at each scale without distortion of the features [18]. From a mathematical point of view, the extracted features occupy different positions in a high dimensional space and are represented by a feature vector. This mathematical aspect is important when trying to distinguish between different classes using different machine learning algorithms like SVM, Random forests, $etc$. Different feature types are used to define this vector:

• parametric

features, which are used for the extraction of surfaces. The surface parameters are estimated by clustering as well as locating the maxima in the parameter space. It is only used for shapes that can be described with few parameters only (e.g. cylinders, spheres or planes) [33].

• sampled

features, in which the geometric relations between 3D points within the local neighborhood are sampled. As a result, a description of the local context is obtained. Histograms represent the different relations, such as angles and angular variations, as well as distances. Blomley et al. [34] emphasis the difficulty for humans to interpret the entries of these resulting feature vectors.

• metrical

features, such as shape measures. In this type, a single value, which specifies one particular property, represents each feature. Calculating the 3D structure tensor from the 3D coordinates of all the points in a given neighborhood, generates interpretable features. For more details on structure tensor, see Section 3.2.1.2 on selecting scale of the neighborhood. The use of eigenvalues of this 3D structure tensor introduces features describing various structures (linear, planar or volumetric) [35]. One can also add even further geometric features to the standard geometric features (such as anisotropy or omnivariance) [36]. In some cases more metrical features like full wave and echo based features are added to the feature vector in order to distinguish between classes, which have large similarity in terms of geometric features [37].

2.1.3. Semantic Segmentation

In terms of segmentation and classification of point clouds, one usually differentiates between three types of machine learning approaches, namely, unsupervised, semi-supervised and supervised learning. In unsupervised learning techniques, the algorithm will analyze the data for the features within it, to determine features that actually correlate between two data items. This approach is being used for clustering, dimensionality reduction and feature learning tasks. A survey on unsupervised machine learning algorithms for different proposes is presented by Khanum et al. [38]. As indicated by its name, semi-supervised learning is partially linked to both supervised and unsupervised learning. It minimizes the input of the user and at the same time maximizes the generalization ability of the classifier. In this case, the algorithm makes use of the labeled and unlabelled data to improve the learning accuracy and thus improving the classification performance of the model. The literature suggests that this learning method can be used for point cloud semantic segmentation tasks where there is difficulty to get robust supervision information [39]. An up-to-date overview of semi-supervised learning methods focusing on semi-supervised classification is introduced by Van Engelen, Jesper E. and Hoos, Holger H. [40]. In a supervised approach, the classification model will find the separation between categories based on the training samples given to the model, which contain the features and their categories. The strength of this method is that the outputs always have a probabilistic interpretation and the algorithm can be regularized to avoid overfitting.

Supervised machine learning algorithms have shown reasonable results when dealing with point cloud semantic segmentation processes [28,41]. This method is also applied by Brodu, Nicolas and Lague, Dimitri [26] to segment and classify complex natural scenes using features derived from different scales.

Figure 5. Result of the classification process for the mountain river data set. A) Original river scene, B) Classification( green: vegetation, gray: bedrock, red: gravel, blue: water) [26].

Figure 5. Result of the classification process for the mountain river data set. A) Original river scene, B) Classification( green: vegetation, gray: bedrock, red: gravel, blue: water) [26].

There are different supervised machine learning algorithms (random forest, k-nearst neighbor, naive bayes, $etc$) that are used for 3D point cloud semantic segmentation. Support Vector Machine (SVM) is among the most preferred methods for these tasks, due to its high performance on linear and non linear problems. In the field of computer vision and image processing SVM showed high accuracy in identification and classification tasks [42,43]. SVM was also investigated along with various algorithms for point cloud semantic segmentation and delivered a reasonable performance [28,41]. In other scientific fields, Osisanwo et al. [44] investigated different supervised machine learning techniques and compared between different classification algorithms, based on the data set given, to show which algorithm give the best performance. The comparison between seven algorithms showed that SVM was found to be the algorithm with the most accuracy. Using the Kernel trick, it is possible to map the input features into high dimensional feature spaces, in order to identify the optimal hyperplane separating all samples of each category [45], thus, achieving better results, see Figure 2.3.

Figure 6. Using kernel function to map the data set from 2D space to 3D space and constructing a separating surface between two classes [46].

Figure 6. Using kernel function to map the data set from 2D space to 3D space and constructing a separating surface between two classes [46].

3D point cloud of infrastructure scenes usually contains more than two objects (vegetation, roads, bridges, $etc$), meaning it is a multi-class classification task. As such, a suitable strategy is needed to solve this kind of tasks using supervised binary classifiers, which differentiate only between two classes. Multi-class classification problems can be reduced into multiple numbers of binary classification problems using different numbers of binary classifiers. This is done by training each binary classifier to distinguish between two classes in a scene, thus covering all the classes of an infrastructure asset. Such a strategy is known as transformation to binary [47]. Two techniques are developed to reduce the multi-class problem. The first technique is called one vs one. Here, each class is compared to another class, resulting in a number of classifiers that can be calculated using the following equation:

$$k=\frac{n(n-1)}{2}$$

(2.1)

where n is number of classes in the scene. The multi-class classification is split into one binary classification problem for each pair of classes. The second technique is called one vs rest, where each class is compared to all other classes and the number of classes is equal to the number of classifiers. Here, the multi-class classification is split into one binary classification problem per class. This technique was used in the following literature [48].

The two techniques discussed above are applied in road inventory studies, where the 3D point cloud of basic structures obtained from MLS is classified [49]. In that work a hierarchical segmentation and classification was adapted, where the scene is firstly rough segmented into two main categories based on the major differences between their features (e.g. ground and above ground objects). Afterwards, the objects located in each category will be further classified to sub-classes by assigning a label to each segment based on its properties. The scene is thus semantically segmented through different stages.

As a binary classifier, SVM was used in several works to solve multi-class classification tasks using the different techniques mentioned previously [48,50]. An idea was introduced to use SVM classifier with multi-class classification problems by combining it with a decision tree approach [50].

2.2. Deep Learning Method

State-of-the-art point cloud semantic segmentation approaches are based on deep learning techniques [51,52]. The workflow of these techniques can be summarized into three main steps shown in the following figure.

Figure 7. Deep learning workflow showing the three main steps [53].

Figure 7. Deep learning workflow showing the three main steps [53].

• forward propagation,

where the input point cloud is fed into the deep learning model and the model generates an output. During this step, the point cloud is transformed into a set of feature maps, which are then passed through multiple layers of the neural network to generate a final prediction.

• backward propagation,

the error calculated from the loss function is propagated back through the network to adjust the weights and biases of each layer in the model. This involves calculating the gradient of the loss function with respect to the weights and biases and using this information to update the values of these parameters.

• optimization,

an optimization algorithm, such as stochastic gradient descent (SGD) or Adam, to adjust the weights and biases of the neural network based on the gradients calculated in the backward propagation step. The objective is to find the values of the parameters that minimize the loss function.

The most common problem of points in a point cloud is that these points are irregular and cannot easily be handled. Most researchers have to transform these points to different forms, such as 3D voxel grid [54] or collection of images before passing these data to a deep net architecture. Since this transformation has drawbacks, another way for handling this problem is using a unified architecture called PointNet. PointNet is a pioneering architecture that processes set of points without voxelization by using multi-layer perceptrons to extract features for each point [55,56]. Then all point features are accumulated using a symmetric function. Finally, the information of neighboring points is effectively handled jointly. However, PointNet does not detect local features induced by the points in the metric space limiting its ability to generalize complex scenes. To address this drawback, an extension of PointNet, called PointNet++, is designed [57]. PointNet ++ applies PointNet recursively on a portioning of input point cloud to capture local features at different scales. The architectures of PointNet and PointNet++ utilize a Multi Layer Perceptron (MLP) based method, which has been previously employed in the literature for various classification and segmentation tasks. MLP is a type of feedforward neural network that consists of multiple layers of perceptrons or nodes, each of which performs a linear transformation on the input followed by a non-linear activation function. This allows for the MLP to model non-linear relationships between input and output data.

Other deep learning-based methods have been proposed in the literature for point cloud semantic segmentation [51,52].

2.3. Advantages and Limitations

Although deep learning approaches deliver good results, they still have some shortcomings, such as their requirement of large data sets and computational resources during the training stage and the difficulty of interpreting deep networks [58].

Classical machine Learning methods might be useful to address these shortcomings. Zhang et al. [59] proposed a machine learning method, called the PointHop method, for the semantic segmentation of point cloud.

Figure 8. Comparison between deep learning based methods and PointHop method showing that PointHop requires only one forward pass to learn parameters of the system [59].

Figure 8. Comparison between deep learning based methods and PointHop method showing that PointHop requires only one forward pass to learn parameters of the system [59].

In terms of time complexity, PointHob system was compared to deep learning based methods, see Figure (2.6). The training time need by the developed system was significantly lowen than the other methods.

Figure 9. Comparison of the time complexity between Pointnet, Pointnet++ and PointHop method [59].

Figure 9. Comparison of the time complexity between Pointnet, Pointnet++ and PointHop method [59].

SVM as a classical machine learning algorithm was investigated alongside two other artificial neural networks, Recurrent Neural Networks (RNN) and Feed Forward Neural Networks (FFNN), in the field of autonomous driving [19]. The task aimed to compare all three techniques within the context of lane change behaviour by humans in a simulated environment. The different approaches were evaluated using different feature combinations and the results showed that SVM generated the best outcomes and was able to successfully define the most appropriate feature combinations. This shows that under the right conditions, SVMs can compete with or even outperform deep learning techniques.

In point cloud semantic segmentation different classical machine learning algorithms were implemented and compared to deep learning models [28]. The results of the study indicated that, with regard to the utilization of characteristic features, classical methods may outperform deep learning models.

3. Methods

3.1. Hierarchy Support Vector Machine Approach

In 3D point cloud semantic segmentation using supervised learning methods, the necessary inputs for the model are features and the ground truth labeling. As a supervised machine learning model, SVM is used in this work to semantically segment the 3D point cloud of the infrastructure scene. Using the two techniques mentioned in Section 2.1.3, one vs one and one vs rest, the whole scene can be segmented using a hierarchical approach derived from knowledge-based rules or insights about the scene. In this approach the 3D point cloud is semantically segmented through three layers, see Figure 3.1. At each layer different features and different techniques are used to separate between the multiple categories in that layer. In the first layer, the scene is classified into two categories, namely vegetation and man made objects, using one vs one technique. In the second layer, the classifier differentiates between ground and above ground objects using the same technique in the previous layer. Lastly, the 3D point cloud of the above ground objects is semantically segmented into different segments as abutment, footing, girder, railing and pier. In this layer, SVM classifier is applied using one vs rest technique.

Figure 10. The hierarchy classification approach combining SVM solutions and decision tree framework.

Figure 10. The hierarchy classification approach combining SVM solutions and decision tree framework.

3.2. Neighborhood Selection

This section presents the method used to determine the neighborhood of a 3D point in point cloud semantic segmentation. Traditional methods often rely on randomly selected scale parameters, such as r or k, based on empirical knowledge of the scene. In contrast, our approach, inspired by [28], leverages the local 3D structure of the neighborhood to define the optimal scale parameter for each 3D point. This integration between neighborhood’s parameter selection and its local 3D structure ensures that each point’s neighborhood is the most representative, or optimal, for that specific point.

3.2.1. Optimal Neighborhood Definition

The concept of an optimal neighborhood refers to the largest cluster of spatially proximal 3D points that belong to the same category as a point of interest. The process of determining this neighborhood involves answering two primary questions: (1) what type of neighborhood should be used, and (2) what is the appropriate scale for this neighborhood? In the following sections, each of these questions will be discussed in detail to provide a comprehensive understanding of the process.

3.2.1.1. Neighborhood Type

For better understanding of the type of the neighborhood applied in this work it is worth to discuss the main difference between the common types. Unlike spherical and cylindrical neighborhoods, k-nearest neighborhoods do not include fixed spatial neighborhood sizes. Moreover, the selection of k points is based on the nearest euclidean distance to reference point X, both in 2D and 3D [24,25]. On the other hand, the radius neighborhood definitions correspond to a specific part of the space and this property is a vital component to assign a more consistent geometrical meaning to the features. For these reasons, k-nearest neighborhood type was excluded for the semantic segmentation task in this work. Figure 3.2 shows the difference between the behaviour of spherical neighborhood and k-nearest neighborhood under two different scales.

Due to the characteristic of the cylindrical neighborhoods, this type of neighborhoods is found to be useful for delivering robust height information when the scene contains vertical man-made objects ($e.g.$ buildings, facades). As a result, this type will be applied in this work along with the spherical neighborhood definition.

Figure 11. Behavior of the KNN and spherical neighborhoods under two different scales (scale 1, scale 2) at three different locations in the scene (A, B, C) [18].

Figure 11. Behavior of the KNN and spherical neighborhoods under two different scales (scale 1, scale 2) at three different locations in the scene (A, B, C) [18].

Under the Euclidean norm, the spherical neighborhood ${\mathcal{N}}_i^r$ of point $p_i$ at scale r is defined as the set of points $p_k$ verifying:

$$p_k\in {\mathcal{N}}_i^r\iff ∥p_i-p_k∥\le r$$

(3.1)

3.2.1.2. Selecting Scale of the Optimal Neighborhood

Selection of the radius of 3D point neighborhood is considered a heuristic or an empiric selection and specific to each data set. This means that the decision about the suitability of a neighborhood from a geometrical point of view is delayed to the classifier after obtaining the results. Showing the usefulness of the individual neighborhood, discussed in Section 2.1.1, a generic selection of the neighborhood scale may be an alternative allowing a generalization of data sets and results in obtaining an optimal neighborhood.

The optimization of the parameter selection may be based on different criteria. The methodology introduced in this work aims to find automatically the optimal neighborhood radius based on the entropy feature ($E_f$). The idea is to find the optimal radius that achieves the best separation among the three main geometric patterns (linearity, planarity, scatter). Using Shannon entropy [60] to find the distribution of these three geometric behavior, an energy function can be developed to define the optimal radius:

$$E_f\left({\mathcal{N}}_{p_i}^r\right)=-a_{1D}ln\left(a_{1D}\right)-a_{2D}ln\left(a_{2D}\right)-a_{3D}ln\left(a_{3D}\right)$$

(3.2)

This energy function is based on the covariance matrix, which is also called 3D structure tensor and is calculated from the 3D coordinates of the neighborhood (${\mathcal{N}}_{p_i}^r$). $a_{1D}$, $a_{2D}$ and $a_{3D}$ represent linearity, planarity and scatter, respectively, and can be calculated using the eigenvalues of the covariance matrix. The following equation can be used to calculate the 3D structure tensor:

$$\frac{1}{|N_i|+1}\sum _{p_j\in \left\{N_i,p_i\right\}}(p_j-\overline{p}){(p_j-\overline{p})}^T$$

(3.3)

where $\overline{p}$ denotes the center of gravity of $\left\{N_i,p_i\right\}$, $N_i$ is the spherical neighborhood around reference point $p_i$.

Denoting the eigenvalues of the 3D structure tensor by ${⋋}_{1,i},{⋋}_{2,i},{⋋}_{3,i}\in \mathbb{R}$, where ${⋋}_{1,i}\ge {⋋}_{2,i}\ge {⋋}_{3,i}\ge 0$, the three geometric features ( $a_{1D}$, $a_{2D}$ , $a_{3D}$ ) can be calculated as follows:

$$a_{1D}=\frac{{⋋}_1-{⋋}_2}{{⋋}_1}$$

(3.4)

$$a_{2D}=\frac{{⋋}_2-{⋋}_3}{{⋋}_1}$$

(3.5)

$$a_{3D}=\frac{{⋋}_3}{{⋋}_1}$$

(3.6)

The lower $E_f\left(N_{p_i}^r\right)$ in equation (3.2) is, the more one dimensionality prevails over the two other ones. This criterion allows to define an optimal radius $r_{E_f}^{*}$ that minimizes $E_f\left(N_{p_i}^r\right)$ in the [$r_{min},r_{max}$] space:

$$r_{E_f}^{*}=argmin{E}_f\left(N_{p_i}^r\right)$$

(3.7)

where $r\in [r_{min},r_{max}]$

An illustration of the neighborhood selection can be shown in Figure 3.3. Here, the optimal scale that achieves the best separation between the three main geometric features for the selected part from the scene, the white box in Figure 3.3.a, is determined. This value corresponds to the lowest value of $E_f\left(N_{p_i}^r\right)$ in Equation 3.2. One can observe that by increasing the scale, the pier of the bridge looks more linear and the linearity feature dominates over the other two features.

Figure 12. Finding the optimal neighborhood around the pier of the bridge.

Figure 12. Finding the optimal neighborhood around the pier of the bridge.

It is challenging to choose the boundaries of the radius, i.e, $(r_{min},r_{max})$, because they depend on multiple characteristics of the given data and are therefore specific for each data set. In one piece of literature, where TLS and MMS data set are investigated, 16 values were selected to represent the [$r_{min},r_{max}$] space [29]. There, the choice of the upper boundary was not critical and was selected based on the largest object in the scene, facades (3m). However, the selection of the lower boundary was found to depend on different aspects such as: the noise in the point cloud, the specifications of the sensor and the computational constraints. These different aspects were addressed in detail in the original work. Given that the radius of interest is frequently close to the ($r_{min}$), r values are not increased linearly but with a square factor instead. As a result, one notices an increase in samples near the radius of interest and a decrease of sample when reaching maximal values.

The paragraphs above highlighted the problem of point density and the scale of the neighborhood used. The problem concerning the lower boundary can be solved by controlling the number of points in each scale. An approach was introduced to avoid the strong variations of point density at different scales using the following relation [18]:

$$\rho =\frac{r}{l}$$

(3.8)

where $\rho$ is point density, l is the size of the grid cell used for downsampling the point cloud and r is the radius of neighborhood.

4. Results

(i)

precision (measure of correctness)

$$\mathrm{precision}=\frac{TP}{TP+FP}$$

(4.1)

(ii)

recall (measure of completeness)

$$\mathrm{recall}=\frac{TP}{TP+FN}$$

(4.2)

(iii)

accuracy (measures overall performance of the model)

$$\mathrm{accuracy}=\frac{TP+TN}{TP+FN+FP+TN}$$

(4.3)

Figure 13. Confusion matrix .

Figure 13. Confusion matrix .

4.2. Evaluation Data Sets

To assess the effectiveness of the hierarchical semantic segmentation method employing SVM algorithm and the deep learning method using Pointnet architecture, several real-world data sets of infrastructure objects are utilized. These data sets contain different types of scenes from Straßen NRW Nord in Germany. One type of scene displays roads without bridges and the second type of scene illustrates roads with bridges (see Figure 4.2 and 4.3, respectively). All scenes were captured using MLS.

Nevertheless, certain differences exist between the evaluation of the two methods with respect to the characteristics of the data sets utilized as well as the number of points employed for training and testing the models.

Figure 14. Data set, scene one: Noise barriers, road, vegetation and signs in the scene of Straßen NRW with laser scan intensities shown in gray scale color.

Figure 14. Data set, scene one: Noise barriers, road, vegetation and signs in the scene of Straßen NRW with laser scan intensities shown in gray scale color.

Figure 15. Data set, scene two: bridge, vegetation and road in the scene of Straßen NRW with laser scan intensities shown in gray scale color.

Figure 15. Data set, scene two: bridge, vegetation and road in the scene of Straßen NRW with laser scan intensities shown in gray scale color.

4.2.1. SVM Data Sets

In order to assess the efficacy of the SVM algorithm, two data sets from different scenes were utilized for training and testing. In the training phase, each class was represented by 1000 points, yielding a total of 5000 points for scene one and 7000 points for scene two, respectively. During the validation process, a total of 209,400 points were used for scene one and 149,114 points were used for scene two. Notably, the validation data sets were independent from the training data sets. The ground truth label of these scenes contained a total of 9 classes, encompassing all infrastructure buildings present in the scene. Table 4.1 summarizes the classes and number of points used in training and validation stages.

Table 1. Summary for data points used in the training and validation of SVM model

Table 1. Summary for data points used in the training and validation of SVM model

Data set	Classes	Training ($\mathit{p}\mathit{t}\mathit{s}$)	Validation ($\mathit{p}\mathit{t}\mathit{s}$)
scene one	plant, road, railing, noise barrier, signs	1000	209,400
scene two	plant, road, railing, abutment, girder, pier, footing	1000	149,114

4.2.2. Pointnet Data Sets

The evaluation of the PointNet model was performed using 22 data sets comprising the objects of scene one. It is noteworthy that the PointNet model utilized a significantly greater number of points for both training and evaluation than the Support Vector Machine (SVM) model. Specifically, 8,691,712 points were employed for training the PointNet model and 1,634,304 points for evaluation.

4.3. Hierarchical Support Vector Machine Approach

Our developed approach was divided into three classification layers and is explained in detail in Section 3.1. In the following table, the features used in each classification layer are summarized, along with the number of classifiers. Moreover, during the training process of the classifiers two different labelling strategies are employed and their performance is evaluated in this section.

Table 2. Hierarchy classification features at each layer with the number of classifiers for the first scene

Table 2. Hierarchy classification features at each layer with the number of classifiers for the first scene

Classification layer	Features	Number of classifiers
layer 1	planarity, linearity, sphericity	1
layer 2	planarity, linearity, sphericity, maximal height, standard deviation of height	1
layer 3	planarity, linearity, sphericity, maximal height, standard deviation of height, verticality	3

The first scene from the data set captures a high way and contains 5 classes. The overall accuracy of the SVM model for this scene was 93.94% . The classification performance for each class is shown in detail in the following table.

Table 3. Performance metrics of all classes for HSVM semantic segmentation approach for scene one

Table 3. Performance metrics of all classes for HSVM semantic segmentation approach for scene one

Classes	Precision[%]	Recall[%]
plant	80.7	100
road	98.1	97.7
railing	97.2	67.6
noise barrier	95.4	94.63
signs	98.3	97.4

Figure 16. The semantic segmentation result of Straßen NRW data set for scene one, using Hierarchical SVM approach.

Figure 16. The semantic segmentation result of Straßen NRW data set for scene one, using Hierarchical SVM approach.

The second scene contains more classes due to the presence of the bridge elements. The performance of the SVM approach along with the evaluation metrics for each class are shown in Table 4.4 and Figure 4.5, respectively.

Table 4. Performance metrics of all classes for HSVM semantic segmentation approach for scene two.

Table 4. Performance metrics of all classes for HSVM semantic segmentation approach for scene two.

Classes	Precision[%]	Recall[%]
plant	90.00	81.67
road	95.52	100
railing	79.81	85.84
abutment	96.70	99.11
girder	79.61	100
pier	97.06	99.85
footing	94.01	90.93

Figure 17. The semantic segmentation result of Straßen NRW data set for scene two, using Hierarchical SVM approach.

Figure 17. The semantic segmentation result of Straßen NRW data set for scene two, using Hierarchical SVM approach.

4.3.1. Evaluating Hierarchical SVM Approach Using Different Learning Strategies

To explore the effects of the employed learning strategies during the training phase of the HSVM approach, the second data set of scene two was utilized. In each layer, the model was trained on both strategies, and the evaluation results based on each training method are presented in the following figures.

Figure 18. Comparison between both learning strategies employed in the training process of the HSVM approach for data set of scene two.

Figure 18. Comparison between both learning strategies employed in the training process of the HSVM approach for data set of scene two.

4.3.2. Evaluating Hierarchical SVM Approach on Additional Data Sets

The trained SVM model is applied to additional point clouds from infrastructure scenes and the results are shown in the figures below. Two data sets are used with total number of 296,141 points. Table 5 summarizes the classes and number of points used for the evaluation for each data set. These data sets are evaluated using different scales for the neighborhoods. The ranges of the radii were set to 0.5 ($r_{min}$) and 1 ($r_{max}$). These scales were chosen heuristically based on the characteristics and the quality of each data set. The data sets are shown in following figures.

Table 5. Additional data sets used to evaluate the model.

Table 5. Additional data sets used to evaluate the model.

Data set	Classes	Evaluation ($\mathit{p}\mathit{t}\mathit{s}$)
additional scene 1	plant, road, railing, abutment, girder, pier, footing	201,029
additional scene 2	plant, road, railing, noise barrier, signs	95,112

Figure 19. Classification results of additional data set 1 of infrastructure scene containing 7 classes.

Figure 19. Classification results of additional data set 1 of infrastructure scene containing 7 classes.

Figure 20. Classification results of additional data set 2 of infrastructure scene containing 5 classes.

Figure 20. Classification results of additional data set 2 of infrastructure scene containing 5 classes.

4.4. PointNet Model

Despite the simplicity of the input feature vector for the PointNet model, consisting solely of the raw x, y, and z coordinates of each point in the 3D point cloud, the results of its semantic segmentation performance are noteworthy and exhibit its ability to accurately segment objects in a scene. Despite this success, the optimization of its parameters remains a laborious task with further potential for improvement. The direct impact of these parameters on the model’s performance makes their definition highly significant.

4.4.1. Hyperparameters of Pointnet Model

• Learning rate:

is a hyperparameter that determines the speed at which the model learns from the training data, and can have a significant impact on the accuracy and convergence of the model.

• Batch size:

is a hyperparameter that defines the number of training examples used in one iteration.

• Number of points in each batch:

the number of points in each batch refers to the quantity of down-sampled points extracted from the original point cloud and presented to the model as batches. These points should be representative of the characteristics of the original point cloud within the voxel, thereby ensuring that the down-sampling process retains the essential features of the original data.

• Epochs:

an epoch refers to a single iteration through the entire training data set during the training of a neural network. The number of epochs determines the number of times that the learning algorithm will work through the entire training data set.

The parameters employed in the training of the model are detailed in the accompanying table.

Table 6. Training parameters of the PointNet model.

Table 6. Training parameters of the PointNet model.

Parameters	Values
learning rate	0.001
batch size	32
number of points in each batch	4096
epochs	500

The results of the training and evaluation stages of the PointNet model are summarized in Figure 4.5, where the accuracy of the model is presented.

Figure 21. Performance of the PointNet model using the raw xyz coordinates of each point as an input feature vector.

Figure 21. Performance of the PointNet model using the raw xyz coordinates of each point as an input feature vector.

As evidenced by the figures presented above, the training accuracy of the model is 95%, demonstrating the proficiency of its deep learning architecture in capturing intricate relationships within the data. Conversely, the test accuracy is approximately 80%, indicating the potential of the model to generalize to unseen data sets. Given the imbalanced nature of the training and test data sets, it is more informative to examine the accuracy of each class in the test data set to gain a comprehensive understanding of the model’s performance. The following figures present the distribution of each class in the test data set and the respective precision scores.

Figure 22. Imbalanced test data set and different precision values indicating the performance of the model which is also trained on imbalanced data set.

Figure 22. Imbalanced test data set and different precision values indicating the performance of the model which is also trained on imbalanced data set.

5. Discussion

Figure 23. Comparison between HSVM approach and Pointnet in terms of precision values for classes in data set of scene one.

Figure 23. Comparison between HSVM approach and Pointnet in terms of precision values for classes in data set of scene one.

6. Conclusions

Abbreviations

References

1. Bryde, D.; Broquetas, M.; Volm, J.M. The project benefits of Building Information Modelling (BIM). International Journal of Project Management 2013, 31, 971–980. [Google Scholar] [CrossRef]
2. Wong, A.; Wong, F.; Nadeem, A. Attributes of Building Information Modelling Implementations in Various Countries. Architectural Engineering and Design Management 2010, 6, 288–302. [Google Scholar] [CrossRef]
3. Vilgertshofer, S.; Borrmann, A.; Martens, J.; Blut, T.; Becker, R.; Blankenbach, J.; Göbels, A.; Beetz, J.; Celik, F.; Faltin, B.; König, M. TwinGen: Advanced technologies to automatically generate digital twins for operation and maintenance of existing bridges. 2022.
4. Ndekugri, I.; Braimah, N.; Gameson, R. Delay Analysis within Construction Contracting Organizations. Journal of Construction Engineering and Management-asce 2008, 134, 692–700. [Google Scholar] [CrossRef]
5. Huang, Z.; Wen, Y.; Wang, Z.; Ren, J.; Jia, K. Surface Reconstruction from Point Clouds: A Survey and a Benchmark. 2022; arXiv:cs.CV/2205.02413. [Google Scholar]
6. Sharma, R.; Abrol, P. Parameter Extraction and Performance Analysis of 3D Surface Reconstruction Techniques. International Journal of Advanced Computer Science and Applications 2023, 14. [Google Scholar] [CrossRef]
7. Pătrăucean, V.; Armeni, I.; Nahangi, M.; Yeung, J.; Brilakis, I.; Haas, C. State of research in automatic as-built modelling. Advanced Engineering Informatics 2015, 29, 162–171, Infrastructure Computer Vision. [Google Scholar] [CrossRef]
8. Rodríguez-Gonzálvez, P.; Jiménez Fernández-Palacios, B.; Muñoz-Nieto.; Angel.; Arias, P.; González-Aguilera, D. Mobile LiDAR System: New Possibilities for the Documentation and Dissemination of Large Cultural Heritage Sites. Remote Sensing 2017, 9, 189. [Google Scholar] [CrossRef]
9. Ariyachandra, M.; Brilakis, I. Understanding the challenge of digitally twinning the geometry of existing rail infrastructure. 2019. [CrossRef]
10. Grilli, E.; Menna, F.; Remondino, F. a Review of Point Clouds Segmentation and Classification Algorithms. ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 2017, 42W3, 339–344. [Google Scholar] [CrossRef]
11. Li, Q.; Yuan, P.; Lin, Y.; Tong, Y.; Liu, X. Pointwise classification of mobile laser scanning point clouds of urban scenes using raw data. Journal of Applied Remote Sensing 2021, 15, 024523. [Google Scholar] [CrossRef]
12. Lu, H.; Shi, H. Deep Learning for 3D Point Cloud Understanding: A Survey. CoRR 2020, abs/2009.08920. [Google Scholar]
13. Martens, J.; Blankenbach, J. VOX2BIM+ - A Fast and Robust Approach for Automated Indoor Point Cloud Segmentation and Building Model Generation. PFG – Journal of Photogrammetry, Remote Sensing and Geoinformation Science 2023. [Google Scholar] [CrossRef]
14. Martens, J.; Blankenbach, J. An evaluation of pose-normalization algorithms for point clouds introducing a novel histogram-based approach. Advanced Engineering Informatics 2020, 46, 101132. [Google Scholar] [CrossRef]
15. Thomson, C.; Boehm, J. Automatic Geometry Generation from Point Clouds for BIM. Remote Sensing 2015, 7, 11753–11775. [Google Scholar] [CrossRef]
16. Anandakumar, R.; Nidamanuri, R.; Krishnan, R. Semantic labelling of Urban point cloud data. 2014, Vol. XL-8. [CrossRef]
17. Mukhamediev, R.I.; Symagulov, A.; Kuchin, Y.; Yakunin, K.; Yelis, M. From Classical Machine Learning to Deep Neural Networks: A Simplified Scientometric Review. Applied Sciences 2021, 11. [Google Scholar] [CrossRef]
18. Thomas, H.; Goulette, F.; Deschaud, J.E.; Marcotegui, B.; LeGall, Y. Semantic Classification of 3D Point Clouds with Multiscale Spherical Neighborhoods. 2018 International Conference on 3D Vision (3DV); IEEE: Verona, 2018; pp. 390–398. [Google Scholar] [CrossRef]
19. Dogan, Ü. ; Edelbrunner, J.; Iossifidis, I. Autonomous driving: A comparison of machine learning techniques by means of the prediction of lane change behavior. 2011 IEEE International Conference on Robotics and Biomimetics, 2011, pp. 1837–1843. [Google Scholar] [CrossRef]
20. Lu, Z.; Happee, R.; Cabrall, C.D.; Kyriakidis, M.; de Winter, J.C. Human factors of transitions in automated driving: A general framework and literature survey. Transportation Research Part F: Traffic Psychology and Behaviour 2016, 43, 183–198. [Google Scholar] [CrossRef]
21. Yang, S.; Hou, M.; Li, S. Three-Dimensional Point Cloud Semantic Segmentation for Cultural Heritage: A Comprehensive Review. Remote Sensing 2023, 15. [Google Scholar] [CrossRef]
22. Lee, I.; Schenk, T. Perceptual organization of 3D surface points. In: The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. XXXIV, Part 3A, 2002, pp. 193–198.
23. Filin, S.; Pfeifer, N. Neighborhood Systems for Airborne Laser Data. Photogrammetric Engineering & Remote Sensing 2005, 71, 743–755. [Google Scholar] [CrossRef]
24. Linsen, L.; Prautzsch, H. Local Versus Global Triangulations. Eurographics 2001 - Short Presentations. Eurographics Association, 2001. [CrossRef]
25. Niemeyer, J.; Rottensteiner, F.; Soergel, U. Contextual classification of lidar data and building object detection in urban areas. ISPRS Journal of Photogrammetry and Remote Sensing 2014, 87, 152–165. [Google Scholar] [CrossRef]
26. Brodu, N.; Lague, D. 3D terrestrial lidar data classification of complex natural scenes using a multi-scale dimensionality criterion: Applications in geomorphology. ISPRS Journal of Photogrammetry and Remote Sensing 2012, 68, 121–134. [Google Scholar] [CrossRef]
27. Hackel, T.; Wegner, J.D.; Schindler, K. FAST SEMANTIC SEGMENTATION OF 3D POINT CLOUDS WITH STRONGLY VARYING DENSITY. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences 2016, pp. 177–184. [Google Scholar]
28. Weinmann, M.; Jutzi, B.; Hinz, S.; Mallet, C. Semantic point cloud interpretation based on optimal neighborhoods, relevant features and efficient classifiers. ISPRS Journal of Photogrammetry and Remote Sensing 2015, 105, 286–304. [Google Scholar] [CrossRef]
29. Demantké, J.; Mallet, C.; David, N.; Vallet, B. DIMENSIONALITY BASED SCALE SELECTION IN 3D LIDAR POINT CLOUDS. ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 2012, XXXVIII-5/W12, 97–102. [Google Scholar] [CrossRef]
30. Weinmann, M.; Jutzi, B.; Mallet, C. Semantic 3D scene interpretation: A framework combining optimal neighborhood size selection with relevant features. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences 2014, II3, 181–188. [Google Scholar] [CrossRef]
31. Pauly, M.; Keiser, R.; Gross, M. Multi-scale Feature Extraction on Point-Sampled Surfaces. Computer Graphics Forum 2003, 22, 281–289. [Google Scholar] [CrossRef]
32. Blomley, R.; Jutzi, B.; Weinmann, M. CLASSIFICATION OF AIRBORNE LASER SCANNING DATA USING GEOMETRIC MULTI-SCALE FEATURES AND DIFFERENT NEIGHBOURHOOD TYPES. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences 2016, III-3, 169–176. [Google Scholar] [CrossRef]
33. Vosselman, G.; Gorte, B.; Sithole, G.; B, T. Recognising structure in laser scanner point clouds. Inter. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 2004, 46. [Google Scholar]
34. Blomley, R.; Weinmann, M.; Leitloff, J.; Jutzi, B. Shape distribution features for point cloud analysis and A geometric histogram approach on multiple scales. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences 2014, II-3, 9–16. [Google Scholar] [CrossRef]
35. Jutzi, B.; Groß, H. NEAREST NEIGHBOUR CLASSIFICATION ON LASER POINT CLOUDS TO GAIN OBJECT STRUCTURES FROM BUILDINGS. 2009.
36. Weinmann, M.; Jutzi, B.; Mallet, C. Feature relevance assessment for the semantic interpretation of 3D point cloud data. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences 2013, II-5/W2, 313–318. [Google Scholar] [CrossRef]
37. Chehata, N.; Guo, L.; Mallet, C. AIRBORNE LIDAR FEATURE SELECTION FOR URBAN CLASSIFICATION USING RANDOM FORESTS. Laserscanning; , 2009.
38. Khanum, M.; Mahboob, T.; Imtiaz, W.; Ghafoor, H.A.; Sehar, R. A survey on unsupervised machine learning algorithms for automation, classification and maintenance. International Journal of Computer Applications 2015, 119. [Google Scholar] [CrossRef]
39. Chen, L.; Zhang, Y.; Lin, Y.; Jiang, M.; Huang, Y.; Lei, Y. Consistency-Based Semi-Supervised Learning for Point Cloud Classification. 2021 4th International Conference on Pattern Recognition and Artificial Intelligence (PRAI), 2021, pp. 440–445. [CrossRef]
40. Van Engelen, J.E.; Hoos, H.H. A survey on semi-supervised learning. Machine Learning 2020, 109, 373–440. [Google Scholar] [CrossRef]
41. Park, Y.; Guldmann, J.M. Creating 3D City Models with Building Footprints and LiDAR Point Cloud Classification: A Machine Learning Approach. Computers Environment and Urban Systems 2019, 75, 76–89. [Google Scholar] [CrossRef]
42. Hu, J.; Li, D.; Duan, Q.; Yueqi, H.; Chen, G.; Si, X. Fish species classification by color, texture and multi-class support vector machine using computer vision. Computers and Electronics in Agriculture 2012, 88, 133–140. [Google Scholar] [CrossRef]
43. Zhang, Y.D.; Wu, L. Classification of Fruits Using Computer Vision and a Multiclass Support Vector Machine. Sensors (Basel, Switzerland) 2012, 12, 12489–505. [Google Scholar] [CrossRef]
44. Osisanwo, F.; Akinsola, J.; Awodele, O.; Hinmikaiye, J.; Olakanmi, O.; Akinjobi, J. Supervised machine learning algorithms: classification and comparison. International Journal of Computer Trends and Technology (IJCTT) 2017, 48, 128–138. [Google Scholar]
45. Murty, M.N.; Raghava, R. Kernel-Based SVM. In Support Vector Machines and Perceptrons: Learning, Optimization, Classification, and Application to Social Networks; Springer International Publishing: Cham, 2016; pp. 57–67. [Google Scholar] [CrossRef]
46. Hachimi, M.; Kaddoum, G.; Gagnon, G.; Illy, P. Multi-stage Jamming Attacks Detection using Deep Learning Combined with Kernelized Support Vector Machine in 5G Cloud Radio Access Networks. 2020.
47. Rivolli, A.; Read, J.; Soares, C.; Pfahringer, B.; de Carvalho, A.C. An empirical analysis of binary transformation strategies and base algorithms for multi-label learning. Machine Learning 2020, 109, 1509–1563. [Google Scholar] [CrossRef]
48. Chen, C.; Li, X.; Belkacem, A.N.; Qiao, Z.; Dong, E.; Tan, W.; Shin, D. The Mixed Kernel Function SVM-Based Point Cloud Classification. International Journal of Precision Engineering and Manufacturing 2019, 20, 737–747. [Google Scholar] [CrossRef]
49. Pu, S.; Rutzinger, M.; Vosselman, G.; Oude Elberink, S. Recognising basic structures from mobile laser scanning data for road inventory studies. Photogrammetry & Remote Sensing 2011, 66, 528–539. [Google Scholar]
50. Madzarov, G.; Gjorgjevikj, D. Multi-class classification using support vector machines in decision tree architecture. IEEE EUROCON 2009, 2009, pp. 288–295. [Google Scholar] [CrossRef]
51. He, Y.; Yu, H.; Liu, X.; Yang, Z.; Sun, W.; Wang, Y.; Fu, Q.; Zou, Y.; Mian, A. Deep learning based 3D segmentation: A survey. arXiv preprint arXiv:2103.05423, 2021. [Google Scholar]
52. Guo, Y.; Wang, H.; Hu, Q.; Liu, H.; Liu, L.; Bennamoun, M. Deep learning for 3d point clouds: A survey. IEEE transactions on pattern analysis and machine intelligence 2020, 43, 4338–4364. [Google Scholar] [CrossRef] [PubMed]
53. Qu, T.; Di, S.; Feng, Y.T.; Wang, M.; Zhao, T.; Wang, M. Deep Learning Predicts Stress–Strain Relations of Granular Materials Based on Triaxial Testing Data. Computer Modeling in Engineering & Sciences 2021, 128, 129–144. [Google Scholar] [CrossRef]
54. Hinks, T.; Carr, H.; Truong-Hong, L.; Laefer, D.F. Point cloud data conversion into solid models via point-based voxelization. Journal of Surveying Engineering 2013, 139, 72–83. [Google Scholar] [CrossRef]
55. Yao, X.; Guo, J.; Hu, J.; Cao, Q. Using Deep Learning in Semantic Classification for Point Cloud Data. IEEE Access 2019, 7, 37121–37130. [Google Scholar] [CrossRef]
56. Ruizhongtai Qi, C.; Su, H.; Mo, K.; Guibas, L. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation 2016.
57. Qi, C.; Yi, L.; Su, H.; Guibas, L.J. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. NIPS, 2017.
58. Thompson, N.C.; Greenewald, K.; Lee, K.; Manso, G.F. The Computational Limits of Deep Learning. 2022; arXiv:cs.LG/2007.05558. [Google Scholar]
59. Zhang, M.; You, H.; Kadam, P.; Liu, S.; Kuo, C.C.J. PointHop: An Explainable Machine Learning Method for Point Cloud Classification. IEEE Transactions on Multimedia 2020, 22, 1744–1755. [Google Scholar] [CrossRef]
60. Shannon, C.E. A mathematical theory of communication. Bell Syst. Tech. J. 1948, 27, 379–423. [Google Scholar] [CrossRef]
61. Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; Vanderplas, J.; Passos, A.; Cournapeau, D.; Brucher, M.; Perrot, M.; Duchesnay, E. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 2011, 12, 2825–2830. [Google Scholar]
62. Hunter, J.D. Matplotlib: A 2D graphics environment. Computing in Science & Engineering 2007, 9, 90–95. [Google Scholar] [CrossRef]
63. Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; Desmaison, A.; Kopf, A.; Yang, E.; DeVito, Z.; Raison, M.; Tejani, A.; Chilamkurthy, S.; Steiner, B.; Fang, L.; Bai, J.; Chintala, S. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32; Curran Associates, Inc., 2019; pp. 8024–8035.