An Enhanced Adaptive Ensemble Kalman Filter for AUV Integrated Navigation

1. Introduction

Autonomous Underwater Vehicles (AUVs), characterized by its exceptional mobility and advanced intelligence, represent a significant development in the field of ocean technology. The integration of a sophisticated navigation system which is capable of delivering precise, reliable navigation and positioning data, is fundamental for maximizing an AUV's operational efficiency and accuracy, especially when conducting missions in complex underwater environments [1]. The commonly used underwater navigation methods for AUV mainly include dead reckoning based on inertial navigation, underwater acoustic navigation, and geophysical navigation [2]. The underwater environment, fraught with various interferences, can unpredictably impact single-source navigation data, thus compromising the positioning accuracy and motion stability of the AUV. To mitigate these problems, AUVs often utilize integrated navigation systems, incorporating an array of sensors such as electronic compasses, Doppler Velocity Logs (DVLs), GPS receiver, Ultra-Short Baseline (USBL) systems, and Inertial Navigation Systems (INSs) [3]. By applying filtering algorithms, INSs fuse and optimize navigation information from multiple sources, enhancing the overall stability and accuracy of the AUV’s positioning capability [4].

In the latter half of the 20th century, technological limitations were a significant factor in the early stage of AUV development. At that time, the overall size of the AUV as well as the high-power consumption and cost of the navigation system made it challenging for widespread use [5]. Therefore, the related research primarily focused on the optimization of the system hardware construction and cost reduction. In the Autonomous Minehunting and Mapping Technologies Program (AMMT) initiative directed by the United States Defense Advanced Research Projects Agency (DARPA), the developed AUV utilizes a Doppler-assisted navigation system that includes Doppler sonar, an inertial navigation unit (INU), and GPS [6]. This system incorporates an enhanced Kalman filter algorithm to optimize the navigation data. During testing, this system achieved a positioning accuracy of nearly 0.02% of the overall travel distance. Larsen has combined sensor units including Ring Laser Gyro (RLG), DVL, INS, Short Baseline (SBL), and GPS receiver to create the MARPOS navigation system, which has been applied to the Mardian range of commercial AUVs after extensive testing to prove its reliability and accuracy [7]. Under the impetus of the ‘863’ project, China developed the first 6000-meter-class AUV "CR-01" in 2001 [8]. The navigation system of the AUV combined INS, DVL and Long Baseline (LBL), which was able to control the positioning error in 10 meters within the coverage area of the LB positioning system. The AUV successfully completed two research missions on the investigation of manganese nodules at the bottom of the Pacific Ocean.

With advances in technology and further improvements in the hardware of navigation system, significant advances in battery technology have enabled AUVs to perform underwater tasks for extended periods of time, far beyond the capabilities of traditional manned submersibles or Remotely Operated Vehicle (ROV). However, the increased duration and scope of these missions has necessitated greater precision in navigation system [9]. Zhang et al. have provided a comprehensive review and summary of the composition and development history of AUV navigation systems since the 1980s [10]. The technologies employed in AUV navigation are currently transitioning from traditional methods requiring the preliminary deployment of positioning devices like LBL and USBL to approaches that utilize dynamic multi-agent systems. Emerging navigation techniques such as Simultaneous Localization and Mapping (SLAM) are enabling significant enhancements in both the cost-effectiveness and accuracy of AUV navigation [11].

The central issue in the field of AUV navigation is the state estimation [12]. In order to achieve improved and more consistent positioning performance, study on integrated navigation systems is increasingly focusing on enhancing nonlinear filtering algorithms within the software framework. The Kalman filter (KF) is the most commonly used filtering algorithm for processing problems related to multi-sensor data fusion in the navigation field [13]. The KF algorithm was often used in early AUV navigation systems to achieve the data fusion between different hardware components [14]. In order to better dealing with the nonlinear system filtering problem, various different nonlinear filtering algorithms have been further developed on the basis of the KF algorithm. The Extended Kalman filter (EKF) algorithm is a common algorithm derived from the Kalman filter, which is widely used in AUV navigation systems by linearizing and approximating the nonlinear term with a Taylor's first-order expansion. It’s particularly effective for systems that exhibit mild nonlinearity and non-Gaussian characteristics, hence its widespread use in the integrated navigation system of AUV [15]. Fulton et al. used the REMUS AUV to test the EKF algorithm by post-processing the data collected by acoustic navigation system, Doppler sonar and compass mounted on the AUV [16]. Their algorithm extends the EKF by integrating a feature that autonomously detects and removes outliers. The test results demonstrated that the proposed algorithm had consistently maintained stable performance, achieving positioning accuracy with an error margin within a few meters.

The EKF algorithm has the advantages of easy implementation and faster operation speed. However, the method of linearizing the nonlinear terms through the first-order Taylor expansion has a certain error, which may lead to further increasing or even dispersion of the filtering error when dealing with strongly nonlinear systems. To enhance the navigation accuracy, several studies have attempted to combine KF with other nonlinear methods. To address these limitations and enhance the performance of EKF in AUV navigation, Crassidis introduced an advanced Kalman filtering approach named the Sigma Point Kalman filter (SPKF), which specifically aims to optimize the fusion of GPS and INS data [17]. Similar to the SPKF, the Unscented Kalman filter (UKF) is a nonlinear filtering algorithm that applies the unscented transform to the KF, which has been proven to be more accurate and easier to implement than EKF in nonlinear estimation [18,19]. On this basis, Krauss et al. proposed an unscented Kalman filter algorithm on manifolds (UKF-M), which added an additional step of identifying and eliminating the outliers of the sensor measurements in the process of state updating [20]. Experiments demonstrated that the method can not only effectively attenuate the influence of outliers on AUV speed estimation but also improving the convergence ability of heading estimation. In addition, the Particle Filter (PF) serves as another prevalent nonlinear filtering technique [21]. It employs the principles of the Monte Carlo method, utilizing a collection of particles to perform stochastic sampling and weight allocation, thereby representing the system's probability density distribution [22]. This method facilitates the estimation of the system state without the need for deriving analytical solutions [23].

Similar to PF, the Ensemble Kalman filter (EnKF) is a nonlinear filtering algorithm that combines KF with Monte Carlo method. EnKF constructs a sample ensemble to take the place of the covariance matrix operation in KF, which enhances the filter's effectiveness in dealing with nonlinear systems, offering significant improvements in handling the inherent complexity of such models. It uses a finite set of states to approximate the statistical properties of the system states, implicitly estimating the error covariance matrix through the variance between the sample ensembles. This approach avoids the direct manipulation of large error covariance matrices encountered in the conventional KF algorithm on large systems, and significantly reduces the computational resource consumption when dealing with high-dimensional systems [24]. Due to its great ability to handle high-dimensional nonlinear systems, EnKF is now widely used in the field of meteorological model prediction [25].

Cui and Zhang attempted to apply the EnKF algorithm to address the issue of multi-sensor target tracking [26]. Through empirical testing, it was demonstrated that the EnKF method outperforms the conventional Extended Kalman filter (EKF) method in dealing with issues of multi-sensor target tracking, confirming its effectiveness for use in AUV navigation systems. Pornsarayouth et al. integrated the one-step smoothing method with the EnKF method to address the issue of multi-sensor target positioning [27]. Simulation results demonstrated that the EnKF method outperforms traditional methods such as EKF and PF, in handling issues associated with ‘out-of-sequence’ (OOSE) measurements in integrated navigation systems. Tin et al. have conducted extensive research on the application of EnKF in AUV underwater navigation [28]. They independently applied KF, EnKF and the Fuzzy Kalman filter (FKF) to AUV navigation data, evaluated the performance based on computation time and the root mean square error (RMSE) of positioning. The results demonstrated that EnKF outperforms FKF in both computational efficiency and three-dimensional positioning accuracy [29]. They conducted simulations of AUV motion in three-dimensional space to evaluate the positioning performance of EnKF over different ensemble member sizes. Experimental results showed that EnKF maintained consistently low average positioning errors. Furthermore, as the number of ensemble members increases, the positioning accuracy of the algorithm improves, albeit at the cost of reduced computational speed [30].

EnKF has been demonstrated to be effective in AUV integrated navigation system, but it still has notable limitations. First, the EnKF algorithm replaces traditional covariance matrix operations with statistical analyses of randomly generated sample ensembles. The outcomes of this method are significantly influenced by the number and mathematical properties of these samples, resulting in a filter output that exhibits instable variability. In comparison to conventional KF or EKF approaches, EnKF exhibits less smoothness and stability in its filtering trajectory. Secondly, although EnKF can effectively handle the nonlinear systems, its stability may degrade under common non-Gaussian noise disturbances in marine environments. To address these issues, improvements to the EnKF algorithm should start from the sample ensemble generation. This involves enhancing the algorithm's stability by modifying the sample generation process and dynamically adjusting the number of samples in the ensemble. By balancing precision and efficiency, these modifications aim to optimize overall filtering performance.

This paper proposes an enhanced adaptive EnKF algorithm for AUV integrated navigation. The highlights of the enhanced algorithm include:

(1) The algorithm employs the Laplace distribution instead of the conventional Gaussian distribution to generate members of state vector ensembles. The general behavior of the Laplace distribution resembles the Gaussian distribution but includes a higher incidence of outliers. This feature allows the algorithm to more closely mimic the real system state in natural environments, thereby improving the algorithm's robustness to non-Gaussian noise.

(2) The adaptive mechanism allows the algorithm to dynamically adjust the number of members in the state vector ensemble based on an evaluation of the smoothness of the filtered trajectory and the credibility of filtering outputs. This enables the algorithm to adaptively modify its performance according to the actual demand on smoothness and accuracy, thus enhancing the efficiency for real-time navigation.

(3) AUV navigation data obtained from field tests were used for the parameter optimization and performance validation of our proposed filtering algorithm. The advantages of the enhanced adaptive EnKF algorithm were demonstrated through comparisons with other filtering algorithms.

The rest of this paper is organized as follows. Section 2 introduces the AUV platform and the hardware configuration of its integrated navigation system in this study. Section 3 presents the model and the parameters involved in our proposed EnKF algorithm, and also describe our sample generation method derived from the Laplace distribution. In section 4, the key parameters of the algorithm are tuned based on AUV field data to optimize algorithmic performance. Section 5 introduces our adaptive mechanism into the algorithm, enabling it to adjust the number of generated sample ensembles based on the smoothness of the filtered trajectory curves and the credibility of filtering outputs. We detail of the comparison between our enhanced EnKF algorithm with standard algorithms such as the conventional EnKF and EKF in Section 6. The last section summarizes the findings in the research and the future directions.

2. AUV Platform and Sensors

This section introduces the AUV platform as well as its navigation sensors involved in this study. The vehicle is 1.8m length and has a diameter of 0.2m. The maximum working depth is 120m and the nominal forward running speed is 1m/s, with the maximum speed of 2m/s [31]. The vehicle is designed into three modules: an observation module at the bow (capable of 180° rotation for both sea bottom and ice bottom mapping), a navigation and control module in the middle and a propulsion module at the stern.

The AUV platform used in this study, as shown in Figure 1, is equipped with an array of sensors to perform various observational tasks efficiently. These sensors include: a Conductivity, Temperature, Depth (CTD) sensor for measuring the physical properties of marine water, a DVL for water current measurement, an echosounder for altimeter measurement, a side-scan sonar for topography mapping, and a multi-beam sonar for underwater imaging. These sensors collectively enable comprehensive data collection for marine research and exploration.

The AUV's navigation system, as shown in Figure 2, comprises several key components: a USBL, a GPS receiver, a DVL, an electronic compass, and a pressure sensor.

The USBL in Figure 2 is an EvoLogics S2CR 18/34 model with a measurement range of 3500m, slant range accuracy of 0.01m, and bearing resolution of 0.1°. It provides acoustic positioning data for the AUV. The Teledyne RD Instruments Pathfinder 600 kHz phased array DVL is utilized for both bottom and water mass tracking. Its primary function is to gather data on the AUV's navigation speed and measure water flow rates with a long-term accuracy of approximately ±0.2% or ±0.1 cm/s. Additionally, the DVL enables the measurement of the AUV's clearance from the seabed, with a maximum range of up to 150 meters. The system also includes a FieldForce TCM electronic compass, which delivers precise heading data. This compass boasts a resolution of 0.1 degrees and a tilt accuracy of 0.2 degrees RMS (root mean square). Lastly, the pressure sensor measures pressures up to 20 bar with an accuracy of 0.05% full scale under specified temperature conditions. Together, these components form a robust navigation system, enabling the AUV to perform precise and reliable underwater operations if a proper filtering algorithm is well-developed.

3. Setup of Our Proposed Ensemble Kalman Filter

This section will present the fundamental components and workflow of our enhanced adaptive EnKF algorithm. This algorithm fuses multi-sensor data to deliver accurate navigation information for vehicle control.

3.1. General Framework of EnKF

Kalman filter (KF) is a recursive algorithm designed for optimal state estimation in time-varying linear systems. Its operation is mainly divided into prediction and correction phases, which allows the prediction of system states and the integration and updating of these predictions with observed values, thus reducing the variance of prediction errors for future states [32]. Initially, we denote a system state vector $x_k$, which is assumed to contain a set of values describing the state of the system at time $k$. This allows the state from the previous time step to be propagated to time $k$, leading to the state transition equation for KF as below:

$$x_k=F{x}_{k-1}+B{u}_{k-1}+{\omega }_{k-1}$$

(1)

where $F$ denotes the state transition matrix, defining the relationship of system state variables as they evolve over time; $u_{k-1}$ represents the external input to the system at the previous time step; $B$ refers to the control matrix, indicating how external inputs influence the state of the system; and ω denotes the process noise, accounting for errors due to external uncertainties that cannot be entirely predicted by the system model. The system model of the algorithm often exhibits inevitable and unpredictable discrepancies compared to the actual system. These discrepancies may arise from internal uncertainties within the system or from external random disturbances. To mitigate these effects, it is essential to incorporate suitable process noise to enhance the model's robustness against environmental disturbances and to improve the precision of state estimations.

To facilitate the transition from system state variables to observable output quantities, the following observation equation is formulated as:

$$z_k=H{x}_k+{\upsilon }_k$$

(2)

where $z_k$ represents the system observation vector; $H$ denotes the observation matrix, delineating how system states influence observed values and facilitating their mapping into the observation space; $v_k$ describes the system's measurement noise, which encompasses the random errors resulting from the uncertainties encountered during the measurement process. These uncertainties primarily originate from the inherent precision limitations of sensor equipment and distortions occurring during signal transmission. Furthermore, since the KF is primarily designed for linear Gaussian systems, both the process noise ω and the measurement noise $\upsilon$ are typically assumed to consist of random variables that are independent from each other, follow a Gaussian distribution, and have a zero mean. Therefore:

$$\begin{gathered} {\omega }_k~N\left(0,Q\right) , \\ {\upsilon }_k~N\left(0,R\right) \end{gathered}$$

(3)

where $Q$ and $R$ respectively denote the covariance matrices of process noise and measurement noise.

The uncertainty associated with the state prediction from the preceding time step is propagated to the current time step via the state error covariance matrix, as described in Equation (4):

$$P_{\overline{k}}=F{P}_{k-1}{F}^T+Q$$

(4)

where $P_{k-1}$ is the state error covariance matrix corresponding to time $(k-1)$; $P_{\overline{k}}$ denotes the prior state error covariance matrix at time $k$.

As previously stated, Equation (1), (2) and (4) collectively constitute the predictive component of the computational expressions in the core iterative algorithm of KF. Then the filter integrates the data from the prior state vector estimates and observations based on their respective confidence weights. To facilitate this integration, the KF employs a parameter known as the Kalman gain, which adjusts the relative weights assigned to the estimates and observations. This computation is specified as follows:

$$K_k={\displaystyle \frac{P_{\overline{k}}{H}^T}{H{P}_{\overline{k}}{H}^T+R}}$$

(5)

where $K_k$ represents the Kalman gain at time $k$. Using the Kalman gain value $K_k$ derived from this step, the optimal state estimate vector at time $k$ can be calculated using the following Equation (6):

$$x_k=x_{\overline{k}}+K_k\left(Z_k-H{x}_{\overline{k}}\right)$$

(6)

where $x_k$ refers to the posterior estimate of the state vector at time $k$ following filtering optimization, and $x_{\overline{k}}$ denotes the prior predictive state vector value at time $k$ described in Equation (1). After the state vector is corrected through filtering, the error covariance matrix needs to be updated in accordance with the procedure outlined below:

$$P_k=\left(I-K_{k}H\right)P_{\overline{k}}$$

(7)

where $I$ is the identity matrix, which has the same dimensions as the state vector $x_k$; $P_{\overline{k}}$ represents the prior error covariance matrix at time $k$, which has been derived in Equation (4).

Equation (1), (2), (4) to (7) collectively constitute the core iterative equations utilized within the KF computation at a given time step. The posterior estimates $x_k$ and the error covariance matrices $P_k$ are then conveyed to the subsequent time step to continue the filtering process as described.

EnKF integrates concepts from Monte Carlo methods into the conventional KF algorithm, adopting an ensemble approach. This algorithm involves the use of ensembles of randomly generated state vector samples to approximate and recover the statistical properties of the actual system state, thus replacing the traditional error covariance matrix in the iterative computations. Starting from the initial state vector $x_k$, random perturbations are applied to generate a corresponding ensemble of state vector samples, which are structured as follows:

$$X_k=\left[x_1^k,x_2^k,x_3^k,\cdots ,x_n^k\right]$$

(8)

where $X_k$ corresponds to the ensemble of random samples representing the system state vector at time $k$; $n$ indicating the number of members in this ensemble. Similarly, based on the initial observation vector $z_k$, the observation vector sample ensemble $Z_k$ is defined as follows:

$$Z_k=\left[z_1^k,z_2^k,z_3^k,\cdots ,z_n^k\right]$$

(9)

As a substitute for estimating and updating the error covariance matrix, compute the mean of the generated state vector sample ensemble ${\overline{X}}_k$:

$${\overline{X}}_k={\displaystyle \frac{\sum _{i=1}^n{x}_i^k}{n}}$$

(10)

Furthermore, we can utilize the ensemble mean to execute the calculation outlined below, which serve to replace the iterative and corrective steps involving the error covariance matrix, as traditionally performed in KF Equations (4) and (7):

$$P_H={\displaystyle \frac{\sum _{i=1}^n(x_i^k-{\overline{X}}_k\left)\right(H{x}_i^k-H{\overline{X}}_k)}{n-1}}$$

(11)

where $P_H$ represents an intermediate variable that substitutes for the traditional error covariance matrix, thereby facilitating the implicit propagation of error covariance. Once $P_H$ has been computed, it is incorporated into Equation (5) to directly calculate the Kalman gain:

$$K_k={\displaystyle \frac{P_H}{H{P}_H+R}}$$

(12)

Additionally, modifications based on Equation (6) yield the filtering optimization correction formula for the state vector under the EnKF algorithm:

$$x_k={\displaystyle \frac{\sum _{i=1}^n{[x}_i^k+K_k\left(z_i^k-H{x}_i^k\right)]}{n}}$$

(13)

From Equation (13), one can derive the EnKF algorithm's filtering estimation for the current time step. The operations detailed in Equations (8) to (13) outline the principal computational steps of the EnKF algorithm. As the algorithm progresses to the subsequent time step, the process is initiated anew from Equation (8) with the reconstruction of the state vector's random sample ensemble, followed by the subsequent calculations.

3.2. AUV State Model and Settings

In this study, the EnKF algorithm is utilized to address AUV qualitative navigation issues within a given area. To simplify the problem, we consider AUV navigation in the horizontal plane, so that AUV speed measured by DVL, attitude measurements from electronic compass, and position data from GPS are taken into account in the filtering algorithm. In this section, we will focus on the development of AUV system equations and variables that are specifically tailored to the EnKF algorithm.

Generally, AUV’s position $X={[x,y,z]}^T$ is described in the inertial coordinate frame, represented by an orthonormal triad $\{x,y,z\}$, where $z$ is aligned with the local direction of gravity. However, AUV’s velocity $v={[u,v,w]}^T$ is measured in the body-fixed coordinate frame, defined by an orthonormal triad $\{x_b,y_b,z_b\}$, where $\{x_b,y_b,z_b\}$ is aligned with the body’s longitudinal axis [33]. The origin of the body frame sits at the vehicle’s center of buoyancy. A proper rotation matrix $\mathcal{R}$, parameterized by the Euler angles, is used to map free vectors from the body-fixed frame to the inertial frame. The Euler angles include the roll angle $\varphi$, the pitch angle $\theta$ and the yaw angle $\psi$, which are depicted in Figure 3.

The rotation matrix $\mathcal{R}$ is defined as this [34]:

$$\begin{gathered} \mathcal{R}=\left[\begin{array}{cc}\mathit{cos}\left(\psi \right)\cos \left(\theta \right)& -\mathit{sin}\left(\psi \right)\mathit{cos}\left(\varphi \right)+\mathit{cos}\left(\psi \right)\mathit{sin}\left(\theta \right)\sin \left(\varphi \right)\\ \mathit{sin}\left(\psi \right)\cos \left(\theta \right)& \mathit{cos}\left(\psi \right)\mathit{cos}\left(\varphi \right)+\mathit{sin}\left(\psi \right)\mathit{sin}\left(\theta \right)\sin \left(\varphi \right)\\ -\sin \left(\theta \right)& \mathit{cos}\left(\theta \right)\sin \left(\varphi \right)\end{array}\right. \cdots \\ \left. \begin{array}{c}\mathit{sin}\left(\psi \right)\mathit{sin}\left(\varphi \right)+\mathit{cos}\left(\psi \right)\mathit{sin}\left(\theta \right)cos\left(\varphi \right)\\ -\mathit{cos}\left(\psi \right)\mathit{sin}\left(\varphi \right)+\mathit{sin}\left(\psi \right)\mathit{sin}\left(\theta \right)cos\left(\varphi \right)\\ \mathit{cos}\left(\theta \right)cos\left(\varphi \right)\end{array}\right] \end{gathered}$$

(14)

Consequently, by multiplying the rotation matrix $\mathcal{R}$ with the velocity vector $v$, one obtains the AUV's velocity vector expressed in the inertial frame:

$$V={[v_x,v_y,v_z]}^T=\mathcal{R}\cdot v$$

(15)

This process allows the establishment of the state vector of AUV dynamic system in the inertial frame, which includes AUV kinematic parameters:

$$x_{state}={\left[x,y,z,v_x,v_y,v_z\right]}^T$$

(16)

Assuming that the sensors sample the AUV's motion data at intervals of $dt$, one can derive the kinematic equation shown in Equation (17). This equation employs the AUV's motion states at the previous moment in conjunction with the data sampled from the sensors to calculate the motion states at the current moment.

$$x_{state,k}=\left[\begin{array}{c}\begin{array}{c}{x}_k\\ y_k\end{array}\\ \begin{array}{c}{z}_k\\ v_{x,k}\end{array}\\ \begin{array}{c}{v}_{y,k}\\ v_{z,k}\end{array}\end{array}\right]=\left[\begin{array}{c}\begin{array}{c}{x}_{k-1}+v_{x,k}dt\\ y_{k-1}+v_{y,k}dt\end{array}\\ \begin{array}{c}{z}_{k-1}+v_{z,k}dt\\ v_{x,k}\\ v_{y,k}\end{array}\\ v_{z,k}\end{array}\right]=\left[\begin{array}{cc}\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}& \begin{array}{ccc}dt& 0& 0\\ 0& dt& 0\\ 0& 0& dt\end{array}\\ \begin{array}{ccc}0& 0& 0\\ 0& 0& 0\\ 0& 0& 0\end{array}& \begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\end{array}\right]\left[\begin{array}{c}\begin{array}{c}{x}_{k-1}\\ y_{k-1}\\ z_{k-1}\end{array}\\ v_{x,k}\\ \begin{array}{c}{v}_{y,k}\\ v_{z,k}\end{array}\end{array}\right]=F{x}_{k-1}$$

(17)

where $F$ indicates the state transition matrix of the system. Then, the AUV's positional and velocity data are utilized to compile the observation vector z. Since the AUV’s positioning information $X=[x,y,z]$ comes from the observed data, following Equation (2), the observation matrix $H$ is defined as follows:

$$H=\left[\begin{array}{cc}\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}& \begin{array}{ccc}0& 0& 0\\ 0& 0& 0\\ 0& 0& 0\end{array}\end{array}\right]$$

(18)

Thus far, we have derived describe the AUV motion system by the Equation (14) to (18).

3.3. Generation of State Vector Ensemble

EnKF necessitates the creation of a sample ensemble by adding random perturbations to the state vector to simulate model errors. The real error covariance is approximated through the distribution of members in the ensemble [35]. Therefore, generating a random ensemble with an appropriate distribution rule is crucial for the filtering performance of the EnKF algorithm.

The Gaussian distribution is characterized by its broad and general statistical properties. Filtering algorithms such as the EKF and EnKF typically assume that both system and observation errors adhere to a Gaussian distribution during the linear approximation of nonlinear systems. The probability density function for the Gaussian distribution is as follows:

$$f\left(x|\mu ,{\sigma }^2\right)={\displaystyle \frac{1}{\sqrt{2\pi {\sigma }^2}}}\mathrm{e}\mathrm{x}\mathrm{p}\left[-{\displaystyle \frac{{\left(x-\mu \right)}^2}{2{\sigma }^2}}\right]$$

(19)

where $x$ denotes a random variable; $\mu$ represents the mean value of the distribution; and $\sigma$ refers to the standard deviation of the distribution.

EnKF generates ensembles by adding zero-mean Gaussian-distributed perturbations to the state vector, simulating uncertainties inherent in real systems [36]. However, in actual AUV navigation tasks, dynamic ocean processes, human activities and sensor errors may introduce significant non-Gaussian noise disturbances, which characterize by large impulses, short durations, and pronounced tails. This makes Gaussian-based sample generation inadequate for accurate state approximation. To improve robustness against non-Gaussian noise, alternative sample generation methods are essential.

The Laplace distribution, also known as the double exponential distribution, derives its name from its graphical resemblance to the combined representation of two exponential distribution probability density functions. It is defined by a location parameter and a scale parameter, with its probability density function given by Equation (20):

$$f\left(x|\mu ,b\right)={\displaystyle \frac{1}{2b}}exp\left(-{\displaystyle \frac{\left|x-\mu \right|}{b}}\right)$$

(20)

where $\mu$ denotes the location parameter, indicating the mean value of the distribution, and $b$ represents the scale parameter, which determines the distribution’s spread, and this value is estimated by $b={\displaystyle \frac{1}{n}}\sum |x^i-\mu |$ in this study. The introduction of additional parameters on this basis enables the development of various Laplace distribution variants, making it versatile enough to accommodate different research questions and data characteristics [37].

Figure 4 presents a comparative graph of the function curves for the standard Gaussian distribution ($\mu =0,\sigma =1$) and the standard Laplace distribution ($\mu =0,b=1$). Both distributions exhibit a comparable functional form and overall distribution trend. However, the Laplace distribution's curve, as illustrated in Figure 4, features a more pronounced peak and thicker tails compared to the Gaussian distribution. This indicates that, while maintaining properties similar to the Gaussian distribution, the Laplace distribution is more prone to include extreme values or outliers, providing a more accurate reflection of the statistical characteristics for systems with significant non-Gaussian noise.

The Laplace distribution is extensively utilized in various fields, including finance and industry. It is often employed as an alternative to the Gaussian distribution in robustness research [37]. This distribution is characterized by the maximum entropy property, meaning it exhibits the highest level of entropy among all probability distributions with a given mean and variance. This results in greater uncertainty and informational content under comparable conditions, making the Laplace distribution particularly valuable for both practical applications and scholarly research [38].

To further verify the effectiveness of the Laplace distribution in modelling the positioning deviations of AUV in underwater conditions, the deviation between the actual navigation trajectory and the predefined trajectory at each sampling point during the AUV field test was calculated, and Figure 5 depicts the probability density distribution curve of the aforementioned deviation, along with the Gaussian and Laplace distribution curves, which exhibit the same mean and standard deviation as the deviation data. As illustrated in the figure, the probability distribution of the AUV positioning trajectory exhibits pronounced peaks and relatively thick tails, closely approximating the probability density curve of the Laplace distribution.

Moreover, we employ the Akaike Information Criterion (AIC) to quantify and compare the fit of the trajectory error distribution with two specific distribution functions. AIC derives from the concept of entropy, which is a widely utilized metric for assessing the goodness of fit between models or distributions and observed data [39]. For the same dataset, a lower AIC value indicates a superior fit of the distribution to the data.

The definition of AIC is given by Equation (21):

$$AIC=2k-2\ln \left(L\right)$$

(21)

where $k$ represents the number of parameters in the formula of the distribution function, and $k$ is equal to 2 for both the Gaussian and Laplace distributions; $L$ denotes the log-likelihood function corresponding to the probability density function of the two distributions, as shown for the Gaussian distribution in Equation (22) and for the Laplace distribution in Equation (23):

$$\begin{gathered} \ln \left(L_{gauss}\right)=\sum _{i=1}^n\ln \left[{\displaystyle \frac{1}{\sqrt{2\pi {\sigma }^2}}}\exp \left(-{\displaystyle \frac{{\left(x_i-\mu \right)}^2}{2{\sigma }^2}}\right)\right] \\ =-{\displaystyle \frac{n}{2}}\ln \left(2\pi {\sigma }^2\right)-{\displaystyle \frac{1}{2{\sigma }^2}}\sum _{i=1}^n{\left(x_i-\mu \right)}^2 \end{gathered}$$

(22)

$$\begin{gathered} \ln \left(L_{laplace}\right)=\sum _{i=1}^n\ln \left[{\displaystyle \frac{1}{2b}}\exp \left(-{\displaystyle \frac{|x_i-\mu |}{b}}\right)\right] \\ =-n\ln \left(2b\right)-{\displaystyle \frac{1}{b}}\sum _{i=1}^n|x_i-\mu | \end{gathered}$$

(23)

Using the Equation (21) to (23), the AIC value for the Gaussian distribution applied to the positioning deviation data is calculated as 192688.647, while the value for the Laplace distribution is 136045.836. These values indicate that, according to the AIC criterion, the Laplace distribution provides a superior representation of the deviation distribution in the AUV's navigation trajectory.

Consequently, employing data sets that adhere to the Laplace distribution enhances the EnKF algorithm's ability to adapt to non-Gaussian noise. This adaptation allows for more accurate approximations of systems within actual marine environments and increases the stability of the algorithm's positioning capabilities.

4. Parameters Optimization in EnKF

In the EnKF algorithm framework outlined in Section 3, the performance of the algorithm is primarily influenced by parameters such as system process noise and system measurement noise. This section will focus on tuning the parameters of the EnKF algorithm based on data from field trials. The tuning process will utilize the method of controlled variables, systematically adjusting and analyzing the impact of different parameter changes on the filtering performance of the algorithm. This approach will guide the optimization of parameters, ultimately aiming to enhance the precision of the filtering process.

4.1. Data Sources

The AUV navigation data used in this study were collected from field trials conducted in the coastal waters surrounding Jiangmen City, Guangdong Province, China, between May 31 and June 2, 2022. The test area was located at 21°50′N, 112°55′E, as illustrated in Figure 6.

The example trajectories of the AUV's dead reckoning and the GPS positioning, derived from the test data, are depicted in Figure 7. This figure includes the data spanning approximately 4000 time steps, with each step sampled at intervals of about 0.2 seconds. It is evident that the discrepancy between the dead reckoning and GPS positioning trajectories shown in Figure 7 increases over time, leading to a significant deviation at the end of the motion.

In this section, we aim to refine the EnKF algorithm to process sensor data, including speed, attitude, and GPS fix data collected from the AUV field trials. By integrating dead reckoning data with GPS information, the study seeks to mitigate the cumulative positioning errors inherent in dead reckoning over time and correct for the GPS positional drift and fluctuations caused by environmental disturbances during experiments [40]. In order to visually illustrate the optimization effect of the algorithm on the GPS information subsequent to the integration of dead reckoning data, GPS positioning data will be employed as a baseline for comparison and evaluation of the filtering performance of the algorithm under different parameter settings in this section.

4.2. Impact of Key Parameters on EnKF Filtering Performance

In the algorithm utilized in this study, the measurement noise and process noise, represented by the ensemble covariance, cannot be adjusted iteratively by the filter and must be pre-set as key parameters. They describe the accuracy and reliability of system state estimates and directly influence the manner in which the filter strikes a balance between trusting system predictions and measurement data. This balance has a significant impact on the accuracy and stability of the state estimation. Therefore, this subsection will explore how different values of measurement noise and process noise affect the algorithm's filtering performance through a series of tests, providing guidance for optimizing filter performance.

4.2.1. Process Noise

Process noise is a measure of the inherent uncertainty of a system model in relation to actual conditions. Intrinsically, in the EnKF algorithm, the model's uncertainty will be captured and approximated by adding perturbations to each member of a vector ensemble according to a specified distribution and analyzing the statistical characteristics of the ensemble. Consequently, explicit propagation of process noise through Equation (5) is unnecessary in EnKF. Thus, adjusting the process noise parameters in EnKF fundamentally involves tuning the variance of the random perturbations introduced into the generated state vector ensemble.

To investigate the impact of variance changes in the sample vector ensembles on the filtering effectiveness of the EnKF algorithm and to provide guidance for algorithm parameter tuning, we generate sample ensembles with a quantity of 100, a variance of 1, and no measurement noise. All the other parameter adjustment conditions will be made in relation to this baseline.

(1) Variance of the observation vector ensemble ${{\sigma }_{po}}^2$In scenario where no measurement noise is present and a state vector ensemble of size 100 is generated using a Laplace distribution with a scale parameter of 1, we examine the influence of varying variances in the observation vector ensembles on the EnKF's filtering performance.

Figure 8 shows a partial enlarged view of EnKF's filtering trajectories and GPS positioning curves during AUV turning motion under different ${{\sigma }_{po}}^2$ values. When ${{\sigma }_{po}}^2=0.1$ as green curve in Figure 8(a), the EnKF filtered trajectory nearly aligns with the GPS results, maintaining smoothness. At ${{\sigma }_{po}}^2=1$ as blue curve in Figure 8(b), the trajectory still aligns closely with GPS, but noticeable fluctuations appear between 330m and 340m on the $x$-axis, reducing smoothness. At ${{\sigma }_{po}}^2=1$0 as red curve in Figure 8(c), the filtered result fails to converge stably, showing significant fluctuations.

The results of the parameter adjustment shown in Figure 8 indicate that an increase in the variance of the observation vector ensemble correlates with a high degree of filtered trajectory fluctuation. Specifically, in the baseline parameter setting condition, setting the variance parameter to 10 leads to a significant instability in system filtering. This highlights the necessity of calibrating the variance parameters in the observation vector ensemble relative to the scale of the actual observational data. Such calibration ensures that the ensemble can accurately reflect the potential error distribution of the system without compromising stability due to overly high variance values.

(2) Variance of the state vector ensemble ${{\sigma }_{ps}}^2$The adjustment test for the variance of the state vector set is analogous to that of the observation vector ensemble described above. The results of the test indicate that, with the current parameter settings, modifying the variance of the state vector ensemble does not significantly affect the filtering performance of the EnKF. Even when the value of ${{\sigma }_{ps}}^2$ is increased from 0.01 to 100, the algorithm's filtering performance remains largely unchanged, with the filtered trajectory closely aligned with the GPS positioning trajectory. This outcome underscores that, in the current setup, the algorithm disproportionately relies on GPS positioning data, largely overlooking the positional estimates derived from dead reckoning. Therefore, future adjustments are needed to account for different settings of measurement noise in order to refine the algorithm's performance.

4.2.2. Process Noise

Measurement noise represents the inherent uncertainty in the gathering of sensor data and significantly impacts filter performance, which is similar to the process noise [41]. To better approximate measurement errors, Kalman filter measurement noise should be tailored to sensor capabilities. In the absence of precise references, optimal measurement noise can be determined experimentally. In this study, the AUV’s position and orientation in three-dimensional space are derived from sensor observations, assuming independent measurement noise in the $\mathit{x}$, $\mathit{y}$, and $\mathit{z}$ axes with equal variance. This assumption leads to the fundamental form of the measurement noise covariance matrix in Equation (24):

$$R=\left[\begin{array}{ccc}{\sigma }_{obs}^2& 0& 0\\ 0& {\sigma }_{obs}^2& 0\\ 0& 0& {\sigma }_{obs}^2\end{array}\right]$$

(24)

where ${\sigma }_{obs}^2$ denotes the variance of the sensor measurement noise and serves as the scalar factor in the covariance matrix. Subsequently, the model incorporates additional random perturbations ${\upsilon }_{ij}(i,j=\mathrm{1,2},3)$ with a mean of zero, derived from a Laplace distribution, which are superimposed upon the foundational measurement noise covariance matrix. In this matrix, the variance of ${\upsilon }_{ij}$ is denoted by ${\sigma }_v^2$. This setup leads to the result as shown in Equation (25):

$$R=\left[\begin{array}{ccc}{\sigma }_{obs}^2+{\upsilon }_{11}& {\upsilon }_{12}& {\upsilon }_{13}\\ {\upsilon }_{21}& {\sigma }_{obs}^2+{\upsilon }_{22}& {\upsilon }_{23}\\ {\upsilon }_{31}& {\upsilon }_{32}& {\sigma }_{obs}^2+{\upsilon }_{33}\end{array}\right]$$

(25)

Subsequently, experiments will be conducted to separately investigate how variations in ${\sigma }_{obs}^2$ and ${\sigma }_v^2$ within the matrix affect the algorithm's filtering performance, with the objective of confirming the value selection for the matrix $R$ in the algorithm.

(1) Variance of the sensor measurement noise ${\sigma }_{obs}^2$, diagonal elements in $R$

To investigate the impact of diagonal elements in matrix $R$ on the filtering algorithm, the ensemble variance is set to 1 for all 100 members, with ${\upsilon }_{ij}$ values at zero and ${\sigma }_{obs}^2$ values at 0.1, 1, and 10, respectively. The partial enlarged view of the EnKF filtering trajectory curves and the GPS positioning curves with different ${\sigma }_{obs}^2$ values is presented in Figure 9. This figure provides a detailed illustration of the impact of the ${\sigma }_{obs}^2$ value on the EnKF filtering results compared with the GPS positioning trajectory. The red curve in Figure 9(c), which represents the filtering result with ${\sigma }_{obs}^2=10$, exhibits a notable divergence from GPS data, particularly within the range of 790$m$ and 800$m$ on the $y$-axis, which smooths the positioning result when AUV is turning. In contrast, no similar deviation is observed with smaller ${\sigma }_{obs}^2$ values.

In the context of present conditions, an increase in the measurement noise variance ${\sigma }_{obs}^2$ results in a discernible rise in the fluctuation of the filtering curve, particularly in segments involving significant changes in direction, where deviations from the GPS trajectory become more pronounced. This observation underscores how, with elevated measurement noise, the influence of dead reckoning data on the filtering outcome is amplified, although at the expense of reduced smoothness in parts of the filtering curve. The optimal setting of ${\sigma }_{obs}^2$ is of paramount importance, as values that are either too high or too low will have a detrimental effect on the filter's performance.

(2) Variance of the model incorporates additional random perturbations ${\sigma }_v^2$

Assume an ensemble variance of 1 and an ensemble members number of 100, with ${\sigma }_{obs}^2$ set at 1, and varying ${\sigma }_v^2$ for ${\upsilon }_{ij}$ at 0.01, 0.1, 0.3, 0.5, the resulting trajectory plots are depicted in Figure 10. As illustrated in the figure, when the value of ${\sigma }_v^2$ is less than 0.5, the filtering results exhibit a high degree of alignment with the GPS positioning data. When ${\sigma }_v^2$ increases to 0.5, as illustrated by the red curve, the filtering trajectory starts to exhibit significant irregular deviations.

In short, based on the experiment results, we conclude that as long as the variance of the random perturbations ${\upsilon }_{ij}$ remains within a predefined acceptable range, its impact on the algorithm's filtering performance is negligible. However, if the variance is excessively high, it can severely compromise the stability of the filtering results, leading to the generation of anomalies characterized by substantial errors at specific instances.

4.3. Optimization and Comparative Analysis of Parameter Schemes

From the parameter adjustment tests discussed above, we have gained the basic understanding of how individual parameters influence the EnKF filtering performance. However, this approach has predominantly focused on single-parameter adjustments, thus overlooking the potential synergistic or antagonistic effects that could arise from the interactions between multiple parameters. Consequently, it is necessary to formulate and test combinations of the above discussed parameters that are theoretically optimal, with the aim of refining our strategy based on the insights already acquired.

Based on the discussion in the previous subsection, a preferred parameter configuration has been established: assigning a variance of 10 to the observation vector ensemble that represents the process noise, and setting the measurement noise within a covariance matrix characterized by a mean influence of 1 and a variance of 0.1.

Figure 11 illustrates a comparison of trajectory curves with different state vector variance settings under the preferred parameter configuration mentioned earlier. As shown in Figure 11(a), varying parameter configurations lead to different outcomes in terms of filtering smoothness and divergence from the GPS positioning data, particularly noticeable in the area highlighted by the black box. The enlarged view in Figure 11(b) shows that as ${{\sigma }_{ps}}^2$ decreases from 0.5 to 0.01, the filtered positioning shifts from close alignment with the GPS trajectory to a deviation toward dead reckoning. Additionally, as ${{\sigma }_{ps}}^2$ decreases, the filtering curve demonstrates an incremental enhancement in its degree of smoothness. In summary, as the variance of the state vector ensemble increases, the filtered trajectory curve becomes increasingly aligned with the GPS trajectory, resulting in a compromise of both smoothness and filtering effectiveness. When ${{\sigma }_{ps}}^2$ reaches a certain threshold, the filtered trajectory tends to completely overlap with the GPS trajectory, and the degree of curve fluctuations increases correspondingly. Conversely, lower values of ${{\sigma }_{ps}}^2$ result in filtering outcomes that more closely resemble dead reckoning estimates, exhibiting a relative increase in smoothness. Consequently, within the parameters considered in this study, adjusting ${{\sigma }_{ps}}^2$ between 0.01 and 0.1 allows for relatively optimal balance between GPS positioning date and dead reckoning estimates while maintaining overall trajectory smoothness.

Based on these findings, this research employs the EnKF algorithm with a state vector ensemble variance of 0.03 for continued investigation.

5. Adaptive Mechanism in EnKF

The previous section investigated the impacts of various algorithm parameters on its filtering performance and achieved parameter optimization by selecting the optimal configurations. In addition to the parameters discussed in Section 4, the number of random samples included in the state vector ensemble generated by the algorithm also influences filtering stability and operational efficacy. The size of this ensemble dictates the sampling density within the system’s state space. Increasing the number of vector samples with random perturbations enhances the model's ability to approximate the system’s uncertainties and nonlinear features. However, this also increases computational and storage demands, thereby increasing the runtime. Conversely, a smaller ensemble of state vector samples may not adequately represent the system's complexity and uncertainty, potentially compromising the algorithm's filtering stability. Building on the preceding discussion, this subsection introduces an adaptive mechanism to the EnKF algorithm.

5.1. Key Criteria of Adaptive Mechanism

Our proposed mechanism enables the algorithm to assess the smoothness of the filtered results and the credibility of filtering outputs based on defined criteria, then autonomously adjust the number of ensemble members in the filtering process. This allows for a dynamic equilibrium between filtering stability and real-time performance, tailored to the needs of the navigation system.

5.1.1. Smoothness of Filtering Results

The smoothness of the filtering results reflects the extent of noise reduction and the accuracy of trajectory estimation. Smoother filtering outcomes indicate that the algorithm effectively mitigates noise and accurately captures the true motion pattern of the AUV, resulting in more reliable and stable trajectory predictions.

To evaluate the smoothness of the filtering results, we define a smoothness angle, which is the angle between two lines connecting adjacent trajectory points of the AUV after the filtering process. By computing the average angle of all trajectory points within a specified time interval, we can assess the smoothness of the filtered trajectory. The angle between every three consecutive trajectory points can be determined using the dot product of the corresponding vectors:

$$\overrightarrow{AB}·\overrightarrow{BC}=\left|\overrightarrow{AB}\right|\left|\overrightarrow{BC}\right|\mathit{cos}\left(\theta \right)$$

(26)

where $A,B,C$ represent three consecutive trajectory points, $\overrightarrow{AB}$ and $\overrightarrow{BC}$ are the vectors defined by these points, and $\theta$ is the angle between the two vectors. Assume that AUV trajectory point estimated by the EnKF algorithm at time $k$ are $(x_k,y_k)$, and the trajectory points at times $(k-1)$ and $k-2)$ are $(x_{k-1},y_{k-1})$ and $(x_{k-2},y_{k-2})$, respectively. Consequently, the following expression can be derived from Equation (26):

$${\theta }_{k-1}=\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{c}\mathrm{o}\mathrm{s}{\displaystyle \frac{\left(x_{k-2}-x_{k-1}\right)\left(x_k-x_{k-1}\right)+(y_k-y_{k-1})(y_{k-2}-y_{k-1})}{\sqrt{\left[{(x_{k-2}-x_{k-1})}^2+{(y_{k-2}-y_{k-1})}^2\right]\left[{(x_k-x_{k-1})}^2+{(y_k-y_{k-1})}^2\right]}}}$$

(27)

where ${\theta }_{k-1}$ denotes the angle with respect to the trajectory point at time step $(k-1)$ after filtering.

With Equation (27), we can record the smoothness angle at each trajectory point prior to time $k$. This angle serves as an indicator of the local smoothness of the filtering trajectory at that point. The angle ranges from 0° to 180°, with values closer to 180° indicating better smoothness of the trajectory. By calculating the average smoothness angle of all trajectory points within a specified time interval, we can evaluate the smoothness of the filtering results.

In our study, we find that the number of state vector ensemble members significantly impacts the smoothness of the filtering results. Generally, a larger number of ensemble members leads to a smoother filtered trajectory, but this also increases the filtering runtime, potentially compromising the real-time performance of the filter for online operation. With the given dataset, the variation in runtime and the average smoothness angle of the filter over a specified time period with the number of state vector ensemble members is depicted in Figure 12. The dataset comprises 30,000 sampling points, each separated by approximately 0.2 s, resulting in a total duration of approximately 600 s. As illustrated in Figure 12, an increase in the number of state vector ensemble members results in a smoother filtered trajectory. However, this also leads to a significant increase in the runtime of the filtering algorithm.

Based on the method for assessing the smoothness of filtering results mentioned above, an adaptive adjustment approach is integrated into the algorithm to dynamically alter the number of state vector members in the ensemble based on predefined thresholds. During the filtering process, a sliding window is used to average a specified number of smoothness angles along the filtered trajectory. This average serves as the quantitative measure of the trajectory’s smoothness. If it exceeds an upper threshold, indicating a sufficiently smooth trajectory, the algorithm will decrease the size of the generated state vector ensemble, thereby expediting the processing speed. Conversely, if the average smoothness angle falls below a lower threshold, suggesting substantial fluctuations in the trajectory, the number of ensemble members will be increased to enhance filtering accuracy.

5.1.2. Credibility of Filtering Outputs

When the AUV persists in moving along relatively straight paths, the algorithm, influenced by the smoother trajectory points, may consistently generates fewer ensemble members. This could impede the algorithm's capacity to implement prompt and effective corrections to minor deviations that occur during movement, thereby gradually leading to the accumulation of filtering errors.

To address this issue, we further introduce a refined approach for adjusting the number of ensemble members based on the credibility of the filtering outputs. This approach establishes a threshold and calculates the Euclidean distance between the dead reckoning positioning data and GPS data, as specified in Equation (28):

$$d=\sqrt{{(x_{DR}-x_{GPS})}^2+{(y_{DR}-y_{GPS})}^2}$$

(28)

where $(x_{DR},y_{DR})$ and $(x_{GPS},y_{GPS})$ denote the positional coordinates derived from dead reckoning and GPS measurements, respectively; $d$ represents the Euclidean distance deviation between them. If $d$ is found to be significantly large and persists above the threshold $d_{threshold}$ for a specified duration, it indicates a substantial deviation between the dead reckoning positioning data and the GPS data. In such cases, it is necessary to increase the number of ensemble members in order to maintain algorithmic precision. Conversely, if $d$ decrease and remain below the threshold for a certain period, the method reduces the number of ensemble members. The determination of the threshold $d_{threshold}$, the increment in member numbers once this threshold is surpassed, as well as the upper limit of the increase, should be tailored to the specific requirements of the navigation task and the scale of the mission area.

5.2. Validation of Adaptive Mechanism

To validate the filtering performance of the EnKF algorithm enhanced with the adaptive mechanism described above, a comparative analysis is conducted between the EnKF algorithm before and after implementing this adaptive mechanism. The comparative analysis involved the following four scenarios of the EnKF algorithm, each applied to filter and assess navigation data:

A. EnKF with a constant ensemble size of 250;

B. EnKF adaptively adjusting the ensemble size based on both the smoothness of the filtered trajectory and the credibility of the filtering outputs;

C. EnKF adaptively adjusting the ensemble size solely based on the smoothness of the filtered trajectory;

D. EnKF adaptively adjusting the ensemble size solely based on the credibility of the filtering results.

In all four scenarios, the initial ensemble size is 250, which means there are 250 state vectors in the ensemble. The ensemble size adjustment based on the smoothness of the filtered trajectory has a range of ±150, with increments of 30, while the ensemble size adjustment based on the credibility of the filtering results ranges from 0 to 100, with increments of 10. Aside from the variable ensemble size, the other parameters within the algorithm remain unchanged across the scenarios.

To compare the filtering performance of the four scenarios mentioned above, a series of repeated trials utilizing AUV measurements, which encompass complex motion states, are conducted. For each filtering scenario, the AUV trajectories after filtering are presented in Figure 13. The filtering trajectories generated by the algorithm under the four scenarios demonstrate a considerable degree of overlap, making it difficult to accurately assess the precision and smoothness of the algorithm across these scenarios based solely on the curves presented in Figure 13. This consistency is attributed to the parameter optimization described in Section 4, which has rendered the algorithm's filtering effect on the navigation data both stable and reliable, leads to only minor differences in the positioning results across the scenarios.

To compare the algorithm's smoothing performance under different scenarios, Figure 14 depicts the average smoothness angle of filtered trajectories and changes in ensemble member numbers in four scenarios. The AUV trajectory is divided into three segments I, II and III based on its motion state. Segment I features complex movement with frequent attitude changes and trajectory fluctuations, leading to varying filtering smoothness, as shown in Figure 14(a). The red and blue curves in Figure 14(b), representing EnKF algorithm scenarios that take the smoothness of the trajectory into consideration, resulting in smoother, more stable smoothness distributions. Unlike the green and yellow curves, which drop to a local minimum smoothness angle of about 138°, the red and blue curves increase by 10°, mitigating smoothness reduction.

In Segment II, AUV demonstrates a relatively steady movement, exhibiting minimal fluctuations in trajectory smoothness. Consequently, the red and blue curves in Figure 14(b) show a decrease in ensemble sizes, staying below the initial 250. This reduction slightly lowers the smoothness, with the red and blue curves in Figure 14(a) showing an average smoothness angle smaller within 3° than the green and yellow curves. This indicates that moderate reducing ensemble members during stable AUV motion has little impact on filtering smoothness.

The AUV's movement in Segment III becomes complex again, causing frequent fluctuations in filtering smoothness, as shown in Figure 14(a). The red and blue curves in Figure 14(b) respond by frequently adjusting the ensemble sizes. The red and yellow curves, employing an adaptive mechanism based on the filtering output credibility, increase ensemble members due to growing discrepancies between dead reckoning and GPS positioning. The red curve, employing both adaptive mechanisms, effectively limits low smoothness levels, showing relatively optimal performance in Figure 14(a) with a local minimum smoothness improvement of up to 5° compared to the other scenarios.

Furthermore, the algorithm's operational efficiency in the four scenarios was analyzed, with average runtime data presented in Table 1. The scenario that adjusts ensemble size based on filtering smoothness, particularly during smooth AUV movement, operates significantly more efficiently than those without this adaptive mechanism. In contrast, the scenario that adapts based on the credibility of filtering outputs, increasing ensemble sizes to counter navigation errors, shows a slight increase in runtime. The red curve scenario, which adapts based on both smoothness and credibility, accelerates the filtering process, reducing iteration time by about 10% compared to the conventional EnKF algorithm. Despite this efficiency boost as well as the improvement of filtering smoothness shown in Figure 14, there is no noticeable increase in positioning error. In practical underwater operations, as the duration of the task increases, the improvements in filtering accuracy, smoothness, and speed are expected to have an even greater impact on enhancing the AUV's positioning performance.

In conclusion, the EnKF with an adaptive mechanism for adjusting ensemble sizes based on filtering smoothness and output credibility demonstrates superior efficiency and smoothness, confirming that the proposed adaptive EnKF is better suited for real-world observational requirements and outperforms the conventional EnKF in real-time navigation.

6.1. Comparison with Conventional EnKF

The adaptive EnKF algorithm optimized in Section 5 and the conventional EnKF algorithm are each used to filter and optimize the aforementioned AUV navigation data. The resulting filtered trajectories are shown in Figure 16. It shows that the filtering trends of the two EnKF algorithms are similar, both achieving greater smoothness compared to the GPS positioning data. But notably, the filtering results of the adaptive EnKF algorithm are more closely aligned with the GPS data in Figure 16.

Figure 17 presents a comparison of the average smoothness angle distributions after filtering by the two EnKF algorithms. The comparison illustrates that the adaptive EnKF algorithm generally achieves slightly higher average smoothness angle than the conventional EnKF algorithm and displays significantly smaller fluctuations in smoothness. With computation, the average angle of trajectory points filtered by the conventional EnKF algorithm was found to be 172.52°, with a RMSE of 18.81°. The adaptive EnKF algorithm yielded an average angle of 174.16° and a RMSE of 14.60°.

In Segment I of Figure 17 (within the first 60 time steps on the $x$-axis), the conventional EnKF algorithm shows an average angle of 152.20° with an RMSE of 17.14°, while the adaptive EnKF achieves a higher average angle of 157.52° and a lower RMSE of 13.59°. This indicates that the adaptive EnKF better suppresses smoothness variations, maintaining stability when the conventional EnKF’s smoothness drops. In Segment II, during stable AUV movement, the adaptive EnKF maintains a higher smoothness angle of 177.60° with an lower RMSE of 5.34°, compared to the conventional EnKF's 176.72° and RMSE of 7.76°, showing less fluctuation. Overall, the adaptive EnKF consistently delivers higher smoothness and lower RMSE across different motion conditions (either turning or straight running), demonstrating superior filtering stability.

Figure 18 highlights how the number of ensemble members varies with smoothness in the adaptive EnKF filtering process, demonstrating the effectiveness of the adaptive mechanism. The mechanism promptly adjusts ensemble member counts based on smoothness variations, initially using a higher number of members during the initial phases of AUV motion, then reducing the number in later phases while responding to minor smoothness fluctuations. This efficient adjustment helps conserve computational resources. Overall, the adaptive EnKF algorithm enhances both filtering smoothness and stability compared to the conventional EnKF, proving the adaptive mechanism’s effectiveness.

To compare the positioning accuracy of the algorithms, GPS data is used as a reference to calculate the average Euclidean distance between the filtering results and GPS positions. To better illustrate positioning trends, this distance is computed every 30 sampling points. However, due to environmental disturbances during the AUV trial, some GPS data is unstable. Therefore, four stable segments of GPS data were selected for comparison, as shown in Figure 19. Both EnKF algorithms show similar trends, but the adaptive EnKF (red curve) consistently achieves lower errors than the conventional EnKF (blue curve), demonstrating better error control. In the four segments, the average positioning deviations for the conventional and adaptive EnKF algorithms are 4.28 m and 3.47 m, respectively, with RMSE values of 2.54 m and 1.67 m. These results indicate that the adaptive EnKF provides superior accuracy in reducing positioning errors compared to the conventional EnKF.

Finally, the filtering performance data of the two EnKF algorithms are summarized and compared in Table 2. In conjunction with the aforementioned analysis of filtering smoothness and accuracy, the adaptive EnKF algorithm is capable of reducing fluctuations in positioning error without compromising the filtering smoothness. Consequently, the adaptive EnKF algorithm demonstrates superior performance in terms of positioning accuracy, stability, and filtering smoothness in comparison to the conventional EnKF algorithm.

6.2. Comparison with EKF

In this subsection, the filtered results of the adaptive EnKF algorithm are compared with the ones of the EKF algorithm. The filtered trajectories are presented in Figure 20. The figure shows that EKF also exhibits satisfactory filtering performance.

Figure 21 compares the trajectory smoothness curves after filtering with both the EKF and adaptive EnKF algorithms. The EKF shows a slightly higher average smoothness angle than the adaptive EnKF, but with significantly larger fluctuations in some regions. The overall average smoothness angle for the EKF is 175.09° with the RMSE of 13.13°, while the adaptive EnKF's values are 172.52° and 18.81°, respectively. In Segment I, where AUV movement is complex, the EKF achieves an average angle of 160.47° with an RMSE of 12.15°, whereas the adaptive EnKF shows a lower average angle of 152.20° and a higher RMSE of 17.14°. This indicates that, despite improvements over the conventional EnKF, the adaptive EnKF still struggles with filtering smoothness during frequent attitude changes, performing worse than EKF in such scenarios. In the relatively stable middle and later stages of the motion trajectory, represented by segment II in Figure 21, the adaptive EnKF algorithm demonstrates its ability to resist minor disturbances that could affect filtering smoothness, maintaining a high level of smoothness with minimal fluctuations. The average smoothness and the stability of the proposed algorithm are comparable to those of EKF.

Regarding filtering smoothness, the adaptive EnKF algorithm generally performs less better than EKF, especially when the motion state of AUV is varying obviously. Conversely, when the motion state is relatively stable, the smoothness of the adaptive EnKF and EKF filtered trajectories seems similar, and both algorithms exhibit good stability.

To further compare the positioning accuracy of the algorithms, the position deviations between the EKF and the adaptive EnKF filtering results and the GPS positioning data are given in Figure 22. In the figure, the position deviation of both filtering algorithms remains relatively stable within the four selected segments, maintaining errors within a low range. The average position deviation of EKF across the four segments, represented by blue curve in Figure 22, is 5.35 m, with an RMSE of 0.88m, while the adaptive EnKF represented by red curve is 3.47m and 1.67m. Although EKF exhibits relatively stable positioning error, the adaptive EnKF algorithm demonstrates superior overall positioning accuracy across diverse segments.

The summary and comparison of filtering performance data of the adaptive EnKF and EKF in this section are shown in Table 3. From the data, in summary, the proposed adaptive EnKF algorithm presents superior overall performance in terms of positioning accuracy and nearly comparable in terms of filtering smoothness compared to EKF.

6.3. Discussions

By comparing the results presented in Table 2 and Table 3, we can summarize the filtering performance of the three algorithms: EKF, the conventional EnKF and the adaptive EnKF. With regard to positioning accuracy, All of them perform well in positioning accuracy, with the adaptive EnKF showing the most optimal performance by preventing error divergence and maintaining stable errors. In terms of filtering smoothness, the conventional EnKF shows inherent instability and fluctuations due to its reliance on the Monte Carlo method for ensemble state reconstruction. Consequently, its filtering smoothness is generally inferior to that of EKF, particularly under complex AUV motion conditions. However, the adaptive EnKF, which leverages adaptive mechanism and a Laplace distribution-based vector ensemble generation method, achieves filtering smoothness and stability comparable to the EKF algorithm under stable motion conditions. Although EKF demonstrates superior overall smoothness in complex AUV motion conditions, the adaptive EnKF performs equally well under stable conditions and shows superior stability in filtering smoothness and positioning accuracy compared to EKF. This advantage becomes even more pronounced when the system operates in complex marine environments. In comparison to EKF and conventional EnKF, the adaptive EnKF is better suited to address the potential challenges posed by various non-Gaussian noises that may emerge in real underwater settings and impede the accuracy of system observations.

6. Conclusions

This study presented an enhanced adaptive EnKF algorithm for integrating AUV navigation information from sensors like DVL, electronic compass, and GPS to optimize positioning. The conventional EnKF algorithm was modified by using the Laplace distribution for state approximation. Optimal parameter settings were identified through field trial data.

We defined the smoothness angle to evaluate the smoothness of the filtering trajectory by calculating angles formed by the lines connecting three consecutive trajectory points. This evaluation index, along with the runtime of the algorithm, was used to analyze the impact of ensemble size on EnKF filtering performance. The results indicated that increasing the number of ensemble members significantly improved both runtime and filtering smoothness.

An adaptive mechanism was introduced for adjusting the number of ensemble members. This method quantitatively evaluated filtering performance based on the smoothness of the filtered trajectory and the credibility of the filtering outputs to adjust ensemble size accordingly. The effectiveness of this adaptive mechanism in efficiency and smoothness were validated through the comparison among four scenarios with different EnKF configurations.

The performance of the proposed adaptive EnKF algorithm was compared with the ones of the conventional EnKF and EKF algorithms. The adaptive EnKF outperformed both conventional EnKF and EKF in positioning accuracy, stability, and filtering smoothness. While EKF excelled in complex conditions, adaptive EnKF performed comparably well in stable scenarios and showed better stability and accuracy. Future research will explore broader data sources and further parameter adjustments to enhance the algorithm's applicability.