A Review of the Application of Transfer Learning in Fault Diagnosis and its Potential in Aerospace Condition Based Maintenance

1. Introduction

In aerospace, Maintenance, Repair and Overhaul (MRO) is a crucial sector for ensuring the safety and reliability of aircraft, and significant costs have been associated with this sector. It has been estimated that the overall cost of carrying out MRO accounts for 10-15% of an airline carrier’s budget [1]. The global commercial MRO demand is forecasted to be $ 78.5 billion for 2022 and is expected to grow to $ 121.2 billion by 2032 [2]. Condition based maintenance (CBM) is a technique that helps to reduce the cost of MRO. Using run-time data to determine the health state of aircraft, CBM enables operators to conduct inspection and maintenance based on evidence of need rather than routine time-based inspections, thereby reducing the cost of performing routine maintenance [3].

1.1. OSA-CBM

A general architecture to deliver CBM service is depicted by the Open Standard Architecture for Condition Based Maintenance (OSA-CBM), which is shown in Figure 1 [4]. OSA-CBM defines a typical CBM workflow containing data acquisition, data manipulation, state detection, health assessment, prognosis assessment, and finally, an advisory generation stage. Fault diagnosis concerns the ‘Health Assessment’ stage in the OSA-CBM framework, which produces the health state or degradation level of monitored systems, subsystems, or components [5]. Fault diagnosis is one of the most crucial stages in the OSA-CBM framework, because it converts aircraft data into health information. Hence, having an accurate prediction method from aircraft data to the health state of the aircraft is essential to the reliability of CBM.

1.2. Transfer Learning for OSA-CBM

Traditionally, machine fault diagnosis refers to the process of determining the health state of a machine from the symptoms observed through the experience and knowledge of engineers [6]. In recent years, with advances in machine learning (ML) theories, ML-based fault diagnosis, the modern-day equivalent to the traditional fault diagnosis process, has received increasing attention from both academic and industrial researchers [7,8]. Compared with traditional methods that rely on human experts, ML-based fault diagnosis is believed to provide automation in various diagnostic stages and improve diagnostic accuracy [9], removing people from the loop. In addition, as systems become more complex and integrated, machines produce larger and richer datasets, which allow ML-based fault diagnosis to demonstrate its much greater diagnostic capacity than traditional methods [10].

Although machine learning empowers fault diagnosis with greater diagnostic capacity than traditional methods, machine learning models are prone to error if not applied carefully, especially when they are applied to scenarios different from the training scenarios, hence the need to transfer the models from one scenario to another [11]. Aiming to achieve such transfer of knowledge, transfer learning (TL) has been widely applied in fields such as image classification, natural language processing, recommendation systems, and human activity classification [12,13].

In fault diagnosis, TL has received increasing attention in recent years. Currently, TL has been successfully applied to bridge the gap between fault diagnosis in academic research and its application in practical scenarios [14]. The root cause for such a gap can be summarised by four aspects: 1) sample sizes are small for new machines, 2) labels are missing due to the labour-intensive nature of labelling or technical difficulties in labelling, 3) datasets are usually unbalanced with too few faulty case data, since machines generally operate under healthy conditions, and: 4) training data can display a distributional discrepancy from the testing data, for machines operate under various conditions [15]. Furthermore, the ability of TL to transfer diagnostic models between different fault severities has also demonstrated its strength in capturing incipient faults [16,17], which will contribute to achieving the desire for prognosis in the future [18].

Since TL has demonstrated its ability to generate accurate diagnostic results under challenges from real-scenario data and improve the reliability of fault diagnosis, this work aims to review the application of TL in fault diagnosis and discover its fullest potential in aerospace CBM.

1.3. Analysis of Existing Reviews in the Related Field

Several review papers exist for the application of TL in fault diagnosis, which are summarised in Table 1. The common elements among the existing reviews include the description of various TL approaches and examples of applying TL in fault diagnosis gathered from relevant research. Meanwhile, each review focused on different perspectives among the related fields.

Focusing on the most general perspective, [9,14] are the early works that reviewed different categories of TL with examples in fault diagnosis. [14] analysed relevant literatures in terms of their motivations, problem settings, specific approaches, and specific applications. [9] pointed out that TL could be the most high-potential technique for developing fault diagnosis solutions. General discussion of TL and fault diagnosis was also found in a more recent review, [19], which surveyed a much richer collection of relevant literature and systematically categorised them based on multiple criteria from their problem settings, solutions, and applications.

More recently, reviews in this field concentrated their focus on deep transfer learning (DTL) methods. [20] provided a comprehensive introduction to various DTL methods, detailing the major algorithms in each of the four main categories of DTL: instance-based, feature-based, parameter-based, and adversarial-based DTL. It raised examples of the application of these methods in fault diagnosis and suggested weakness in the methods and future progress. Similarly, [21] also discussed the application of DTL in fault diagnosis but respectively analysed it for feature extraction and fault classification. Evaluating DTL algorithms by their performance in specific datasets, [22] obtained conclusions about the choice of DTL algorithms for specific fault diagnosis problems. Other reviews on DTL took several different specific focuses. Focusing on specific category of DTL, [23] comprehensively investigated adversarial-based DTL and detailed how various adversarial-based DTL could be applied to different transfer settings. Focusing on different tasks and problem settings, [24] summarised different DTL methods applied to unlabelled target domains. [8] discussed the application of various DTL methods to common industrial scenarios and suggested DTL algorithms for several industrial use cases. Including both fault diagnosis and fault prognosis in its discussion, [25] reviewed how DTL facilitates fault diagnosis and remaining useful life.

Table 1. Summary of published reviews in the similar field.

Reference	Year of publication	Type of TL method discussed	Main contribution
Zheng et al.[14]	2019	Non-DTL DTL	This is an early review of the application of TL in fault diagnosis. It explained the basics of various TL methods, including instance-based, feature-based, deep learning-based, and adversarial-based TL. Based on literature about the application of TL in fault diagnosis, it produced a summary of motivations, problem settings, specific approaches, and specific applications of relevant research and suggested future directions.
Lei et al. [9]	2020	Non-DTL DTL	This paper reviewed the general application of machine learning to fault diagnosis. It illustrated a development roadmap of fault diagnosis solutions, where traditional machine learning methods is “the past”, deep learning is “the present”, and transfer learning may be “the future”. In the TL section, it provided top-level description to instance-based, feature-based, parameter-based, and adversarial-based TL algorithm with fault diagnosis examples.
Li et al. [21]	2020	DTL	This paper outlined the principle of common DTL methods for fault diagnosis, and it reviewed the application of DTL in fault diagnosis respectively for feature extraction and fault classification.
Zhao et al. [24]	2021	Unsupervised DTL	This paper focused on unsupervised DTL (i.e., DTL with unlabelled target domain), outlined how various unsupervised DTL algorithms apply to label-consistent, label-inconsistent, multi-domain transfer problems, and tested on bearings and gears datasets.
Li et al. [8]	2022	DTL	This paper discussed three main categories of DTL (i.e., instance-based, feature-based, parameter-based) applied to four industrial scenarios for fault diagnosis: general performance improvement, partial domain fault diagnosis, emerging fault diagnosis, and compound fault decoupling, and it suggested DTL solution for various industrial needs.
Qian et al. [20]	2022	DTL	This paper comprehensively reviewed the application of DTL in fault diagnosis by explaining the principle of four major DTL categories: instance-based, feature-based, parameter-based, and adversarial-based DTL.
Yao et al. [25]	2022	Non-DTL DTL	This paper explained general categories of TL methods (feature-based, parameter-based, and adversarial-based) in both sallow and deep networks, and their application in fault diagnosis and fault prognosis (i.e., remaining useful life prediction).
Yang et al. [22]	2023	DTL	This review analysed various DTL algorithms as applied to specific datasets and provided evaluation from the fault diagnosis perspective.
Guo et al. [23]	2023	Adversarial DTL	This paper provided an in-depth review into the application of adversarial DTL (i.e., DTL with adversarial training) in fault diagnosis, and it detailed how non-generative adversarial DTL and generative adversarial DTL applies in different transfer settings.
Azari et al. [19]	2023	Non-DTL DTL	This paper reviewed the application of TL in predictive maintenance, and it systematically categorised relevant research by multiple criteria from problem settings, solutions, and applications.

Table 1. Summary of published reviews in the similar field.

Reference	Year of publication	Type of TL method discussed	Main contribution
Zheng et al.[14]	2019	Non-DTL DTL	This is an early review of the application of TL in fault diagnosis. It explained the basics of various TL methods, including instance-based, feature-based, deep learning-based, and adversarial-based TL. Based on literature about the application of TL in fault diagnosis, it produced a summary of motivations, problem settings, specific approaches, and specific applications of relevant research and suggested future directions.
Lei et al. [9]	2020	Non-DTL DTL	This paper reviewed the general application of machine learning to fault diagnosis. It illustrated a development roadmap of fault diagnosis solutions, where traditional machine learning methods is “the past”, deep learning is “the present”, and transfer learning may be “the future”. In the TL section, it provided top-level description to instance-based, feature-based, parameter-based, and adversarial-based TL algorithm with fault diagnosis examples.
Li et al. [21]	2020	DTL	This paper outlined the principle of common DTL methods for fault diagnosis, and it reviewed the application of DTL in fault diagnosis respectively for feature extraction and fault classification.
Zhao et al. [24]	2021	Unsupervised DTL	This paper focused on unsupervised DTL (i.e., DTL with unlabelled target domain), outlined how various unsupervised DTL algorithms apply to label-consistent, label-inconsistent, multi-domain transfer problems, and tested on bearings and gears datasets.
Li et al. [8]	2022	DTL	This paper discussed three main categories of DTL (i.e., instance-based, feature-based, parameter-based) applied to four industrial scenarios for fault diagnosis: general performance improvement, partial domain fault diagnosis, emerging fault diagnosis, and compound fault decoupling, and it suggested DTL solution for various industrial needs.
Qian et al. [20]	2022	DTL	This paper comprehensively reviewed the application of DTL in fault diagnosis by explaining the principle of four major DTL categories: instance-based, feature-based, parameter-based, and adversarial-based DTL.
Yao et al. [25]	2022	Non-DTL DTL	This paper explained general categories of TL methods (feature-based, parameter-based, and adversarial-based) in both sallow and deep networks, and their application in fault diagnosis and fault prognosis (i.e., remaining useful life prediction).
Yang et al. [22]	2023	DTL	This review analysed various DTL algorithms as applied to specific datasets and provided evaluation from the fault diagnosis perspective.
Guo et al. [23]	2023	Adversarial DTL	This paper provided an in-depth review into the application of adversarial DTL (i.e., DTL with adversarial training) in fault diagnosis, and it detailed how non-generative adversarial DTL and generative adversarial DTL applies in different transfer settings.
Azari et al. [19]	2023	Non-DTL DTL	This paper reviewed the application of TL in predictive maintenance, and it systematically categorised relevant research by multiple criteria from problem settings, solutions, and applications.

Despite that the existing reviews have presented a comprehensive overview of various TL methods and how they have been applied to various fault diagnosis problems, there is a common limitation, which is they have only considered the TL algorithms that have been already applied to fault diagnosis problems without considering whole scope of TL, irrespective of its applications, and covering all the possible ways that TL may facilitate fault diagnosis. Hence, there is the necessity to comprehensively study TL in general and identify any TL methods, from the whole scope of TL, that may benefit fault diagnosis solutions, so that the potential of TL in fault diagnosis applications can be fully explored. This review has chosen aerospace CBM as the topic of concern and aims to explore how the application of TL in fault diagnosis, in ways that exist or not, could facilitate aerospace CBM.

With the aim to understand the general topic of TL irrespective of its applications, Section 2 of this review will investigate the history of TL to identify the whole scope of TL, and then by comparing TL with some common aerospace fault diagnosis algorithms, try to draw inspiration about high-potential TL methods in the fields of concern. Section 3 focuses on the existing application of TL in fault diagnosis. It evaluates the various types of TL applied to fault diagnosis and studies the specific applications and transfer scenarios in the existing literature. Section 4 focuses on the application of TL specifically in aerospace fault diagnosis, which is directly related to how TL has contributed to aerospace CBM. Then in the beginning of Section 5, research gaps and limitation in the existing literature from Section 3 and 4 are raised. Combining all the information above, future progress in applying TL in fault diagnosis and its benefit and potential in aerospace CBM will be suggested in Section 5. Finally, Section 6 summarises the main contribution of this review and provides a conclusion.

2. Transfer Learning

To understand the entire scope of TL and explore its potential to the greatest extent possible, this section discusses TL from several different perspectives. To start with, Section 2.1 looks into TL from a historical perspective, revealing how TL originates and what the entire scope of TL covers. Then, in Section 2.2, the most up-to-date definition and general structure of TL is introduced. Section 2.3 takes a more holistic perspective by not limiting the discussion to TL and compares TL with other methods using the idea of reusing previous solutions for new problems, which aims to find possible inspiration for further developing TL. Finally, Section 2.4 introduces a method called transfer learning by structural analogy, which is believed to have high potential in fault diagnosis.

2.1. History of Transfer Learning

This section presents a detailed overview of the history of TL. By doing so, the scope of TL can be clarified and implications from a historical perspective can be drawn.

It is commonly believed that TL originates from transfer of learning, a method capturing how “people often apply the knowledge gained from previous learning tasks to help learn a new task” [11]. While most researchers marked the similarity between knowledge transfer in human learning activities and machine learning, no discussion has been found that explicitly described how transfer of learning progressed to TL.

Figure 2 summarises how TL evolved from transfer of learning, based on the literature gathered by this work. In the top half of Figure 2, key theories of transfer of learning in the field of human learning are listed against time. In the bottom half of Figure 2, key developments in TL as a machine learning method are shown. Starting from the top half of Figure 2, transfer of learning is commonly studied in two fields: motor learning and education theory. Transfer of learning in motor learning inspired the mainstream TL methods, which are explained in Section 2.1.1. Transfer of learning in education theory inspired the high-level transfer methods, which are explained in Section 2.1.2.

2.1.1. Inspiration from Motor Learning

The branch of transfer of learning in motor learning motivated the early works on TL that developed to the mainstream TL method today. The green path in Figure 2 highlights this historical development.

Transfer of learning in motor learning activities is also referred to as bilateral transfer or cross-education, which describes the behaviour that “practice of one part of the body in performing a skilled act increases the ability of the bilaterally symmetrical part in the same act” [26]. For instance, training the left hand on a certain action could result in an improvement in the right hand when performing the same action. The experiment by Bray was one of the early works attempting to comprehensively validate this phenomenon. In his experiment, the transfer of skills was not only tested between symmetrical limbs, such as between the left hand and the right hand, but also between asymmetrical limbs, such as between the right hand and the right foot [26]. Pretraining other limbs on a drawing task seemed to improve the testing limb on the same task. After interviewing the subjects, Bray concluded that the reasons for the behaviour were: i) the transfer of method, i.e., the subjects applied the method learnt during the pretraining phase to the new tasks rather than the limbs transferring the motor learning, and: ii) a reduction in nervousness after the pretraining phase. Finally, Bray designed an alternative experiment aimed at eliminating the influence of these two factors, but he was unable to prove whether there were other factors explaining the transfer, since it was extremely difficult to completely eliminate all influences from the experimental design. Research into transfer of learning on motor learning activities still remains an active branch of research today. However, with advances in neuroscience during the last century, researchers are now proposing explanations for the transfer of learning by biological brain activities. For example, Boroujeni and Shahbazi [27] observed bilateral transfer of badminton short service skills between the dominant and non-dominant hands, suggesting the reason as an overflow of motor impulses from the pretrained part of the body. Similarly, Kumar and Mandal [28] observed bilateral transfer of mirror-drawing skills and explained the transfer mechanism as interhemispheric transfer.

How transfer of learning in motor learning motivated the mainstream TL has not been formally documented. This work believes that advances in the understanding of how the human brain conducts knowledge transfer shows a strong relation to TL, especially by neural networks. The understanding of biological neural networks inspired the design of artificial neural networks (ANNs), and the design of ANNs helped infer how the brain executes the same action [29]. Based on an ANN, the first mathematical modelling of TL among neural networks was developed in 1976, and the first work on TL in machine learning appeared in 1993 [30].

2.1.2. Inspiration from Education Theory

The branch of transfer of learning in education theory motivated TL methods focusing on high-level transfer, which considered the scenario where the source domain and the target domain do not share low-level similarities such as identical physical parameters and data structures. The red path in Figure 2 highlights this historical development.

Education theory explains transfer of learning on the conceptual level rather than the biological level of human learning. In education theory, “there is little agreement in the scholarly community about the nature of transfer, the extent to which it occurs, and the nature of its underlying mechanisms” [31]. Therefore, two of the most popular theories explaining the knowledge transfer are introduced here. The first mechanism is called the “common element” theory of transfer. As early as 1901, Thorndike and Woodworth proposed this theory to explain the knowledge transfer behaviour by stimulus-response associations and suggested that transfer increased for larger overlapping of common elements between the learning and testing tasks[32]. Taking playing the violin as an example, the early approach of the “common element” theory could explain why violin players can play the same melody in different pitches, as they contain the same motor sequence as the common element [32]. However, the observation of transfer exists not only for the same melody, but also among different sonatas, such as between a violin sonata by Händel and a Mozart sonata [32]. Hence, a modern approach of understanding the “common element” has taken the transferred common element as the production, i.e., “the procedural knowledge comprising of specific conditions followed by actions” [32]. Repeated practice accumulates productions consisting of sets of notes and the corresponding sets of finger movements, and when the same conditions (sets of notes) are met in new pieces, the player could transfer the productions and respond with the sets of finger movements [32]. The second mechanism is analogical transfer, which is a systematic framework capturing how humans rely on analogy to learn new tasks. Steiner [32] explained this framework using an example in which a young girl applied analogical transfer to identify a previously unknown object, a green walnut. The key step of the process is to retrieve known objects bearing analogies to the unknown object of the green walnut and to determine whether the green walnut shares certain properties with the analogical objects [32]. The integration of learning several properties would lead to the final identification of the unknown object as a green walnut. Figure 3 illustrates the first cycle of the process of recognising the green walnut. The cycle starts by finding that a small apple can be analogous to the green walnut, since they are both small green objects. Because apples are biteable, the same property can be tested on the green walnut, and a biting test results in the conclusion that the green walnut is not biteable, which leads to the knowledge that the skin of the unknown object, the green walnut, should be peeled [32].

Two TL methods are motivated by this branch of transfer of learning. The first method is Transitive Transfer Learning (TTL). Inspired by how humans make indirect inferences and learn by connecting many intermediate concepts to transfer knowledge between two seemingly unrelated fields, TTL adopts an intermediate domain to transfer knowledge between a source domain and a target domain with a very large gap [11]. The second method is transfer learning by structural analogy, which is inspired by how humans use analogy to connect seemingly unrelated ideas [33]. In transfer learning by structural analogy, analogical pairs are assigned between entities in two distant domains based on their structural relation, and the transfer of knowledge is applied by treating the entities in each analogical pair as equivalents [33].

2.1.3. Implication from the History of Transfer Learning

Although history shows that TL received inspiration from biological understanding and conceptual understanding of how knowledge transfer occurs in human learning activities, the TL methods motivated by the two branches have not received equal attention in engineering scenarios. Inspired by the biological understanding of knowledge transfer in human learning, the mainstream TL methods have seen abundant applications in engineering problems such as fault diagnosis. By contrast, TTL and transfer learning by structural analogy, inspired by the conceptual understanding of knowledge transfer in human learning, have never been applied to engineering problems. However, it is important to explore these methods, since they consider high-level transfer scenarios where low-level similarities no longer exist between the source and target domain. As one of the most important findings from investigating the history of TL, understanding how to implement high-level transfer to solve fault diagnosis problems should be conducted to complete the research on TL in fault diagnosis.

2.2. Definition and General Structure of Transfer Learning

This work defines TL as “the machine learning paradigm in which an algorithm extracts knowledge from one or more application scenarios to help boost the learning performance in a target scenario” [11]. The general structure of TL is shown in Figure 4. The term source domain is used to indicate where knowledge is borrowed from, and the term target domain is used to denote where the borrowed knowledge is applied. A traditional machine learning process is shown on the left of Figure 4. Borrowing knowledge from data, model or task description from the source domain to solve a new task in the target domain describes a TL-based approach, which is shown on the right of Figure 4. It should be pointed out that while most TL research defines the source and target domain as being “different but related”, the definition followed by this work removes this constraint, since Section 2.1 has concluded that the whole scope of TL includes both low-level TL methods and high-level TL methods, i.e., the TL methods between seemingly unrelated source and target domains.

2.3. Transfer Learning Compared to Other Methods Involving the Element of Knowledge Transfer

To understand TL in a broader context, it should be mentioned that the idea of leveraging previous knowledge is not unique to TL. Three methods that function by reusing previous knowledge for new tasks are compared with TL to provide new insights: case-based reasoning (CBR), procedural reasoning system (PRS), and analogical transfer. CBR and PRS are common algorithms in aerospace fault diagnosis, hence the comparison with TL is believed to be particularly meaningful for inspiring how TL can further facilitate aerospace fault diagnosis. The key features of these methods are summarised in Table 2 and compared with TL. Since this section aims to provide a general overview to compare and contrast the methods on a conceptual level, the technical details of each method will not be discussed.

Starting with CBR, as a method that “solves new problems by adapting solutions that were used to solve old problems” [34], the properties and procedure of CBR inspired the framework of analysing all the methods in Table 2. Three key properties of each method are extracted: the domains between which transfer of knowledge occurs, how the domains should be related, and what knowledge is transferred. For CBR, knowledge is transferred from previous cases to new cases, given that they are similar cases in the same field, and the knowledge being transferred is the previous solution of the matched historical cases [35]. The main procedure of CBR, “Retrieve – Reuse – Revise – Retain”, is taken as a cycle of stages to leverage previous knowledge. In the “retrieve” stage, one or more cases are retrieved from the case base to match the new problem, and then a solution is suggested from the matched cases and tested on the new case in the “reuse” stage [35]. If the retrieved case is not a close match, adaptation of the solution from the retrieved case will be necessary in the “revise” stage, which produces a new case to be retained in the case base [35]. This four-stage description of the CBR process is not only terminology for CBR but can also be viewed as a framework for leveraging previous knowledge in general. The “retrieve”, “reuse” and “revise” stages can be generalised as three logically inevitable stages of any knowledge reuse method, where previous knowledge is first retrieved and then re-applied to the new problem with modification if necessary. The “retain” stage can be used to describe the extension of existing knowledge following newly solved problems.

PRS can also be understood in light of the aforementioned framework. In PRS, previous knowledge is extracted from a library of knowledge areas (KAs) and applied when a new goal appears. The knowledge transferred to the new task is previously stored KAs, which are “sequences of actions and tests that may be performed to achieve given goals or to react to certain situations” [36]. Containing specific actions to take under certain conditions, KAs can only be transferred between similar problems with similar prescribed conditions. Hence, the knowledge transfer can only be considered for problems in the same field. The main stages of a PRS cycle are: 1) for the “retrieve” stage, various KAs are triggered following a new goal or an altered system belief, 2) for the “reuse” stage, one or more KAs are selected and executed within the intention structure of PRS, which leads to a new subgoal or a new belief, 3) this new subgoal or new belief then starts another PRS cycle, which resembles a “revise” stage [37]. The “retain” stage is not obvious for PRS.

In human learning theories, analogical transfer is a method for leveraging previous knowledge. It is a method that facilitates the understanding of unlearnt knowledge from previously learnt knowledge. Working by discovering true analogies in problems sharing “a similar deep structure but not necessarily specific content”, no fixed relations are imposed on the domains of the knowledge transfer process [32]. Specifically, the knowledge transferred includes source analogues and assimilation schemas from the source analogues. A detailed example of an analogical transfer cycle is provided in Section 2.1.2, the “green walnut” example. For the “retrieve” stage, the source analogue of a small apple is retrieved, and the assimilation schema is identified as being biteable. The biting schema is then tested in the “reuse” stage, which leads to the knowledge integration that the green walnut is not biteable and therefore should be peeled in the “retain” stage. Having determined that the green walnut should be peeled, another source analogue is found to be a banana, since they both have skins [32]. Then, another analogical transfer cycle begins, which works as the “revise” stage.

The properties and procedures of TL were also extracted to compare TL with other knowledge transfer methods. Starting with the properties of TL, knowledge is transferred from the source domain to the target domain. The relation between the two domains is commonly defined as “different but related”, and “the label space of the source domain should overlap that of the target domain” [9]. The knowledge transferred in TL depends on the specific category of the TL method. For instance-based TL, the knowledge transferred is borne in the source domain data instances. The main stages are as follows: 1) the auxiliary data instances are retrieved from the source domain, 2) the auxiliary instances are used to train a target domain classifier, 3) the weighting of the auxiliary instances and target domain instances is optimised for target task classification accuracy, and 4) a new classifier is obtained for the target domain task. In feature-based TL, the knowledge transferred is borne in the source domain features. The main stages are as follows: 1) selected data features are retrieved from the source domain, 2) the source domain features and target domain features are unified in the feature space and are used together to train a domain-invariant classifier, 3) the distribution distance between the source and target domain features is minimised so that the domain-invariant classifier performs well in the target task, and 4) a new classifier is obtained for the target domain task. In model-based TL, the knowledge transferred is borne in the model parameters or architectures. The main stages are: 1) a model pretrained in the source domain is retrieved, 2) the pretrained model is adapted to the target task by freezing, fine-tuning, or retraining with target domain data, 3) the transfer strategy and parameter tuning can be changed to suit each specific target application, and 4) a new classifier is obtained for the target domain task.

From the above discussion, it is clear that TL actually shares similarities in its properties and procedures with other methods of leveraging previous knowledge. Sharing a similar goal, the other methods described in this section could inspire improvements in TL. One such inspiration from Table 2 is that, while all other methods transfer knowledge between similar domains, analogical transfer does not require low-level similarity to function. This shows that analogy could be a useful tool for leveraging previous knowledge from seemingly unrelated domains, which opens up wider opportunities for TL. The following section describes a TL method based on analogy.

2.4. Transfer Learning by Structural Analogy: A High-Potential Method in Fault Diagnosis

As Section 2.1 pointed out, the TL methods inspired by the conceptual level understanding of human knowledge transfer are able to address high-level transfer problems between two seemingly unrelated domains. Since high-level transfer has not been seriously considered in engineering scenarios, it is considered a high-potential direction for future TL studies. Section 2.3 concludes that applying analogies could be a way to expand the boundaries of TL methods. Combing these thoughts, one method aimed at high-level transfer problems that shows potential in fault diagnosis is TL by structural analogy. This method, proposed by Wang et al. [33], achieved knowledge transfer by finding analogical pairs of entities from two domains of interest and extracting the relational similarity between the two domains. The analogical pairs are found by simultaneously minimising the distance between entities from the two domains and the distance between the selected entities to the labels in their respective domain, so that the analogy made is also meaningful for label prediction [33]. To demonstrate the validity, Wang et al. [33] applied the algorithm to find analogical pairs of words in diagnosis documents for cardiovascular and respiratory tract diseases. The results are shown in Table 3. By treating the words in every analogical pair as equivalents, the classifier trained on source domain documents achieved 80.5% accuracy when applied to target domain documents, which was significantly higher than the 50% accuracy when the classifier was transferred without the information from structural analogy [33]. It should be noted that the words in the analogical pairs in Table 3 have no literal overlap, so this proves that the algorithm can make an analogy between seemingly unrelated entities and, based on the analogy made, enables positive knowledge transfer between two seemingly different domains.

Although the example is not in an engineering scenario, the task of diagnosing a disease from a symptom document is very similar to a fault diagnosis task in which the machine health state is diagnosed from the observed symptom vector. Hence, applying transfer learning by structural analogy could be a promising method for fault diagnosis scenarios. Upon successful implementation, it would expand the boundary of TL in fault diagnosis from being between similar machines to being between seemingly unrelated machines as well.

3. Transfer Learning in Fault Diagnosis

The following subsections focus on the application of TL in fault diagnosis. Section 3.1 introduces the different categories of TL methods used for fault diagnosis and briefly evaluates each category. Section 3.2 to 3.3 present a literature survey conducted on the existing research on TL in fault diagnosis by gathering literature from the search of “transfer learning” and “machine fault diagnosis” in the article title, abstract, and keywords on Scopus. The search returned 718 entries in January 2023.

Two major aspects are discussed based on these entries, namely the specific applications and the relationship between the source and target domains in these works, which are presented in sections 3.2 and 3.3, respectively. For these two sections, the first 200 most relevant entries out of the 718 entries were selected as representative entries, because a generally repeating pattern of the research trends was identified for the first 100 entries and the second 100 entries. All figures in these sections are obtained from the analysis of representative entries. For transparency in data analysis, details of the representative entries are reported in Appendix A.

3.1. Classification of Existing Transfer Learning Methods in Fault Diagnosis

To understand how TL methods have been applied to fault diagnosis problems in detail, this section describes the popular categories of TL methods in fault diagnosis, and a summary of their pros and cons is listed in Table 4. Based on the nature of the knowledge being transferred, TL methods commonly applied to fault diagnosis can be divided into instance-based TL, feature-based TL, and parameter-based TL.

The differences between the instance-based TL, feature-based TL, and parameter-based TL are shown in Figure 5. The architecture of each category of the TL method is marked along with a typical fault diagnosis process, which takes raw data as the input and transforms raw data into features using feature extraction, before a classifier acts on the features to produce a predicted health state for the machine.

3.1.1. Instance-Based Transfer Learning

For instance-based TL, knowledge transfer is essentially achieved by reusing labelled source domain data as auxiliary data to train a target domain classifier [11]. As shown at the top of Figure 5, knowledge transfer occurred in the domain of the raw data. TrAdaboost is a common instance-based TL method, which applies differential instance weighting mechanisms for the source and target domains, where weights are reduced for misclassification samples in the source training set to exclude unrelated samples but strengthened in the target training set to enhance the training [38]. Using k-nearest neighbours (kNN) as the classifier, Yang et al. [38] extracted fault information from a group of electricity distribution transformers to train a model to diagnose a target transformer with little data. Combining TrAdaboost with support vector machine (SVM) as the fault classifier, Qiu et al. [39] used data from an old model of a fuel pump as the auxiliary training data in the diagnosis task of a new model of the fuel pump with scarce data. Compared with the SVM trained with only target domain data or a combination of target and auxiliary data, implementing TrAdaBoost when training the SVM significantly improved the diagnostic accuracy [39]. This example shows that instance-based TL methods are particularly useful for small target data problems if abundant auxiliary data exist. Working with raw data, it is a highly interpretable approach and can be combined with various classifiers to suit specific applications.

However, this method has two major limitations. Firstly, if there is abundant target domain data, applying TrAdaboost could compromise the prediction accuracy, compared to directly training with target domain data [39]. Secondly, it is suitable only if the source domain data is very similar to the target domain data. If auxiliary data with significant distribution discrepancy is to be borrowed, additional processing is usually required. For instance, Du et al. [40] combined transfer component analysis (TCA) with TrAdaBoost to achieve knowledge transfer in the diagnosis of bearings under different conditions. Three fault modes were studied: an inner race fault, an outer race fault, and a roller fault. TCA is first applied to extract low-dimensional features from both the source and target domains with similar distributions, before TrAdaBoost is applied when using the source domain auxiliary training data to obtain the ultimate strong classifier based on Decision Tree [40].

3.1.2. Feature-Based Transfer Learning

Feature-based TL methods operate in an abstracted feature space rather than the raw data space [11]. The general process is shown in the middle of Figure 5. The source and target domain data are transformed into features in an abstracted feature space, where their distribution discrepancy is reduced to allow a classifier to act on the unified features.

Distribution discrepancy is classified into two categories. The first type is marginal distribution, which refers to the marginal probability distribution of cases regardless of their labels. The second type is conditional distribution, which is often approximated in fault diagnosis applications by the probability distribution of cases under each label.

The most fundamental algorithm is TCA, which addresses the marginal distribution discrepancy. In this method, a distance measure is required. For example, Xu et al. [41] characterised the distribution distance using the maximum mean discrepancy (MMD), and the distance between the transformed source and target domain features was minimised in a high-dimensional space. After TCA, a classifier (kNN) trained with transformed features from the source domain is transferred to predict target domain faults, and high diagnostic accuracy is demonstrated by applying the method on Case Western Reserve University (CWRU) bearing dataset data at different rotational speeds [41]. TCA has the advantage that it can also be combined with a range of classifiers, and it allows the inclusion of updated target training data to enhance its performance. However, conditional distribution discrepancy is not considered in TCA.

To consider both marginal distribution and conditional distribution simultaneously, joint distribution adaptation (JDA) is a commonly used method [42]. Qian et al. [43] took source and target domain data at different rotational speeds and loads and used JDA to process the data and SoftMax as the classifier. Their feature-based TL method resulted in 99.9% and 100% diagnostic accuracy over all fault types studied for the bearing and gearbox datasets, respectively. Hence, including the conditional distribution alignment, JDA is capable of providing high diagnostic accuracy, but it requires more computational time than other TL methods [43].

A common shortcoming of all feature-based methods is that since the classifier acts on abstract features transformed from the raw data, methods such as TCA and JDA are less intuitive than instance-based methods. Recent developments have focused on more advanced processing of distribution discrepancy to further improve the accuracy and reliability of feature-based TL methods. For instance, extending beyond aligning marginal and conditional distributions, manifold embedded distribution alignment (MEDA) also evaluates the relative importance of marginal and conditional distributions [44]. Partial domain adaptation (PDA), which deals with target domain with label space as a subset of that in the source domain, and open-set domain adaptation, where target domain contains unknown labels that do not belong to the source domain, both fall under the category of feature-based TL [45,46].

Combining feature-based TL methods with deep neural networks, feature-based deep transfer learning (DTL), also known as DTL through representation adaptation, is a popular solution for fault diagnosis. It refers to DTL methods with the fundamental aim of eliminating the distance between features from the source and target domains. Feature-based DTL can either align the features in the top layers of a deep model or in multiple layers. Using top layer adaptation, Xiang et al. [47] minimised the feature distance by calculating the MMD distance in the two fully connected layers of a CNN model when transferring it between different operating conditions of bearing datasets. Aligning all intermediate features using the multiple layer adaptation method, Xiao et al. [48] calculated the MMD distance for the intermediate features in the convolutional layers and the fully connected layers to transfer the CNN model between various working conditions of an induction motor. Compared to other feature-based TL, feature-based DTL provides an end-to-end automated fault diagnosis solution, but the lack of interpretability remains a drawback of this method.

3.1.3. Parameter-Based Transfer Learning

Parameter-based TL, also known as model-based TL, is different to the previously mentioned categories in that the transferred knowledge is encoded in “model parameters, priors or model architectures” [11]. This type of TL is most common in a DTL setting, which describes TL methods using deep neural networks as the prediction function [49].

In parameter-based TL, knowledge transfer occurs by reusing the parameters of the deep network trained by the source domain data to solve target problems. In Figure 5, this is represented by the bottom approach, where the source feature extraction and classifier are transferred to the target task. Several different strategies exist for implementing parameter-based DTL. In short, the parameters in each layer of a deep network can either be: 1) frozen, i.e., source domain parameters are used in the target problem directly; 2) fine-tuned, i.e., source domain parameters are fine-tuned by the target data before being applied to the target problem; or 3) retrained, i.e., source domain parameters are disregarded and new parameters trained from random values by the target data [50]. The best strategy for parameter transfer depends on the specific applications, and research suggests that the factors to consider could include the size of the target data and its similarity to the source data [14]. For instance, Dong et al. [50] concluded that fine-tuned feature extraction layers with frozen classification layers are more effective than fine-tuning all layers when transferring diagnostic knowledge from a dynamic bearing model to the experimental data of the same bearing.

Parameter-based DTL has the advantage of providing an end-to-end solution to fault diagnosis problems. Because deep networks combine feature extraction with a classifier, they can use machine data as the input and output the predicted health state, which is a highly automated approach [9]. Furthermore, since knowledge transferred concerns the model rather than any specific dataset, parameter-based TL shows higher potential to generalise on a higher level. However, deep networks being a “black box” solution means these methods generally have poor interpretability [20].

3.2. Application of Transfer Learning-Based Fault Diagnosis

The specific applications of TL in fault diagnosis, determined from what experiment data is used in each research work, is a topic of concern. As shown in Figure 6, bearings and gearboxes dominate the specific applications, and they account for nearly three-quarters of all applications. The remaining applications contain various items such as transformers, pumps, and motors, with the corresponding numbers of publications shown in Figure 6.

This trend in TL applications was also reported by Zheng et al. [14] who wrote a review paper on cross-domain fault diagnosis using knowledge transfer strategy, in which they identified bearings and gearboxes as “the two most widely (used) research and validation objects of current cross-domain diagnosis literature” and that they outnumbered other application objects by large margins. They speculated that this was because it was easier to find open-source datasets for these two objects [14].

3.3. Domains of Transfer

Another important feature is the relationship between the source and target domains in existing studies, as shown in Figure 7.

The most notable finding was that more than half of the analysed studies considered the transfer scenario of varied working conditions of the same machine. Three variations of working conditions are found in these studies, either individually or in combination: 1) varied loadings and rotational speeds, 2) various fault types during operation, and: 3) various degradation levels of the components. For instance, Zhu et al. [51] transferred the diagnostic model between four groups of CWRU bearing data taken at different loads, 0 HP, 1 HP, 2 HP, and 3 HP, while the rotational speeds remained almost the same. Dong et al. [52] explored the transfer of diagnostic models for ball bearing experimental data taken at three different rotational speeds (1000, 1500, and 2500 rpm) while maintaining a constant 5 kg radial load to the bearing. For the transfer scenario between different fault types, Xie et al. [53] designed a method based on generative adversarial networks to transfer diagnostic knowledge between three groups of CWRU data with no fault, inner ring fault, and outer ring fault. Regarding the transfer between various degradation levels, Chen et al. [16] transferred a deep neural network trained on data with a large-diameter fault (0.012 mm) to data with a small-diameter fault (0.007 mm).

The second most popular choice of transfer domains is between different representations of similar machines, which accounts for nearly a quarter of all domain choices studied. Different representations were found on three levels, including the transfer between different but similar machines, the transfer between virtual and physical assets, and the transfer between machines used in the laboratory (MUL) and machines used in real case (MURC). An example of transfer between different but similar machines is that Li et al. [54] considered a group of 15 wind turbines of the same type, and transfer is applied from 14 wind turbines to a similar wind turbine with small data. Bearing research by Dong et al. [50] provides an example of transfer between virtual and physical assets. Dynamic models were first constructed for bearings from two different laboratories, and using simulation data from the dynamic models as the auxiliary source domain data, diagnostic knowledge was transferred to real bearing experiment data [50]. Regarding the transfer between MUL and MURC, Yang et al. [55] used the data from SKF6205 bearing with a 52 mm outrace diameter and 3 HP load as the source domain data and transferred the diagnostic model to data from a locomotive bearing with a 160 mm outrace diameter and 9800 N vertical load [56].

Less common choices of the transfer domain are summarised as “split dataset” and “method transfer”. The “split dataset” refers to the scenarios where source and target domains are randomly split from the original dataset of the same machine under the same working condition. For example, Cao et al. [57] designed a gearbox experiment and generated 104 samples for each health condition. Under each health condition, the 104 samples were randomly selected as the source domain data and target domain data with the source domain data size varying between 2% and 80% of the 104 samples, and the effect on the diagnostic accuracy of the transferred model was discussed. The “method transfer” refers to the research where TL applies on the method level, such as transferring image classification deep networks to fault classification. For example, Zhang and Zhou [58] used the CNN model trained by the ImageNet dataset in the source domain, and by converting time-series signals to two-dimensional grayscale images, the CNN trained by image classification was adopted for fault classification of CWRU bearing data in the target domain.

The remaining transfer domains are collected by the “other” term in Figure 7. This includes various ideas, and three examples are detailed below. Since it is difficult to de-noise the data generated by seawater hydraulic pumps operating under harsh conditions, Miao et al. [59] used oil pump data as the auxiliary data, and by transferring diagnostic knowledge to a seawater hydraulic pump as the target data, the accuracy was improved by 30.5% compared with conventional machine learning methods. Also demonstrating the capacity of TL in dealing with noise, Fan et al. [60] pretrained a CNN on normal CWRU bearing data and fine-tuned the CNN on the same dataset with the addition of Gaussian white noise. A 96.67% diagnostic accuracy was achieved under strong noise at 10 dB signal to noise ratio [60]. The third example was research by Chen et al. [61], who applied TL to deal with missing data due to multi-rate sampling. Although sensors at different sampling rates produce only a few structurally incomplete samples, there are abundant data that are structurally incomplete. To make use of the abundant data that are structurally incomplete, Chen et al. [61] pretrained deep neural networks (DNNs) on the structurally incomplete data and transferred the parameters to boost the diagnostic accuracy of the final DNN trained with sparse structurally complete data.

4. Transfer Learning Research in Aerospace Fault Diagnosis

A thorough review of all existing literature concerning the application of TL to fault diagnosis in the aerospace sector was carried out. The key outcomes are summarised in Table 5.

4.1. Aero-Engines

For research on aero-engines, four studies focusing on engine gas path diagnosis from different transfer problem settings were identified. Zhao and Chen [62] considered the transfer between nominal state data (source dataset) and degraded state data (target dataset), and their work aimed to design an accurate diagnostic model based on very little target domain training data. By developing a turbofan engine simulation based on mathematical models, and using it to generate gas path data, they tested two extreme learning machine (ELM) based TL methods, which yielded better prediction accuracy than ELMs without TL under all noise levels [62]. Aimed at the transfer between various flight conditions, Li et al. [63] tested a unilateral alignment transfer neural network between simulated turbofan data at different operating points, and concluded that their method outperformed other feature-based TL algorithms. In contrast to other studies that mainly used simulation data, Liu [64] validated his TL-based gas path fault diagnostic algorithm against experimental data from China Eastern Airlines, and the diagnostic accuracy reached 95.6% when transferring from four engines, which were used as source data. Furthermore, transfer was also implemented between different aero-engine variants. As Zhou et al. [65] demonstrated, using a Residual-Back Propagation Neural Network (Res-BPNN) as the feature extractor, a deep domain-adaptation module, and a regression module, the model achieved diagnostic knowledge transfer between CFM56-5B2 and CFM56-7B26 datasets.

4.2. Gas Turbines

Owing to their similarity to aero-engines, gas turbines used for power generation are also an application field of interest. This sector has received attention from researchers interested in implementing TL to improve diagnostic accuracy. For instance, Yang et al. [66] transferred a CNN model trained on gas turbines with abundant labelled data to gas turbines with little labelled data and used the little labelled data to fine-tune the pretrained CNN model, and a final version of the CNN was obtained for the data-poor gas turbine. When the data-rich turbine generated 100% labelled data and the data-poor turbine generated 20% labelled data, the proposed method produced 98.37% and 98.92% diagnostic accuracy for GE9FA and Siemens V64.3 gas turbines, respectively [66]. The transfer scenario between different types of gas turbines was also discussed, and a 98.68% prediction accuracy was observed when the model was transferred from the GE9FA gas turbine with 100% labelled data to the Siemens V64.3 gas turbine with 20% labelled data [66]. In addition, following a similar process for transferring diagnostic knowledge from data-rich gas turbines to data-poor gas turbines, Bai et al. [67] investigated how this transfer approach benefitted gas turbine combustion chamber fault detection. The data-rich and data-poor gas turbines were set to be of different types, with the Taurus 70 gas turbine as the source domain, containing abundant faulty cases, and the Titan 130 gas turbine as the target domain, with few faulty cases [67]. The proposed CNN-based TL method produced a higher diagnostic accuracy (95.02%) for the data-poor gas turbine compared to directly mixing the data from data-rich and data-poor gas turbines for training and not conducting the transfer, which gave 91.19% accuracy by the best performing option among all algorithms tested [67]. Another aspect of gas turbines studied was gas turbine rotors. Liu et al. [68] installed vibration sensors on several commercial gas turbines and developed a TL-based rotor fault diagnostic algorithm to classify the rotors as in “normal state”, “air flow excited state”, “imbalance state” and “misalignment state”. The diagnostic accuracy reached 96.45% when diagnostic knowledge was transferred between data under different working conditions, and 95.13% when diagnostic knowledge was transferred between different gas turbines of the same type.

4.3. Sensors

The fault diagnosis of aerospace-related sensors has also received considerable attention. Gao et al. [69] employed a deep transfer learning algorithm based on representation adaptation to transfer a CNN model trained on offline samples to online samples of micro-electromechanical systems (MEMS) inertial sensors on unmanned aerial vehicles (UAVs). For spacecraft attitude determination and control systems (ADCS), He et al. [70] attempted to address the problem of the lack of faulty spacecraft samples. Selecting data generated from a digital simulation platform as the source dataset and data from a semi-physical experiment of a triaxial air bearing table as the target dataset, their TL method demonstrated a higher overall diagnostic efficiency than non-transfer methods. Experimenting on real mission-scale equipment, Mansell and Spencer [71] trained a diagnostic model consisting of a long short-term memory (LSTM) neural network on ADCS fault simulation data and transferred the model to the LightSail 2 mission data. As a major achievement, their TL approach discovered magnetometer glitches that were “completely unknown to the flight team before their discovery and diagnosis by the LSTM network” [71]. Moreover, the transfer scenario between different position profiles and the output directions of the sensors is discussed. Electro-Mechanical Actuators (EMAs) face a major challenge in their condition monitoring owing to their sensors [72]. Specifically, Siahpour et al. [72] expressed concerns that existing algorithms could not identify changes in the sensor location, directions, or characteristics when the actuator fails or in case of a sensor replacement. To mitigate the risk of degrading EMA diagnosis accuracy, Siahpour et al. [72] developed a deep transfer learning method to transfer diagnostic knowledge between data with three sensor position profiles (trapezoidal, triangular, and sine sweep) and three output directions (direction of actuator motion, vertical, and horizontal). Their cross-sensor fault diagnostic problems achieved over 95% overall diagnostic accuracy.

4.4. Structural Components

Another aerospace topic where TL helps in fault diagnosis is with structural components, such as fuselage damage identification and localisation. In this area, TL mainly helps by facilitating image recognition tasks. For inclusion defect detection of aeronautic composite material (ACM), since obtaining sufficient X-ray images showing inclusion defects of this high-quality product is difficult and expensive, Gong et al. [73] leveraged the abundant labelled samples from a welding database. Specifically, having observed a similar appearance of the inclusion defect by group “welds” and the inclusion defects in ACM imaging, Gong et al. [73] trained a deep network on 2300 samples from the GDXray database and transferred it to the non-destructive testing (NDT) X-ray imaging of ACM containing 40 training samples. The transferred deep network had 96.8% prediction accuracy. For the damage localisation of aircraft wings, since post-repair data is usually unlabelled, Gardner et al. [74] considered the need for TL from labelled pre-repair data to unlabelled post-repair data. By implementing domain adaptation by JDA, their final method produced a 96.4% prediction accuracy for the target domain test data [74]. The authors then conducted further TL research on aircraft wings, and in the work by Gardner et al. [75], the transfer scenario between two aircraft wings of different structures was studied – from the labelled Gnat aircraft wing to the unlabelled Piper Tomahawk aircraft wing dataset. In detail, an abstract graphical representation (attributed graph) was constructed for each wing in the beginning, and then by identifying the largest common substructure from the attributed graphs, “the most appropriate subset of label combinations from Gnat that can be mapped into the Piper Tomahawk and produce positive transfer” [75]. A 100% damage localisation accuracy was reported for the target domain wing following a domain adaptation TL method [75]. Furthermore, TL was also applied to transfer damage detection knowledge between different tailplanes. Bull et al. [76] applied a feature-based TL method, Transfer Component Analysis (TCA), to match the normal condition data from three tailplanes of different variants of the Piper PA-28 aircraft, which increased the true positive rate from 13% from the conventional method to 100% from the TCA method.

4.5. Other Aerospace Topics

Other TL studies include those on aircraft fuel pumps, quadrotors, and commercial aircraft flight data. For centrifugal aircraft fuel pumps, Qiu et al. [39] collected data from an old model and a new model of similar pumps on a test bench and showed that the instance-based TL method, TrAdaboost, significantly outperformed other methods when only sparse target domain data of the new pump was available. In a quadrotor unmanned aerial vehicle (UAV) study, Liu et al. [77] focused on detecting UAV propeller failures based on audio data and used CNN-based TL to transfer diagnostic knowledge between two UAV quadrotors of different models and propeller diameters. The CNN-based TL method produced a 91.82% prediction accuracy compared to a 55.00% accuracy using a non-TL method [77]. Regarding the flight data of commercial aircraft, Xiong et al. [78] used deep transfer learning with a recurrent neural network to transfer between ground taxiing data and stable flight data, focusing on the task of predicting the x-axis vibration acceleration at the next moment in time as an abnormality detection basis. A 30% prediction improvement was reported after applying the TL approach [78].

5. Limitation in Existing Research and Future Progress

The application of TL in fault diagnosis is a burgeoning field of research. However, through this literature review, existing research lacks diversity in the specific application and the domains of transfer. Future progress in applying TL to fault diagnosis and how it can facilitate CBM are suggested in this section.

5.1. The Research Gap in Application and Domains of Transfer

Section 3.2 concludes that bearings and gearboxes dominate the application choices, which is largely due to the existence of high-quality open-access datasets. This study supplements the similar comments made in 2019 by Zheng et al. [14], in a way suggesting that the lack of open-access data in other engineering applications might have impeded the wider application of TL. To explore an opportunity to widen the application in aerospace fields, Section 4 examines all TL research in the aerospace sector and discovered examples in aero-engines, gas turbines, sensors, actuators, structural components, fuel pumps, UAVs, and flight data. Structural components and flight data were considered as examples in structural health monitoring and motion tracing, and the rest were considered as examples of component-level fault diagnosis, but there was no application in system-level research. Compared to component-level research, the major difference is the component interactions in system-level analysis, and it requires research attention to understand how TL may help transfer knowledge of system-level fault diagnosis under the influence of component interactions. Future work by the authors will be based on the first research gap in applying TL to system- or subsystem-level fault diagnosis in aerospace systems.

The second research gap was inspired by the findings on the transfer domain choice. Section 3.3 reported that over three-quarters of the research work analysed focused on diagnostic knowledge transfer between the same machine under different working conditions or similar machines under different representations. While almost all existing TL research on fault diagnosis requires the existence of low-level similarities in similar data structures and physical parameters, future opportunities exist by finding ways to expand TL research to more dissimilar source and target domains and, ultimately, to address high-level transfer problems unbounded by the requirement of low-level similarities.

The lack of diversity in the specific application and domains of transfer in the existing literature is a good indication of the potential of developing high-level TL for fault diagnosis. With high-level TL solutions in fault diagnosis, the application of relevant research could see wider diversity, because the open-access dataset could be leveraged to design fault diagnosis solutions for other applications as well.

5.2. Future Progress

Section 2 revealed, through the history of TL, that the entire scope of TL should include both low-level transfer and high-level transfer solutions. Because high-level transfer has not been extensively researched in fault diagnosis, successful implementation of the idea would greatly expand the capability of TL. In fact, a few studies have demonstrated similar ideas and justified the feasibility of implementing TL between more dissimilar domains and seemingly unsimilar domains.

Regarding early attempts to expand TL research in fault diagnosis to more dissimilar source and target domains, two studies have been identified. Li et al. [79] considered the incorporation of new faults in the target domain that are unseen in the source domain. The problem was tackled by adding a classifier specialised in distinguishing new faults in the target domain from known faults in the source domain. Taking this thinking further, Deng et al. [80] explored how TL can be implemented in partial transfer scenarios in which the labels in the source and target domains are not identical, considering that “the partial transfer scenario is more common for industrial applications” [80]. A double-layer attention based generative adversarial network (DA-GAN) was built to address the partial transfer problem by guiding the model to identify which part of the data should be processed or ignored before domain adaptation, thereby promoting positive transfer while alleviating negative transfer [80]. The proposed model demonstrated the ability to handle transfer on different machines (TDM) and partial transfers with different faults. The model can bear five differences simultaneously: 1) different bearing types, e.g., ball bearing vs. rolling bearing, 2) different fault characteristics, e.g., artificial damage vs. early-stage degradation, 3) different damage modes, e.g., plastic deformation vs. inner race fault, 4) different machines, e.g., bearing datasets from different labs, and 5) different working conditions, e.g., 1500 rpm vs. 2400 rpm [80]. In these transfer scenarios, the proposed method resulted in 89.4% overall prediction accuracy, which is higher than that of other GAN-based methods (71.4% and 79.4%), and significantly higher than that of a feature-based method (39.6%) because of negative transfer.

Regarding high-level transfer problems, only one work on TL for fault diagnosis has been found, which is the research done by Liu et al. [81]. In their work, the target domain data was taken from an oil-gas treatment station, which contained very few samples in each fault state, and the source domain data was taken from a Tennessee Eastman (TE) process, which contained abundant samples in each fault state. Their work illustrated a high-level transfer problem, because the oil-gas treatment process in the target domain did not share any identical components or variables with the TE process in the source domain, and the components of each system did not share the same configuration. The adopted TL approach was to first pretrain a Residual Neural Network (ResNet) model on the source domain TE process data and then fine-tune the ResNet hyperparameters using the target domain oil-gas treatment station data. This method generated a diagnostic accuracy of 97.00% compared with 90.67% without the TL process, which demonstrates the possibility of improving the diagnostic accuracy in the target domain with a small amount of data by exposing the model to more training data in a dissimilar source domain via TL. To account for this result, Liu et al. [81] applied an explainable AI technique to visualise the attention regions of the ResNet network, i.e., what the network focuses on in terms of variables and time domains, with and without TL. They concluded that the improvement in the diagnostic accuracy after TL was due to the effect of TL on concentrating the attention regions of the ResNet network in the target domain. Furthermore, the concentrated attention regions in the target domain also share the highest similarity with the attention regions in the source domain. Hence, the reason for the improvement in diagnostic accuracy by TL can be described as the TL process selecting the most similar regions of the source and target domains and concentrating the focus of the diagnostic network on the selected region in the target domain. However, despite Liu et al. [81] demonstrating the successful implementation of TL on a high-level transfer problem, other high-level transfer examples are absent in fault diagnosis, revealing a research gap.

As a reflection of the work by Liu et al. [81], firstly, one crucial condition for achieving the knowledge transfer between the two dissimilar systems is that the diagnostic model used, ResNet, is capable of extracting features from datasets of different complexity. In their study, the ResNet model was pretrained on source domain data with 52 × 10 dimensions and fine-tuned on target domain data with 36 × 10 dimensions. This serves to inspire future work on the high-level transfer problem along the lines that the model used should be capable of handling input data with different complexities, which is inevitably associated with dissimilar source and target domains.

Secondly, after discovering that TL concentrated the attention region on the part of the target domain data with the highest similarity to that in the source domain, no physical explanation was found to accompany this finding. Had such explanations been provided, for example, by pointing out how the seemingly unrelated components in the two systems have similar effects on the systems or interact with the rest of the systems in a similar way, the clarity of the method and explainability of the results would have been further improved. Unlike low-level transfer scenarios where identical components are expected in the source domain and target domain systems, high-level transfer problems rely on the similarity between seemingly unrelated components of different systems, hence providing a physical explanation of why the seemingly unrelated components should be considered similar could be crucial when validating the result. From another perspective, finding similarities between seemingly unrelated components in different systems based on physical insight may also be the key to implementing TL in such scenarios. The method mentioned in Section 2.4, transfer learning by structural analogy, could be adapted to fault diagnosis applications as a high-level TL method following this approach and could provide ideas for future research.

From the above, two major directions for developing future solutions for high-level transfer problems are: 1) developing TL models that can handle source and target domain data with different complexities, and 2) establishing similarity between seemingly unrelated components in source and target domain systems with the help of physical insight. The authors of this work believe that a combination of the two approaches would result in an accurate and credible solution to high-level transfer problems.

5.3. Benefit and Potential for Aerospace Condition Based Maintenance

Currently, TL has demonstrated its ability to improve the diagnostic accuracy of fault diagnosis models when dealing with imperfect data in the target domain, such as scarce data, lack of labels, lack of faulty cases, or different distributions from the source domain. Hence, applying TL to the diagnostic model in aerospace CBM would improve its robustness and accuracy when encountering similar problems with real-case data. This advantage of TL would not only improve the diagnosis model, but would also help alleviate the challenges faced by CBM.

The first common challenge for aerospace CBM that TL can help alleviate is scarce failure data [82]. It can be dealt with using TL methods that leverage knowledge from a source domain where abundant failure data exists, for example, for a similar machine or under test conditions. This is particularly useful for aerospace CBM. Because aerospace vehicles are usually expensive and reliable, generating failure data for the safety-critical components that very rarely fail could only be practically and economically done by lab experiment [83]. The second challenge is, as CBM in aerospace faces strict legislation challenges before implementation, the improved accuracy and robustness against real-scenario problems brought about by applying TL would contribute to satisfying the strict legislative requirements. Currently, aviation legislators have restricted CBM applications to non-critical component, which means although CBM is a well-established concept in academic research, its application in aviation industry is still premature and unable to relieve the burdens brought by most of the interval-based tasks [83]. For aerospace CBM to relieve such burden and bring cost benefit by its real-world application, the CBM models must be reliable enough to be certified, hence the enhanced reliability brought by incorporating TL to fault diagnosis models could be critical to the implementation of aerospace CBM.

In the future, if the application of high-level TL in fault diagnosis is successfully developed, further benefits can be achieved, which should be considered as the potential of TL for aerospace CBM. The potential is the ability of high-level TL to reduce the costs associated with data generation. Although TL is designed to handle imperfect data in the target domain, a source domain with an adequate quantity of labelled data and proportion of failure data is still required. The collection of such high-quality datasets is expensive in most industries, including the aerospace industry. With high-level TL, knowledge may be leveraged from dissimilar domains, such as those with available high-quality datasets in another industry (e.g., leveraging bearing datasets to diagnose faults in automobile timing belts) or from a different machine in the same industry (e.g., leveraging a hydraulic system dataset for pneumatic system fault diagnosis). This would greatly improve the efficiency of using available high-quality datasets and potentially reduce the effort and expenses of generating high-quality datasets for new applications. As a result, it potentially encourages further research on aerospace CBM.

6. Summary and Conclusion

Aiming to explore the greatest extend of how TL facilitates fault diagnosis and aerospace CBM, this literature review studied beyond the existing TL methods applied to fault diagnosis. The investigation led to two major findings:

1.

The whole scope of TL has been explored by studying the history of TL and comparing TL with similar methods, revealing:

i) Learning from the history of TL, the scope of TL should include both low- and high-level transfer scenarios. TL algorithms exist for both scenarios, although high-level transfer scenarios have received little attention in their application to fault diagnosis.

ii) By comparing TL with other methods that leverage previous knowledge, analogy has been identified as a powerful tool to leverage previous knowledge from seemingly unrelated domains, thus pointing to a way to develop TL algorithms for high-level transfer scenarios.

2.

Research gaps have been identified by reviewing the existing research. These include:

i) The paucity of applications beyond bearings and gearboxes. In aerospace fault diagnosis, little is known about how TL contributes to system- or subsystem-level fault diagnosis.

ii) The lack of TL research between dissimilar source and target domains.

In conclusion, this paper explores TL and its application to fault diagnosis and its potential in aerospace CBM. The history of TL reveals that its scope exceeds low-level transfer scenarios, which corrects the common belief that TL relies only on low-level similarity and highlights the potential for developing TL between dissimilar domains. The research gap found in applying TL to dissimilar source and target domains also suggests that developing high-level TL methods for fault diagnosis would expand the boundary of research interest. A general approach for high-level problems, such as the one proposed, is to combine a model that handles data with different complexities and physical insight to account for the similarity between seemingly unrelated components. Upon successful implementation, high-level TL has the potential to greatly improve the efficiency of leveraging high-quality data from diverse fields for training fault diagnosis models and encouraging more research effort in aerospace CBM.