Preprint
Article
A Semantic Framework to Detect Problems in Activities of Daily Living Monitored through Smart Home Sensors
Altmetrics
Downloads
66
Views
33
Comments
0
A peer-reviewed article of this preprint also exists.
This version is not peer-reviewed
Submitted:
18 December 2023
Posted:
18 December 2023
You are already at the latest version
Alerts
Abstract
Activities of daily living (ADLs) are fundamental routine tasks that the majority of physically and mentally healthy people can independently execute. In this paper, we present a Semantic Framework for detecting problems in ADLs execution, monitored through Smart Home sensors. In the context of this work, we conducted a pilot study, gathering raw data from various sensors and devices installed in a smart home environment. The proposed Framework combines multiple Semantic Web technologies (i.e. ontology, RDF, triplestore) to handle and transform these raw data into meaningful representations forming a Knowledge Graph. Subsequently, SPARQL queries are used to define and construct explicit rules to detect problematic behaviours in ADLs execution, a procedure that leads to generating new implicit knowledge. Finally, all available results are visualized in a clinician dashboard. The proposed framework can monitor the deterioration of ADLs performance for people in the spectrum of dementia, by offering a comprehensive way for clinicians to describe problematic behaviors in the every-day life of an individual.
Keywords:
Subject: Computer Science and Mathematics - Artificial Intelligence and Machine Learning
1. Introduction
Activities of daily living (ADLs) are essential and routine tasks that most healthy individuals can perform without assistance. ADLs are divided into two major categories, Basic ADLs (BADLs) and Instrumental ADLs (IADLs) [1]. BADLs, also known as physical ADLs, are a category of daily activities required of each individual to manage their basic physical needs. These activities include self feeding, dressing, toileting, ambulating and any other physical action that is related to personal hygiene. On the other hand, IADLs are more complex activities that people perform in order to be functional and independent within society. Managing finances and medications, food preparation, housekeeping and laundry are only some elementary examples of this category. Based on the literature, cognitive and functional decline becomes tractable first in the IADLs due to their complexity [2]. Most people who show early signs of cognitive decline tend to develop an abnormal behaviour that makes them deviate from the correct execution of ADLs. This weakness exacerbates gradually and it is a common phenomenon that patients who exhibit such cognitive decline do not perceive the occurred deviation [1]. The inability to accomplish essential activities of daily living may be an indicator of an upcoming health problem [1][3]. Investigating the accuracy when performing daily activities can contribute to tracking the health deterioration of people with chronic diseases such as dementia or multiple sclerosis. Engaging in monitoring ADLs is motivated by the need to have an objective way to assess whether there is a diminishing ability to perform certain tasks, compared to or as a ground truth measure to an individual’s self-reporting. That process can offer meaningful insights to clinicians when assessing ADLs to enhance patient care and management.
The ADL’s assessment process could be classified into several stages, in which clinicians look for different types of problems. Any observations or conditions during the execution of ADLs that deviate from a reasonable and normal situation can be considered problems. The recurrent existence of abnormal behaviours, like the need of excessive time to complete an ADL and difficulty in comprehending instructions on the execution of a specific ADL when those are provided, are only some of the different factors to be evaluated by clinicians. The complexity of monitoring and assessing ADLs imposes the need to facilitate this procedure through the use of sensor technology in a semantic framework.
There is a variety of affordable and accurate devices in the market that can be used to monitor patients by collecting their daily life data [4]. Smartphone apps and wearable sensor devices, such as smart watches, have been widely used to monitor heart rate, steps and sleep stages [5,6]. Another category of these devices are sensors that capture presence and motion in a room. Consequently, there is a significant need for the establishment of a framework that would integrate, analyze and visualize all available data in an efficient way. Such a framework would support the evaluation of the received data by expert clinicians allowing them to draw useful conclusions. The introduction of Semantic Web (Web 3.0) [7] and its constantly developing technologies such as Ontologies, RDF (Resource Description Framework) [8], SPARQL [9], reasoning engines and triplestores provide us with all the essential tools to construct a framework that could implement all the processes mentioned above.
In this paper, we propose a Semantic Framework for detecting problems in ADLs monitored through Smart Home sensors. This framework consists of different components that are responsible for monitoring and modeling ADLs, executing specific rules to detect problematic behaviours in ADLs performance and visualizing the results. In this way, we can detect problems related to cognitive decline and observe the scale of patient deterioration within a period of time. Initially, we gather all raw data provided from sensors and devices established in a smart home environment. Next, a specially adapted ontology is developed to model and represent all the related information of this environment. This includes information and data acquisition from patients, sensors, devices and rooms of the smart home. Then, all collected data are converted to RDF triples and stored in a semantic database constituting a Knowledge Graph. Furthermore, by using SPARQL queries we define and construct explicit rules to detect problems, a procedure that leads to generating new implicit knowledge. Last but not least, all available results are visualized in a clinician web dashboard.
The rest of the paper is structured as follows: Section 2 reviews the related work regarding the use of Semantic Web technologies for e-health solutions. Section 3 describes the architecture of the proposed Framework, illustrating all the background processes. Section 4 presents a case scenario providing visualized results. Finally, Section 5 draws our conclusions and describes our intentions for future work.
2. Related Work
The proposed Framework is positioned in a combination of multiple research fields including the Semantic Web Technologies, ADLs recognition and monitoring through Smart Home sensors. In this section, we examine previous related studies that incorporate these technologies and methods to provide e-health solutions.
Wearable sensors are an affordable and effective solution in order to collect daily life data. Heart rate, steps, and sleep data have been used as the data source in semantic frameworks aiming to health problem detection in patients suffering from chronic diseases [10,11].
[10] proposes an assistive technology system based on wearable trackers to support the care of people living with Multiple Sclerosis. This system collects unanimously daily life data through wearable sensors and stores them in an ontology-based Knowledge Graph. It aims to detect problems related to sleep, movement or heart rate for self-awareness purposes or for clinician experts to monitor their patient’s progress. In this process, Semantic Web technologies such as reasoning engines, SPARQL rules and triplestores are incorporated.
In [11], a rule-based Framework for the detection of health-related problems of people in the spectrum of dementia is presented. This is a complete Semantic System that includes a component for IoT data collection from wearable sensors and a Knowledge Ingestion process that translates received data into RDF representations. It also proposes a novel OWL ontology as well as a set of SPARQL Inferencing Notation (SPIN) rules to infer problems from collected data. It aims to provide clinicians with information about people with dementia to support decision making for further care management. Therefore, the system provides insights by visualizing measurements and results through a clinician platform.
Moreover, ontologies and reasoners have been used for health group classification purposes [12]. In this study, the proposed Semantic System supports the supervision of people living with chronic diseases and categorizes them in groups depending on their health status. It is worth to mention, that this paper presents a modular ontology composed of several external public domain related ontologies.
In previous papers, a set of rules were applied to ontology based Knowledge Graphs in order to derive implicit statements declaring a problematic health situation. However, the combination of an ontology and a query language that implements rules has been also utilized in order to retrieve explicit information about the environment that is modeled. [13] implements a rule-based approach system to inform deafblind individuals about their surroundings and support them in increasing their spatial awareness. In this case, the ontology models all objects involved in the users’ environment, and a set of rules is applied on it so as to respond to their queries regarding their surroundings.
A complete rule based system that is capable to detect the type of a disease and recommend suitable treatments is presented in [14]. This system receives real data from people suffering from blood immune thrombocytopenia disease. The proposed ontology represents the knowledge of this medical domain, while a set of rules alongside with a reasoning engine detects the disease type based on real data received and suggests a suitable treatment.
Semantic Web technologies have been also applied in systems aiming to validate diagnoses of medical experts in complex domains such as cancerous tumours [15]. This paper presents an ontology based solution to model the international rules for various types of cancer. In this way, a defined set of rules and axioms can be executed through a reasoner so as to validate the diagnosis of medical experts based on the clinical results of each case.
However, these technologies can also indirectly contribute to e-health solutions by providing medical experts with different aspects of patient’s health status. More specifically, [16] provides an ontology based solution for ADLs recognition in a Smart Home environment. This study introduces a System that is able to monitor ADL patterns by using a novel OWL ontology and a reasoning engine. The proposed ontology represents the environment and the devices in a Smart Home, while the reasoner is responsible to discover an ADL. This system uses a rule based reasoner that infers an ADL when explicit statements are inserted in the Knowledge Graph and they satisfy the corresponding declared rule.
3. Semantic Framework
This section presents the proposed Semantic Framework for monitoring ADLs in a smart home environment with the aim of detecting problems related to their execution performance. An overview of the architecture of this Framework is shown in Figure 1. The first stage of the pipeline is the IoT data collection from sensors in a Smart Home setting to monitor ADLs. IoT data collection is the process of collecting data from sensors connected to the Internet Of Things (IoT), which is the interconnected network of physical devices, capable to exchange large amounts of data [17]. Next stage includes the pre-processing of the raw data and their conversion to RDF triples. The combination of RDF data and the implemented ontology that defines their structure constitute the Knowledge Graph. Afterwards, predefined rules in SPARQL format are applied to the Knowledge Graph aiming to detect problems in ADLs execution. Finally, the visualization of the results in a clinician dashboard concludes this whole procedure.
3.1. Monitoring of ADLs in a Smart Home environment
For the purposes of the present work, the CERTH-ITI Smart Home premises were used [18]. The Smart Home is a prototype and novel technologies demonstration infrastructure resembling a real domestic building where occupants can experience actual living scenarios. Alongside their accommodation, they can explore various innovating smart IoT-based technologies provided with Energy, Health, Big Data, Robotics and Artificial Intelligence (AI) services. In the context of a pilot study, 40 participants with varying cognitive status in the spectrum of dementia were invited, for a daily visit or staying overnight in the Smart Home. Prior to their enrollment, a neuropsychiatrist with expertise in dementia established participants’ diagnoses (Healthy Controls - HC, participants with Subjective Cognitive Decline - SCD and participants with Mild Cognitive Impairment - MCI). Diagnoses were set taking into account participants’ medical history, structural magnetic imaging (MRI) and a detailed neuropsychological evaluation. The MCI group fulfilled the Petersen criteria [19], while the SCD group met IWG-2 Guidelines [20] as well as the SCD-I Working Group instructions [21]. They were instructed to execute a protocol including three ADL activities (namely, Task 1 - Hot Meal Preparation, Task 2 - Hot Beverage Preparation and Task 3 - Cold Meal Preparation). The protocol was accompanied by a detailed, step-by-step description for each Task. For example, in Task 3, participants were instructed to get a plate, bread, cheese and turkey from appropriately labelled cabinets and the fridge, and prepare their sandwich. Afterwards participants were instructed to turn on the appliance, place the sandwich on the toaster and remove it once ready. In order to monitor participants’ ADLs we installed a set of Smart Home devices, including multiple types of sensors that are able to perform functions such as monitoring, controlling or alerting. For our research purposes we equipped the Smart Home environment with wall plugs, motion, door and flood sensors. Wall plugs are consumption monitoring devices in which all other electrical appliances are plugged. Motion sensors are responsible for capturing presence or movement in a room, door sensors detect the opening and closing of doors, drawers and cabinets, flood sensors detect any water leaks or flooding. For safety reasons, a panic button sensor was also installed so as to provide a quick way to trigger an alarm in case of emergency. It is important to mention that there are many solutions in the market for smart home devices that can be used in similar studies. In our study, we have selected the "Fibaro" solution [22] which provides its Home Center software allowing users to control and collect data from all devices through a single interface. All above devices are listed in Table 1.
Fibaro’s raw data consists of two time series, one for "Signal" and one for "Consumption". Signal is generated by the motion, door, flood and panic sensors, while Consumption is generated by wall plugs. Both Signal and Consumption time series represent changes in the values that occurred at a specific point in time. Signal, in contrast to Consumption, takes into account the previously reported values in addition to the current values.
A Signal data object for a device (motion, door, flood, panic) is a pair of Boolean values that represent its change of state. If a data object has a new value of 1 and an old value of 0, it generates a new Signal and the timestamp of the data object acts as the starting point, while the next data object with opposite values represents the end of this Signal.
Consumption data, on the other hand, are Float values of wattage consumption. Most household appliances consume electricity even when they are idle. We have set a minimum threshold of 5 watts to mark the beginning and the end of a consumption event. This is an empirical threshold we set after careful study and analysis of the generated data. If the wattage of the data object exceeds the threshold, we assume that the appliance is switched on, and if the wattage is below the threshold, the appliance is considered "not in use" (i.e. idle).
All sensor data are transferred to our databases through APIs, in order to be gathered and processed. In that way, the data are grouped together and can be further processed by conducting validity checks in order to identify and normalize false negative signals (i.e. Cabinet didn’t fully close due to brakes but sensor failed to recognize it). This procedure of validity checking ensures the reliability of the collected data which is necessary for their analysis.
3.2. Sensor Data Knowledge Ingestion
The raw data described in the previous section are in a complicated form making them incomprehensible and confusing. Therefore, further analysis is necessary to group them in clusters depending on their timestamps and their source devices. Each cluster consists of raw data that indicate a single event inside the Smart Home environment. For instance, if a motion sensor in the kitchen detects presence in a specific timestamp while a door sensor detects the opening of a kitchen drawer at the same or subsequent time, then it is reasonable to conjecture that someone was in the kitchen and opened the drawer at that time. By manually inspecting and clustering all raw data into specific meaningful events, we can proceed to the next step of data processing which is the recognition of an entire activity. Figure 2 is a chart that presents all events derived from raw data in a specific period of time for a specific patient. The names of the events are shown in the y-axis while the x-axis presents the duration of each one. More specifically, Figure 2 shows a sporadic use of kitchen drawers and cabinets, an almost continuous human presence in the kitchen and the simultaneous operation of the electric cooker for several minutes.
Following the same procedure, we can create some larger clusters that consist of correlated events corresponding to the room of the Smart Home and the time series that occurred. For instance, grouping events that occurred in the kitchen, such as motion detection in space, opening and closing of drawers and operation of the electric cooker in a specific period of time composes a sequence of events for a hot meal preparation activity. This activity is depicted on Figure 3 with the horizontal light blue line. Below that line some of the previously mentioned events that reveal the cooking activity are included. Similarly, we can identify all different ADLs that patients were instructed to perform based on the protocol, during their accommodation in the Smart Home.
In order to test the effectiveness and reliability of the sensors and the developed platform, the activities collected were compared to ground truth information. For this, during the study, researchers gathered information through free-text notes on participants’ activities, serving as ground truth. In Figure 4 activities defined by the platform are compared to the ground truth information collected. Discrepancies are noted due to power/internet outages (missing “Meal Preparation” and “Beverage Preparation” activities) or sensor connectivity issues (two missing “Cold Meal Preparation” activities) in the Smart Home.
Having completed all the necessary pre-processessing of the raw data, it is imperative to translate our unstructured clustered data into meaningful interlinked graph representations. In this way, our data are linked through meaningful relationships that are humanly understandable, establishing a knowledge base capable to infer complicated implicit relations. This procedure is called Knowledge Ingestion (KI) and is achieved through the conversion of our data to RDF statements. An RDF Statement is a triplet that links a subject with an object via a predicate. The subject and the object elements represent the two entities that are linked together and the predicate element strictly defines their relationship. Multiple RDF statements create a larger linked graph that depicts all possible entities and relations that exist in each case we try to model. However, before converting our data to RDF triplets we have to define a precise schema that all of them should follow.
3.3. Ontology-based knowledge representation
In the Semantic Framework introduced in this paper it was essential to construct an Ontology representing all entities involved in our research. Indicatively, the Events and Activities that were generated after the pre-processing of the raw data, the Smart Home sensors as well as the patients that participated in that study constitute our primary entities. Therefore, an ontology named "ADL Recognition Ontology" was designed and developed in OWL 2 language with the usage of the open-source software Protégé 5.5.0. OWL (Web Ontology Language) is an ontology language for the Semantic Web with well defined meaning that can model and represent complex knowledge [23]. In addition, OWL offers more modeling primitives than RDF Schema and has quite clean formal semantics, which assign an unambiguous meaning to its logical statements. Formal semantics precisely describe the meaning of a language which means that semantics can not be interpreted in more than one way. Also, formal semantics are a prerequisite for automatic reasoning support which is of primary importance, since it allows us to check the correctness of our ontology with respect to it’s consistency and the existence of unintended relations between classes. Moreover, a reasoner is able to infer logical consequences combining explicit knowledge with the formal semantics of the ontology’s language. In other words, a reasoner generates new relations between the entities that are classified according to the ontology. This mechanism helps us avoid declaring explicitly a vast amount of necessary statements, which we would inevitably have to do if we had a classic relational database schema instead of an ontology.
Figure 5 shows the OWL classes constituting the "ADL Recognition Ontology". This ontology is an adapted and extended version of the Semantic Sensor Network (SSN) ontology and the Sensor, Observation, Sample, and Actuator (SOSA) ontology [24]. The combination of these two well-established ontologies is intended to provide a comprehensive framework for describing sensors and their corresponding observations. This combined ontology has been used as a fundamental core in other applications [25]. The utilization of these well established representation models ensures the interoperability of our framework. One of our primary goals was to develop an extensible ontology in order to be flexible to support more scenarios in the future by adding further subclasses to each class category. For instance, the "Sensor" class can represent any other sensors, that may be used in a Smart Home environment. Likewise, this ontology could be enriched with additional "Activity" subclasses depending on the cases being studied. Furthermore, the structure of the "Problem" class gives us the opportunity to add more problem types that may have occurred during the execution of an activity. In our ontology The "Problem" class has four subclasses, each modeling a specific problem that may be observed by assessing an Activity execution. As mentioned in previous sections, these problem types are related to deviations from the protocol that Smart Home participants were instructed to follow. After consulting clinicians, we ended up modeling four problem types with the corresponding "Problem" subclasses. These problem types are divergence from the protocol of an ADL, observation of extra or missing steps in each ADL protocol and excessively long duration of an Activity until it is completed. Generally, the "Observation" class represents an event that was monitored by a "Sensor" instance. Consequently, the "Activity" class represents an activity that was synthesized by one or more "Observation" instances. Each "Observation" is linked to a specific "FeatureOfInterest" that describes what is being observed or measured by a sensor. In the "ADL Recognition Ontology", this class is exteneded with four subclasses that specify the type of the sensor that generated each observation. The "ObservableProperty" class is associated with "Observation" class and represents the attributes captured by a sensor. For example, in case of a sensor capturing a presence in a room, the timestamp and the switch from 0 to 1 value are the attributes of this observation. The classes "Sensor", "Observation", "FeatureOfInterest" and "ObservableProperty", along with their associated relationships are important components of the SSN/SOSA ontologies. These classes have been expanded with numerous subclasses and linked to new classes to fulfill the representation requirements of our study.
Both "ADL Recognition Ontology" and converted linked data compose our Knowledge Graph which is hosted in an instance of the GraphDB, a semantic database offered by Ontotext [26]. GraphDB is a highly efficient and scalable RDF database that can handle massive loads of triples, queries and OWL inferencing in real time. In addition, it supports all RDF syntaxes for data insertion and retrieval and it also offers the capability to submit all types of SPARQL queries. One major benefit that led us to choose GraphDB as our triplestore is that it can perform semantic inferencing at scale, allowing users to derive new semantic statements from existing facts due to the existence of an integrated reasoner.
Figure 6 visualizes an example of the RDF data in our Knowledge Graph. In this example, the central red node is an instance of the "Activity" class as it is described in the purple node. The four edges with label "consists of" end up in the four different instances of the "Observation" class that constitute the main Activity. In order to comprehend better this graph instance, we can consider the red node as a "Hot Beverage Preparation" activity. The "Observation" objects represent the events that occurred during this activity execution. In this case, these events are the opening of specific kitchen drawers and cabinets, as well as turning on the coffee machine. Last but not least, the yellow node represents the patient that performed this activity. The graph pattern presented in Figure 6 is just one instance of the thousands that are linked together to construct the entire Knowledge Graph.
3.4. Problems detection based on daily activities
The primary aim of the proposed framework is to be able to detect problems related with the execution of ADLs. To achieve this we had to define a set of rules as they are presented in Table 2. These rules define upper or lower limits to filter out activities that deviate from a normal execution. The "correct order" and "correct count" variables that appear in the table refer to the predefined protocol instructions that all patients had to follow. The structure of the rules and the numerical values of the thresholds were decided after consultations with clinicians. By setting these thresholds, our intention is not to categorize patients into health groups, but individually monitor their performance and compare it with their former results. In Semantic Web these rules can be formulated with SPARQL queries. We used the interface of GraphDB Workbench, an admin web application offered by GraphDB, to compose and execute a set of SPARQL CONSTRUCT graph patterns to derive problematic situations expressed in RDF triples and enrich the Knowledge Graph.
The following code blocks show an example of the form and structure of SPARQL rules which are responsible to derive problematic situations in "Hot meal preparation" activities. The first one is a rule for "Too Long Duration" problem. Applying this query, constructs four new RDF triplets that are related with the problem, the patient to whom it is referred, the date that it occurred and the total duration of the problematic activity. In this example, the condition that must be satisfied for the activity is a total duration of more than 2100 seconds (35 minutes).
 |
The second rule is for the “Extra Steps” problem. Similarly, it constructs four new triplets describing the problem, if the total count of steps needed to complete the activity was higher than 20.
 |
The third rule implements the “Missing Steps” problem, following the same logic with the previous rules. The condition that must be satisfied in this rule is the total count of steps needed to complete the activity to be lower than 10.
 
|
The last rule implements the "Divergence from protocol" problem. This rule helps us comprehend the value of utilizing a powerful language such as SPARQL when implementing a more complex query, instead of performing a simple threshold comparison. In this query, we formulate a set of rules that must be strictly satisfied, which refer to the chronological sequence that certain steps were implemented. More specifically, this rule intends to detect the occurrence of any divergence from the protocol that instructs participants to firstly open the food cabinet, then the cooker and lastly the fridge door. Therefore, this query retrieves all activities that consist of these steps and then checks their time sequences by assigning to them a "correct" or "wrong" status. Finally, it filters out all activities having only correct sequences of these three steps and returns the activities that had at least one divergence from the protocol.
 |
4. Evaluation
The Smart Home Study lasted for a few weeks, so each participant could take part in our study only once. Therefore, we could not monitor participants’ ADLs performance for a long term, in order to have enough results to compare them individually. However, for the evaluation of the proposed Semantic Framework we consider the following case scenario, in which we examine the procedure of monitoring ADLs, analyzing the data and detecting problems for two different participants.
The first participant, who was diagnosed with MCI, stayed at the Smart Home on the 1st December of 2021, while the second one, who was categorized in HC group, stayed on the 11th February of 2022. Both participants followed the same ADLs protocol and were monitored during its execution. After the collection and the analysis of the data, we compared the duration results for the execution of the "Hot meal preparation" and "Hot beverage preparation" activities, as they are shown in Table 3. In both activities, a significant longer duration in completion time for Participant 1 was observed, compared to Participant 2. Based on the protocol, the maximum normal duration for the first and second activity was set to 2100 seconds (35 min) and 240 seconds (4 min), respectively. As a result from the activity duration rule (Rule 1), the Framework automatically generates an "Activity takes too long" problem for both activities for Participant 1, while the same rule did not detect any problem for Participant 2.
Another aspect that we measured and evaluated for this case, were the total number of events that each participants needed to complete the "Hot meal preparation" activity. Their performance is presented in Figure 7a and Figure 7b. In these Figures we depict two similar graph instances having a central red node, representing the activity, which is linked with the multiple blue nodes, representing the events occurred during this activity. It is evident from the depicted graphs that Participant 1 took considerably more steps than Participant 2. Based on the protocol, the maximum normal count of events for this activity was set to 20. As a result from the count of events rule (Rule 3), the Framework automatically generates a "Took extra steps" problem for Participant 1, while the same rule did not detect any problem for Participant 2. In both aspects that were examined in this example, our Framework confirmed the initial claim for the participant’s health groups by detecting specific problematic cases only for the MCI participant.
The above analysis can be scaled to all participants who took part in our study. Table 4 shows the percentage of participants from the three groups (HC, SCD, MCI) for whom our framework detected problems regarding the activity "Hot Meal Preparation". A significant difference in "Too Long Duration" problem is observed among the three groups. The activity duration rule (Rule 1) generated a problem for the 18.18% of the HC participants, while almost half of the SCD participants (45.45%) and 62.5% of the MCI were detected with the same problem. The results related to protocol divergences are particularly interesting, given that 81% of both HC and SCD participants exhibited deviations, and all MCI participants did as well. These especially high percentages reveal that despite the specific protocol instructions, most participants executed the activity in a more intuitive and subjectively natural way, due to its simplicity and flexibility. Regarding the count of events rule (Rule 3), the framework detected the "Extra Steps" problem in 12.5% of participants from both the HC and SCD groups, with an increase to 37.5% in the MCI group. It is worth noting that the framework did not detect any "Missing Steps" problems in any of the participants from the three groups. This result may indicate that none of the participants quit the activity without successful completion.
This case scenario shows the ability of our Framework to handle and process data from Smart Home sensors, translate them into meaningful representations of events and ADLs and finally detect problems in the execution of these ADLs. In this way, our Framework can be used as a helpful tool, for monitoring the deterioration of ADLs performance for an individual by collecting more long-term observations for each participant of the study. These results can be used as an alert mechanism warning the clinician, when there is a deterioration of the participant in the performance of executing ADLs.
5. Conclusions - Future Work
In this paper we present our semantic approach to detect problems in ADLs execution, that are monitored through Smart Home sensors. The proposed Framework combines different Semantic Web technologies (i.e. ontology, triplestore, SPARQL) to handle and transform Smart Home sensor data into meaningful representations forming a Knowledge Graph. The main purpose of our Framework is to detect problematic conditions in ADLs execution by applying predefined rules to the generated Knowledge Graph. The main architecture of this framework and its components was analytically presented. In order to validate our system, we have also provided a proof-of-concept case scenario that utilizes real life data from participants who took part in the Smart Home study.
It is of high importance that our Framework offers an easy to comprehend way for clinicians to describe the problematic cases that are observed in a patient. Instead of observing a generic functional and cognitive decline, it can specify the actual problems relying on patients’ data. In addition, it is a modular framework that can easily scale-up with more features, activities and problems while preserving the integrity of its core logic.
As a future work, we plan to enrich the problematic situations that this Framework can detect, by defining more rules evaluating different and more complicated aspects in ADLs performance. Moreover, we aim to adjust it to support input data from other similar IoT sensors that are available in the market. Our intention is for our framework to be adopted by clinicians as an objective measure for detecting the progression of cognitive decline, in an unobtrusive and natural way.
Author Contributions
Conceptualization, S.N.; I.K.; and T.G.S.; methodology, S.N.; T.G.S.; G.G.; V.A. and L.M; software, G.G. and V.A.; validation, V.A. and M.G.; formal analysis, V.A.; M.G.; and I.L.; investigation, G.G.; V.A.; L.M.; M.G.; I.L.; and M.T.; writing—original draft preparation, G.G.; L.M.; writing—review and editing, S.N.; I.K.; and M.T.; visualization, G.G.; supervision, S.N. and I.K.; project administration, S.N. and I.K.; funding acquisition, T.G.S.; S.N.; and I.K.; The RADAR-AD Consortium approved the submission of the present manuscript. All authors have read and agreed to the published version of the manuscript.
Funding
The RADAR-AD project has received funding from the Innovative Medicines Initiative 2 Joint Undertaking under grant agreement No 806999. This Joint Undertaking receives support from the European Union’s Horizon 2020 research and innovation programme and EFPIA and Software AG. This communication reflects the views of the RADAR-AD consortium and neither IMI nor the European Union and EFPIA are liable for any use that may be made of the information contained herein.
Institutional Review Board Statement
The study was carried out in accordance with the Declaration of Helsinki and received approval by the Research Ethics Committee of Center for Research and Technology Hellas (CERTH) (ETH.COM 54/17-06-2020) and the Scientific and Ethics Committee of Greek Association of Alzheimer’s Disease and Related Disorders (GAADRD) (242/2022 AI_07/10/2021).
Informed Consent Statement
Informed consent was obtained from all participants involved in the study.
Data Availability Statement
Not applicable.
Acknowledgments
Not applicable.
Conflicts of Interest
The authors declare no conflict of interest.
Abbreviations
The following abbreviations are used in this manuscript:
| ADLs | Activities of Daily Living |
| BADLs | Basic Activities of Daily Living |
| IADLs | Instrumental Activities of Daily Living |
| RDF | Resource Description Framework |
| OWL | Web Ontology Language |
| IoT | Internet of Things |
| HC | Healthy Controls |
| SCD | Subjective Cognitive Decline |
| MCI | Mild Cognitive Impairment |
| SSN | Semantic Sensor Network |
| SOSA | Sensor, Observation, Sample, and Actuator |
References
- Edemekong, P.F.; Bomgaars, D.L.; Sukumaran, S.; Schoo, C. Activities of Daily Living. In StatPearls; StatPearls Publishing: Treasure Island (FL), 2023.
- Hall, J.R.; Vo, H.T.; Johnson, L.A.; Barber, R.C.; O’Bryant, S.E. The Link between Cognitive Measures and ADLs and IADL Functioning in Mild Alzheimer’s: What Has Gender Got to Do with It? Int J Alzheimers Dis 2011, 2011, 276734. [CrossRef]
- Carmona-Torres, J.M.; Rodríguez-Borrego, M.A.; Laredo-Aguilera, J.A.; López-Soto, P.J.; Santacruz-Salas, E.; Cobo-Cuenca, A.I. Disability for basic and instrumental activities of daily living in older individuals. PLoS One 2019, 14, e0220157. [CrossRef]
- Stavropoulos, T.G.; Papastergiou, A.; Mpaltadoros, L.; Nikolopoulos, S.; Kompatsiaris, I. IoT Wearable Sensors and Devices in Elderly Care: A Literature Review. Sensors 2020, 20. [CrossRef]
- Stavropoulos, T.G.; Andreadis, S.; Mpaltadoros, L.; Nikolopoulos, S.; Kompatsiaris, I. Wearable Sensors and Smartphone Apps as Pedometers in eHealth: a Comparative Accuracy, Reliability and User Evaluation. In Proceedings of the 2020 IEEE International Conference on Human-Machine Systems (ICHMS), 2020, pp. 1–6. [CrossRef]
- Muurling, M.; de Boer, C.; Kozak, R.; Religa, D.; Koychev, I.; Verheij, H.; Nies, V.J.M.; Duyndam, A.; Sood, M.; Fröhlich, H.; et al. Remote monitoring technologies in Alzheimer’s disease: design of the RADAR-AD study. Alzheimer’s Research & Therapy 2021, 13, 89. [CrossRef]
- Antoniou, G.; Van Harmelen, F. A semantic web primer, 3rd ed.; MIT press, 2004. ISBN 978-0-262-01828-9.
- RDF 1.2 Semantics. Available online: https://www.w3.org/TR/rdf12-semantics/, Available online: (accessed on null). (accessed on 08 November 2023).
- SPARQL 1.2 Query Language. Available online: https://www.w3.org/TR/sparql12-query/, (accessed on null). (accessed on 08 November 2023).
- Stavropoulos, T.G.; Meditskos, G.; Papagiannopoulos, S.; Kompatsiaris, I. eHealth4MS: Problem Detection from Wearable Activity Trackers to Support the Care of Multiple Sclerosis. In Proceedings of the Ambient Intelligence – Software and Applications; Novais, P.; Vercelli, G.; Larriba-Pey, J.L.; Herrera, F.; Chamoso, P., Eds., Cham, 2021; pp. 3–12.
- Mpaltadoros, L.; Alepopoulos, V.; Pliatsios, A.; Vasileiadis, A.; Stavropoulos, T.; Nikolopoulos, S.; Kompatsiaris, I. Rule-Based Detection of Health-related Problems of People with Dementia from Lifestyle Wearables: The support2LIVE Approach. International Journal on Advances in Life Sciences 2022, 13, 40–49. [CrossRef]
- Elhadj, H.B.; Sallabi, F.; Henaien, A.; Chaari, L.; Shuaib, K.; Al Thawadi, M. Do-Care: A dynamic ontology reasoning based healthcare monitoring system. Future Generation Computer Systems 2021, 118, 417–431. [CrossRef]
- Kassiano, V.; Stavropoulos, T.G.; Nikolopoulos, S.; Kompatsiaris, I.; Riga, M. Spatial Awareness for the Deafblind in Natural Language Presentation using SPIN Rules: A Use Case in the SUITCEYES Platform. In Proceedings of the eTELEMED 2020 : The Twelfth International Conference on eHealth, Telemedicine, and Social Medicine, 2020.
- Shishehchi, S.; Banihashem, S.Y. A rule based expert system based on ontology for diagnosis of ITP disease. Smart Health 2021, 21, 100192. [CrossRef]
- Nicholson, N.C.; Giusti, F.; Bettio, M.; Negrao Carvalho, R.; Dimitrova, N.; Dyba, T.; Flego, M.; Neamtiu, L.; Randi, G.; Martos, C. An Ontology to Model the International Rules for Multiple Primary Malignant Tumours in Cancer Registration. Applied Sciences 2021, 11. [CrossRef]
- Bae, I.H. An ontology-based approach to ADL recognition in smart homes. Future Generation Computer Systems 2014, 33, 32–41. Special Section on Applications of Intelligent Data and Knowledge Processing Technologies; Guest Editor: Dominik Ślęzak. [CrossRef]
- Shafiq, M.; Gu, Z.; Cheikhrouhou, O.; Alhakami, W.; Hamam, H. The Rise of “Internet of Things”: Review and Open Research Issues Related to Detection and Prevention of IoT-Based Security Attacks. Wireless Communications and Mobile Computing 2022, 2022, 8669348. [CrossRef]
- CERTH - Smart Home. (accessed on 08 November 2023). Available online: https://smarthome.iti.gr/.
- Petersen, R.C.; Roberts, R.O.; Knopman, D.S.; Boeve, B.F.; Geda, Y.E.; Ivnik, R.J.; Smith, G.E.; Jack, Clifford R., J. Mild Cognitive Impairment: Ten Years Later. Archives of Neurology 2009, 66, 1447–1455. [CrossRef]
- Dubois, B.; Feldman, H.H.; Jacova, C.; Hampel, H.; Molinuevo, J.L.; Blennow, K.; DeKosky, S.T.; Gauthier, S.; Selkoe, D.; Bateman, R.; et al. Advancing research diagnostic criteria for Alzheimer’s disease: the IWG-2 criteria. Lancet Neurol 2014, 13, 614–629. [CrossRef]
- Molinuevo, J.L.; Rabin, L.A.; Amariglio, R.; Buckley, R.; Dubois, B.; Ellis, K.A.; Ewers, M.; Hampel, H.; Klöppel, S.; Rami, L.; et al. Implementation of subjective cognitive decline criteria in research studies. Alzheimers Dement 2016, 13, 296–311. [CrossRef]
- Fibaro Sensors. Available online: https://www.fibaro.com/.
- OWL 2 Web Ontology Language Document Overview (Second Edition). Available online: https://www.w3.org/TR/owl2-overview/, (accessed on 08 November 2023).
- Haller, A.; Janowicz, K.; Cox, S.; Lefrançois, M.; Taylor, K.; Phuoc, D.; Lieberman, J.; García Castro, R.; Atkinson, R.; Stadler, C. The Modular SSN Ontology: A Joint W3C and OGC Standard Specifying the Semantics of Sensors, Observations, Sampling, and Actuation. Semantic Web 2018, 10. [CrossRef]
- Stavropoulos, T.G.; Meditskos, G.; Lazarou, I.; Mpaltadoros, L.; Papagiannopoulos, S.; Tsolaki, M.; Kompatsiaris, I. Detection of Health-Related Events and Behaviours from Wearable Sensor Lifestyle Data Using Symbolic Intelligence: A Proof-of-Concept Application in the Care of Multiple Sclerosis. Sensors 2021, 21. [CrossRef]
- GraphDB by Ontotext. Available online: https://graphdb.ontotext.com/, (accessed on 08 November 2023).
Figure 1.
Architecture of the proposed Framework.
Figure 2.
Events representation in a Gantt Chart.
Figure 3.
Activity representation in a Gantt chart.
Figure 4.
Comparison between the number of activities per Task recorded by the platform and the available ground truth data.
Figure 4.
Comparison between the number of activities per Task recorded by the platform and the available ground truth data.
Figure 5.
Owl Classes of ADL Recognition Ontology.
Figure 6.
Knowledge Graph: Visual example of an Activity instance and its relations.
Figure 7.
Count of events for "Hot meal preparation" activity.
Table 1.
Set of devices installed in Smart Home.
| # | Device | Description | Data type |
|---|---|---|---|
| 1 | Motion sensor | Presence/motion capture | Boolean (0,1) |
| 2 | Door/drawer/cabinet sensor | Door/drawer/cabinet opening - closing | Boolean (0,1) |
| 3 | Flood sensor | Water leaks or flooding | Boolean (0,1) |
| 4 | Panic button | Emergency alarm triggering | Boolean (0,1) |
| 5 | Wall plugs | Consumption monitoring | Float number |
Table 2.
Rule table for problems detection.
| # | Variables | Rule | Problem |
|---|---|---|---|
| 1 | activity duration | activity duration > threshold | Activity takes too long |
| 2 | order of events | given order != correct order | Divergence from protocol |
| 3 | count of events | given count != correct count | Took extra steps |
| 4 | count of distinct events | given count < correct count | Missing steps |
Table 3.
Activities duration comparison for Participant 1 and Participant 2.
| Participant 1 | Participant 2 | |
|---|---|---|
| Hot Meal Preparation | 3083 | 1432 |
| Hot Beverage Preparation | 263 | 174 |
Table 4.
Problems detected for the activity "Hot Meal Preparation".
| Too Long Duration (%) | Divergence (%) | Extra Steps (%) | |
|---|---|---|---|
| HC | 18.18 | 81 | 12.50 |
| SCD | 45.45 | 81 | 12.50 |
| MCI | 62.50 | 100 | 37.50 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
MDPI Initiatives
Important Links
© 2024 MDPI (Basel, Switzerland) unless otherwise stated