Prerpints.org logo
Preprint
Article

Skeleton-based Privacy-Preserving Smart Activity Sensor for Senior Care and Patient Monitoring

Altmetrics

Downloads

218

Views

145

Comments

0

Submitted:

01 January 2024

Posted:

03 January 2024

You are already at the latest version

Alerts
Abstract
This paper introduces the AltumView Sentinare smart activity sensor for senior care and patient monitoring. The sensor uses an AI chip and deep learning algorithms to monitor the activity of people, collect activity statistics, and notify caregivers when emergencies such as falls are detected. To protect privacy, only skeleton (stick figure) animations are transmitted instead of videos. The sensor is highly affordable, accessible, and versatile. It was a CES 2021 Innovation Award Honoree, and has been selected by Amazon as one of only three fall detection devices integrated into its Alexa Together urgent response service, and has received very positive reviews from Amazon customers. It has also been used in different senior care settings in about ten different countries. The paper presents the main features of the system, the evidences and lessons learned from its practical applications, and future directions.
Keywords: 
Subject: Computer Science and Mathematics  -   Artificial Intelligence and Machine Learning

1. Introduction

The world is aging at an unprecedented rate. In 2018, for the first time in history, seniors aged 65 or above outnumbered children under five years of age globally. In 2022, there were 771 million seniors aged 65+ years globally, which was almost 10% of global population. In 2020, 13 countries were super-aged, with more than 20% of their population were seniors. The number will rise to 34 by 2030, including Canada and US. By 2050, the world’s senior population will reach 1.5 billion, and the number of persons aged 80 years or over will be tripled, from 143 million in 2019 to 426 million in 2050.
Due to people’s preference and the limited resources, most seniors grow old in their own homes (aging in place) and in the communities nearby, and many of them live alone, especially in developed countries. For example, 38.4% of seniors in France live alone. In US, data for 2012 show that about 59% of seniors lived with a spouse or partner only, and another 28.5% lived alone. Together, almost 88% of the seniors were in independent living arrangements. In Canada, data from the 2011 National Household Survey show that 56.4 % seniors lived as part of a couple and another 25 % lived alone.
Seniors face many safety threats, such as falls, medical emergencies, chronic diseases, and wandering. In particular, fall is a main threat to senior’s health and safety, which often causes pressure ulcer, muscle necrosis, pneumonia, disability, loss of independence, and even death. According to the World Health Organization, about 28 % − 35 % of seniors would go through at least one fall each year, and this danger rises to 42 % for individuals over the age of 70. Seniors have a 25% chance of dying within six months to a year if they fall and break a hip. Among age-related death, falls account for 40 % . More than 50 % of injury-related hospitalizations among seniors are caused by falls, which account for 10 − 15 % of all emergency department visits. In US, 29 million seniors fall in a year, resulting in 7 million injuries, and around 2.8 million seniors need emergency medical attention. Within the homes, the National Health Interview Survey of 1997-1998 found that 43 % of falls happen in the bedrooms and bathrooms. Therefore, how to detect falls in these places while protecting the privacy of seniors is very critical.
In the COVID-19 pandemic, seniors are the most vulnerable people, and many long-term care facilities have been hit hard. The social distancing requirements made it even more difficult for seniors to receive helps from family members or care workers.
Aging population also leads to serious labor shortage. Therefore it is necessary to use new technologies to take care of the seniors, maximize their autonomy and independence, while minimizing the risks to their privacy.
Technologies can also improve the health inequity problem faced by people in remote areas, people with low income, people with disabilities, or indigenous people.
As a result, many governments in the world have started to invest heavily in smart senior care technologies. For example, in 2021, the US government invested $ 80M to Georgia Tech, Johns Hopkins University, University of Pennsylvania, and University of Massachusetts, Amherst, to establish research centers focusing on AI-based senior care technology and commercialization. In Dec. 2022, the Canadian government invested C $ 47 M to establish the envisAGE network to support 100 startup companies in Age Tech. Recently, the Australia government is also reforming its senior care policy and strategy, with focus on introducing new technology to take care of the seniors.

1.1. Limitations of Traditional Medical Alert Systems

Despite great advances in the technologies during the last several decades, the technologies used by most senior care facilities and seniors living at home are very outdated. Existing medical alert systems or personal emergency response systems usually have two types: pull cords (or call bells), and wearable devices such as pendants, panic buttons, and wristbands. Both of them have many limitations.
The pull cords (or call bells) are only installed at fixed locations, and may not be readily available in case of emergencies. They are also useless when seniors lose consciousness. The wearable devices also have various problems: 1) Their functionalities are very limited. Some only have a panic button. 2) Some devices can detect falls, but they are based on accelerometer, and cannot detect slow falls. 3) Wearable devices need to be worn all the time, which is not convenient, especially for persons with dementia or are very weak. Many seniors refuse or forget to wear them. Some patients with dementia could press them randomly and create many false alarms. 4) Seniors cannot press the buttons if they lose consciousness. 5) Wearable devices need frequent recharging.

1.2. Limitations of Other Types of Fall Detection Devices

Since fall is a main threat to seniors, many different technologies have been developed for fall detection. Vision-based fall detection is an active field. Some systems are similar to traditional surveillance systems, where some human operators are needed to watch multiple video monitors in the control center around the clock to find out if there are any accidents. This is labor-intensive, and can only be used in hospitals or common areas of long-term care facilities, but not in bedrooms and bathrooms.
Some other systems send the videos to a local or cloud server, where some intelligent al-gorithms are performed, such as fall detection. This centralized approach requires more resources in the server and high-speed network to send the videos continuously. It also could not protect the privacy of the people. Some systems use AI algorithms in the camera to detect emergencies such as falls, but still transmit videos out of the cameras, which cannot protect the privacy of the persons being monitored.
Some other systems use thermal cameras, depth cameras, or radar devices. These approaches can preserve people’s privacy, but have various limitations, such as low resolutions, small range of coverage, and difficulty in distinguishing between people and pets.

1.3. The AltumView Sentinare Smart Visual Sensor

To address the challenges of aging population, since 2018, we have developed the Sentinare smart activity sensor system for senior care, which includes the Sentinare smart activity sensor, Amazon AWS-based cloud server, and mobile app.
We believe the recent advancement in artificial intelligence (AI) and computer vision offers the enabling technology to allow us to take care of our senior population in a cost-effective manner.
The Sentinare sensor is designed to be affordable, accessible, and versatile. It uses an AI chip and the latest deep learning algorithms to monitor the activity of people, collect activity statistics, and notify caregivers when emergencies such as falls are detected. To protect privacy, only stick figure animations are transmitted instead of videos, allowing it to be used anywhere in the home.
The Sentinare sensor was selected as an Innovation Award Honoree at CES 2021, the largest consumer electronics show in the world. In 2022, it was also selected by Amazon as one of only three fall detection devices for its Alexa Together urgent response service [1], and has received very favorable reviews from Amazon customers.
In addition to US and Canada, the Sentinare sensor has been adopted by customers in about ten other countries and regions, including China, Hong Kong, Japan, Australia, Italy, Spain, and some Southeast Asian countries. It is being deployed in all senior care set-tings, from aging at homes, to independent living, assisted living, and long-term care. It is a powerful tool for senior care facilities to improve the efficiency of the care workers and the quality of service, reduce staff workload, stress, and turnover rate. It has also been applied to remote patient monitoring, not just for seniors, but other patients, for example, young patients with intellectual/developmental disabilities (I/DD), such as autism and Down syndrome.
In this paper, we will discuss the features of the AltumView Sentinare system, the lessons learned from its practical applications in different countries and different senior care set-tings, and future applications.

2. Main Features of the AltumView Sentinare System

Figure 1 shows the overall architecture of the AltumView Sentinare smart activity sensor system, which includes the Sentinare sensor, the Amazon AWS-based cloud server, the Sentinare mobile app, the web hub, and the API (Application Programming Interface) for third-party integration.
Three generations of the Sentinare sensor have been developed. The latest version uses a powerful RV1126 SoC (system on chip), which is equipped with a quad-core ARM Cortex-A7 CPU and a RISC-V MCU. More importantly, it has a NPU (Neural Processing Unit) with a capacity of 2.0 TOPS (trillion operations per second) and supports INT8/INT16 precisions.
Thanks to the powerful CPU, we have implemented many unique features in the Sentinare sensor.

2.1. Privacy Preservation

Preserving the privacy of users is the core concern of our design. The powerful AI chip in the Sentinare visual sensor allows it to run various latest deep learning algorithms inside the sensor, without having to transmit videos to the cloud for processing, thereby protecting user’s privacy, and also reducing the transmission cost and cloud cost.
Moreover, to provide necessary visual information, the sensor converts people’s activities into skeleton (stick figure) animation, which can be streamed to the cloud and the mobile app or included in the alert notification when emergencies such as falls are detected. This approach strikes a good balance between protecting the privacy of the seniors and providing necessary visual information for the caregiver to know what has happened and identify false alarm, allowing the sensor to be used in bedrooms and washrooms.
The Sentinare sensor also gives users the options to save the raw videos in the SD card of the sensor, which are never transmitted out of the sensor. These raw videos can be used for detailed analysis or as legal evidences.

2.2. Automatic Fall Detection [31,32]

We have developed and implemented a deep learning-based automatic fall detection algorithm in the Sentinare sensor. When the sensor detects a fall (and other emergencies as described below), it will automatically send a push notification to the mobile app via the server. It does not need any action from seniors, such as pressing the emergency button, which is not a viable solution when seniors lose consciousness, or when the seniors have dementia. We have made numerous improvements to the algorithm based on field trials and customer feedback, and achieved state-of-the-art performance.

2.3. Fall Risk Assessment [41]

The Sentinare mobile app includes a fall risk assessment tool. In the past, patients had to go to the hospital for the doctors to assess their risk of falling. Our system allows them to do the assessment at home whenever they want. This makes it easier for the users to keep track of the fall risk of seniors, review the corresponding recordings, and identify early sign of high risk of falling, so that preventive precautions can be taken to prevent falls.

2.4. Low-Cost Stick-Figure Recording and Storage [42,43]

Since the Sentinare sensor only transmits stick figure animation to the server and the app, it needs much less data bandwidth and cloud cost than traditional surveillance video cameras. For example, the bit rate of surveillance videos is at least 50KB/sec, or 180MB/hour. Compared to this, one hour of stick figure streaming only needs about 5MB, which can be further reduced by using some simple compression methods. Therefore the transmission and storage costs can be easily reduced by more than 30 times. This allows us to store the stick figure data for much longer time in the cloud for review and analysis, and with much lower cost than videos.
The stick figure recording can provide many useful information. For example, how long a senior sleeps at night, and how many times the senior goes to kitchen or bathroom. These statistics can provide good indication of the senior’s health, and allow caregivers to identify some conditions earlier, such as urinary track infection. It can also be used by doctors and family members to monitor the results of treatments for some chronic diseases, such as dementia.
The Sentinare system is currently the only commercial system that can collect large-scale, real-world behavioral data from patient’s homes in a low-cost and privacy-preserving way. Therefore it has huge potential in many medical applications.

2.5. Face Recognition [33,34,36,37,39,40]

The Sentinare sensor has built-in deep learning-based face recognition feature, which can be very useful in various applications, as explained below.

2.6. Daily Activity Statistics [42]

Based on the built-in face recognition and action recognition features, the Sentinare sensor can collect the statistics of a person’s activity in each day, for example, how much time is spent in different rooms, different locations of a room, and the time spent on lying, sitting, and standing. The information can be used by caregivers for analysis and diagnosis. It can also be used by the seniors and their family members to understand the senior’s health condition, and identify changes to their health earlier.

2.7. Region of Interest (ROI) Monitoring

The Sentinare sensor provides flexible ROI features, including Restricted Region, Overstay Detection, and Absence Detection. Users can define regions of interests via the app, and receive push notifications when the targeted events happen in the ROI. The restricted region feature can also be used to prevent residents from wandering away from the facility, or entering rooms that they are not supposed to go (e.g., to prevent resident-to-resident aggressive actions). The overstay detection can be used to encourage the seniors to do more exercise, or as a backup if the fall detection algorithm does not work. The absence detection can be helpful to detect emergencies even if the person has an accident in an area not covered by the Sentinare sensor.

2.8. Hand-Waving Detection

The fall detection is a passive feature. To allow seniors to actively seek help, for example, in emergencies such as heart attack or stroke, the Sentinare sensor implements a hand-waving detection feature. It also serves as a backup if the senior falls but the fall de-tection algorithm fails to detect it. This can also be used if the seniors need other non-emergent helps from the caregivers. Different sensitivities are provided for hand-waving detection. In the high sensitivity, only raising hand is needed to trigger the alert, and there is no need to wave hand. This is suitable if the person is weak.

2.9. Voice Calls

The Sentinare sensor has a built-in microphone and a speaker. It allows the caregiver to call the Sentinare sensor directly. The seniors do not need to do anything to answer the call. This has been found to be very valuable to check the seniors’ condition and relieve the anxiety of seniors in case of emergency.

2.10. Night Vision

The Sentinare sensor has built-in infrared lights, allowing the sensor to be able to detect people and emergencies in the night.

2.11. Secondary User

Each Sentinare account can invite some secondary users to access selected sensors and people in the account. This can be useful for senior care facilities or home care companies to assign jobs to different care workers, so that each of them only receives alerts from selected residents of the organization. They can also invite family members of a senior to access the sensor in their loved ones’ room, but not from other residents.

2.12. API for Third-Party Integration

The Sentinare system provides cloud API for third-party integration. All the data from the sensor can be retrieved from our server via the API, and integrated into other systems.

2.13. Email Summary

The account activity statistics can be emailed to the account holders on a weekly or daily basis, so that they can know if everything is normal without opening the Sentinare app.

2.14. Security

The Sentinare system has adopted various security protocols, including the TLS/SSL cryptographic protocols, the secure AWS protocols, the OAuth 2.0 authentication protocol, and the light-weight, secure, and scalable MQTT protocol designed for IoT devices. Together with the fact that only stick figure animation is transmitted, the system is more secure than traditional surveillance camera systems, and can easily meet the privacy and data security regulations in various countries, such as HIPAA in US and the GDPR in Europe. The system also has Amazon AWS cloud servers in different countries to make sure the data do not leave its original country.

3. Main Algorithms in the Sentinare Sensor

We design various deep learning-based neural networks using different optimization approaches, so that they can run in real-time on the embedded CPU in the Sentinare sensor, including network pruning, quantization, and depth-wise convolution. As a result, the Sentinare sensor can perform real-time deep learning-based pose estimation, action recognition, fall detection, face detection, and face recognition. The main block diagram of the different algorithms is shown in Figure 2.

3.1. Pose Estimation

The sensor monitors human action by first detecting and estimating the pose of each per-son in each video frame, i.e., localizing all human keypoints (also known as joints) such as head, arms, hands, and legs. A skeleton diagram of the person can be obtained by connecting neighboring keypoints with straight lines. A typical method uses 18 keypoints to represent a person (two eyes, two ears, nose, neck, two shoulders, two elbows, two wrists, two hips, two knees, and two ankles), and the resulting skeleton has 17 line segments, as shown in Figure 3.
In order to achieve real-time pose estimation algorithm in the resource-limited embedded CPU, we applied various channel pruning techniques to reduce the feature map width of the network. We also quantize the model and use 8-bit integer precision instead of 32-bit floating-point precision. During the training, we apply data augmentation to improve the pose estimation performance from different visual sensor angles.

3.2. Frame-Level Action Recognition

After locating the keypoints from pose estimation, we crop the person from the image by forming a bounding box from the detected keypoints. Then, we feed the cropped person image into an action classifier to predict the probability of each action of interests to us. To improve the accuracy, we develop a two-level convolutional neural network (CNN) approach to detect falls and other activities from a single image, as shown in Figure 4.
For the first level, a CNN with a binary classifier is used to detect “fall” actions from “normal” actions. The second level contains two CNNs, with one of them further classifying “fall” actions into “lying” or “struggling” actions, and the other one further classifying “normal” actions into “standing”, “sitting in chair”, “sitting on floor”, “bending”, and “squatting” actions.

3.3. Fall Detection Algorithm [31,32]

The pose estimation model described above provides the locations of the human key-points in the image, and the frame-level action recognition model generates frame-by-frame action labels, and can distinguish dangerous actions and normal actions. To achieve more reliable fall detection, image layout and temporal information need to be considered. We build a reliable fall detection system with automatic scene segmentation and state machine [31].

3.4. Automatic Scene Segmentation

To robustly detect falls, especially falling from bed and sofa, the system needs to distinguish different lying and struggling actions: regular actions (e.g. lying in bed or sofa) and dangerous actions (e.g. lying or struggling on the floor). Therefore, a scene segmentation network is developed. It segments the indoor scenes into three categories: the dangerous region which contains floor and carpet; the safe region which contains the objects where one can lie down, such as bed and sofa; and the background region. We also made various efforts to reduce the complexity of our network so that it can run faster. Note that the segmentation algorithm does not have to run in real time, because it is only called one time, and only needs to be updated when the installation or the room layout changes.

3.5. State Machine for Fall Detection

To reduce false alarms caused by the frame-level action recognition method described above, we develop a state machine to incorporate temporal information from consecutive video frames, making the falling detection system more robust and reliable. Our state machine includes four states representing increasing levels of falling possibility. The probability increases if the person is detected as falling in consecutive frames, and reduce if the person is detected as not falling in a frame.

3.6. Fall Detection Performance

Our fall detection algorithm was initially trained using data collected in our lab based on the typical installation position of our sensor. In addition, our algorithm has been tested for several years in several senior care facilities in different countries. We also received feedback from many other customers, including Amazon users, who have identified various special cases that our original algorithm did not work well. Based on the feedback from these real-world examples, we have made numerous modifications to improve our performance in different scenarios, such as body postures, lighting conditions, room layout, sensor heights, and angles. Therefore our solution is much more robust than other research papers, which are usually only based on data collected in the lab environments.
In this section, we test the fall detection performance of our sensor on two popular public available datasets. The results show that our method achieves state-of-the-art accuracy compared to existing methods.

3.6.1. Dataset 1 – CNRS Fall Dataset [10]

The CNRS dataset is proposed by the French National Centre for Scientific Research. It consists of 130 simulated fall videos in total. There are 4 room setups (Home1, 2, Coffee room 1, 2). The camera position is medium to high, simulating mounting position of typical surveillance cameras. Ground truth bounding box information is provided for the test videos. The start and end frames of fall events are also provided. In the literature there are 2 different experiment setups for this dataset.
1) Following [2], 96 video clips are extracted with 45 frames before the start of the fall action and 15 frames after the end of fall action. In [2], frame-level accuracy is used to evaluate the performance of fall detection, i.e., each frame is labelled as fall or normal frame. The training and testing split also has two setups, 2-fold cross-validation and 4-fold cross-validation. In the 2-fold test, the models are trained on either home or coffee room setup, and tested on the other setup. In the 4-fold test, the models are trained on 3 of the 4 setups, and tested on the remaining 1 setup. The average accuracy is reported for both 2-fold and 4-fold tests.
Table 1 shows the results of our fall detection algorithm and results reported in [2]. Our method achieves higher frame-level results in the 2-fold test and slightly lower than theirs for the 4-fold test.
We also test the performance of our event-level fall detection algorithm, which achieves 100 % accuracy for both 2-fold and 4-fold tests. No event-level result was reported in [2].
2) Following [3], 127 videos are randomly selected as training and testing set in a 2-fold cross-validation setup. The event-level accuracy is reported in [3]. Table 2 shows the experiment results of our method and results reported in [3]. Our method achieves significantly higher sensitivity and specificity compared to the method in [3]. Our method only has 2 false negative detections (sensitivity 97.92 % ) and no false positive detection (specificity 100 % ), and the overall accuracy is 98.43 % .

3.6.2. Dataset 2 – UMontreal Dataset [11]

The UMontreal dataset is developed by the University of Montreal, Canada. Simulated fall events are recorded in a single room setup with 8 cameras around the room. The camera position is high, simulating mounting position of typical surveillance cameras. There are 184 fall actions and 920 normal actions annotated in the videos. Videos are randomly selected as training and testing sets in a 2-fold cross-validation setup.
Table 3 shows the experiment results of our method and results reported in [4,5,6,7,8,9]. Our method achieves one of the best sensitivity among the existing methods. [5] and [8] have better sensitivity, but their specificities are lower than our method. We also achieve the second best specificity after [7], but [7] has a very low sensitivity of only 80.60 % . In short, our method achieves the best overall sensitivity, specificity, and accuracy among the existing methods.

3.7. Backups of Fall Detection

The fall detection performance of our product (and any other products) in real-world application can be affected by many factors, such as distance, sensor’s angle, the person’s pose after fall, lighting condition, and occlusion.
To improve the fall detection performance in the real world, the hand-waving detection and the overstay detection features can be used as backups for fall detection. The former needs the person to actively wave hand to the sensor. The overstay detection can be set up as another backup in places where fall detection could fail and the person is not supposed to stay long, such as bathrooms, bedside, or hallways.
The absence detection can serve as another layer of backup. It can be set up in an area where a person should show up regularly. An alert will be generated when nobody is detected for the defined duration, which could happen when the person has accidents in another place not covered by the Sentinare sensor.

3.8. Fall Risk Assessment [41,42]

Fall risk assessment is another desired feature, which can prevent falls. The Sentinare app implements a fall risk assessment method, which allows users to evaluate the fall risk of seniors or patients regularly in long-term care facilities, community centers, or at homes, without having to go to hospitals to do the test, and identify people with high fall risk earlier, so that necessary interventions can be taken to reduce falls and the damages they could cause.
The fall risk assessment method implemented in the Sentinare app is a combination of two widely used methods in the medical community, the Morse Fall Risk Assessment Scale and the three-meter Timed Up and Go (TUG) method [12,13,14].
The Morse Fall Risk assessment method assigns scores based on answers to six questions, and the total score determines the fall risk [13]. The six questions are: 1) History of falling in the last 3 months; 2) Secondary diagnosis; 3) Ambulatory aid; 4) Intravenous or saline assistance; 5) Mental illness; 6) Gait quality.
The Morse method’s has a sensitivity of 78 % , specificity of 83 % , and interrater reliability of 0.96, meaning repeated tests by different examiners are highly consistent. Therefore the method has been widely adopted by hospitals and senior care facilities around the world.
The first five questions of the Morse Fall Scale are quite straightforward, but the last step of the Morse Fall Scale is to assess the gait or walking quality of the person. In order to avoid relying on the doctors or nurses to do the gait assessment, we obtain the gait quality assessment of the Morse fall scale using another popular method, the three-meter Timed Up & Go (TUG) test [14], which was initially designed for seniors, but has also been found to be very accurate for people with Parkinson’s Disease, Alzheimer’s, hip fracture, routine orthopaedic surgery, and other conditions. The TUG method’s sensitivity and specificity are both 87 % , and interrater reliability is 0.98. which is as good as the Morse Fall Scale. The TUG test is used by Centers of Disease Control (CDC) in US and many other organizations.
To do the TUG test [14], users only need to have a room with an empty area of at least 3m long. It is better to do the assessment within the view of a Sentinare sensor, so that the process can be recorded for future review and analysis. The person being assessed starts by sitting on an armchair. Another user of the Sentinare app will start the timer in the app. After starting the test, the person being assessed stands up from the chair, walk 3m away from the chair (a marker can be put on the floor in advance), turn around, walk back to the chair, and sit down. The app user will then stop the timer.
The time (in seconds) needed to complete the TUG assessment will be used to determine the gait quality in the Morse Fall Scale. If the time is more than 12s, the gait is impaired, and 45 points are assigned to the gait quality in the Morse Fall Scale. If the time is between 10-12s, the gait is considered weak, and 30 points are assigned to the gait quality of the Morse Fall Scale. If the time is less than 10s, the gait is normal, and 0 point is assigned to the gait quality of the Morse Fall Scale.
Finally, if the total score of the six factors in the Morse Fall Scale is 0-24, the fall risk is low. If the total score is 25-44, the fall risk is medium. If the score is more than 45, the fall risk is high.
Since our fall risk assessment method combines the two popular methods of Morse Fall Scale and TUG, and avoids the need of doctors or nurses (and thus the interrater discrepancy), it is more accurate that each of the two methods. It also provides the stick figure re-cording of the assessment, which can be reviewed and analyzed in the future.

4. Practical Applications and Less Learned

4.1. Long-Term Care Facilities

Since 2019, the Sentinare sensor has been deployed in various long-term care facilities in several countries, including China, Canada, Japan, and Australia. It has been shown that the Sentinare sensor can improve the efficiency of the care workers and the quality of service, reduce their workload, stress, and turnover rate.
Based on our experience, several factors could affect the successful adoption of the system (and other new technologies).
First, the facility needs to have a WiFi network in order to use the Sentinare system. This can be a challenge for some facilities, because their IT infrastructure is outdated. Some facilities do not have WiFi at all, or the existing WiFi coverage and bandwidth is not good enough. Some initial investments are needed to upgrade the WiFi networks. Sometimes it is necessary to provide a dedicated WiFi network for Sentinare, if there are already too many users in the existing network. Fortunately many telecom operators or IOT (internet of things) service providers offer WiFi packages for these facilities at affordable price. Some of them use SIM-card-based mobile WiFi routers, and do not need too much construction efforts.
Second, some existing workflows or protocols of the facility need to be changed to incorporate the Sentinare system. For example, the facility needs to provide mobile phones or tablets to the staffs (if they do not currently use the mobile devices) so that they can use the Sentinare app. If the facility is using other systems to manage all of their devices and alerts, they might need to integrate the Sentinare system into their existing systems, which requires some software development from their platform developer.
Some protocol changes might need approval from the governments or changes of the corresponding regulations. For example, some governments might require the care worker to do a room check once every two hours. If these regulations are not changed, there will be less motivation for the facility to adopt new technologies. Therefore the support from the government is important to encourage the facilities to adopt new senior care technologies.
Insurance policy also plays an important role that affects the adoption of new technology. If Sentinare or other new technologies are accepted by the government or private insurance, it will be easier for the facilities to adopt the new products. For example, recently Sentinare has been approved by the long-term care insurance in Japan.
Since no product is perfect, how to reduce the number of false alarms in the facility is another important factor. If there are too many false alarms, it could lead to staff alert fatigue. In the Sentinare system, we have implemented the following methods to reduce the number of alerts (can be true alerts or false alarms), based on user feedback. We are also always improving our algorithms to reduce the number of false alerts.
  • Delay Fall Alert: If this flag is turned on, the sensor will wait for 30s before sending a fall detection alert.
  • Duplicate Alert Prevention: If this flag is turned on, after an alert is generated from a sensor in a room, the subsequent alerts of the same type from any sensor of the same room within the specified period of time will not be sent to the app, even if they are true alerts.
  • Ignoring Similar Alerts: If certain types of false alarms are frequently generated from the same location, users can turn on the Ignoring Similar Alerts flag when resolving the alert. Subsequent alerts of the same type from the same location will be ignored.

4.2. Consumer Market for Aging at Homes

Amazon has invested extensively in healthcare in the last few years. Alexa Together is a new service launched in Dec. 2021 by Amazon to help seniors, especially those living alone, with 24/7 emergency response. It is a paid service currently only available in the United States. Alexa Together is based on the Alexa Echo smart speaker and certified third-party fall detection devices. Since early 2022, AltumView’s Sentinare sensor has been selected by Amazon as one of only three fall detection devices integrated into Alexa Together. As a result, the Sentinare sensor has been available on Amazon US and Canada for the consumer market since May 2022. Note that the Sentinare system can still work independently without Alexa Together.
We have integrated the Sentinare system into the Alexa Together via Amazon’s API. As a result, when the Sentinare sensor detects a fall, it will send a message to the Alexa, and the Alexa speaker in the Amazon account of the seniors will talk to the seniors if they have fallen and if they need any help. If they answer Yes, a 24x7-available human operator will call the seniors to provide emergency support. This process is illustrated in Figure 5.
Since May 2022, we have got feedback from many Amazon customers about the Sentinare system. We have also made numerous improvements to improve our design. As a result, Sentinare receives very favorable reviews from Amazon users, which confirm the need of the product and its advantage in the market.
From the direct feedback from Amazon customers (including Amazon reviews), we have learned the following important knowledge and lessons.
  • Almost all customers of our product in the consumer market are adult children between 40-60 years of age. They use Sentinare to take care of their parents, mostly living alone. The seniors usually have high risk of falling and cognitive impairments, but they still live in their own homes and not in the facilities, due to various reasons. Having a device like Sentinare can protect the safety of the seniors, and provide much needed peace of mind to the family.
  • The adult children need to get the permission of the seniors in order to install our product, even if the seniors have cognitive impairments.
  • The privacy-preserving stick figure view provided by the Sentinare sensor is well received and appreciated by both the adult children and the seniors. Many customers state that this is a good use of AI. The visualization provided by the stick figure gives the family members a convenient and comfortable way to check the condition of their loved ones while respecting their privacy. Even occasional false alarms are welcomed by many customers, which show the system is running. Besides, false alarms can be easily identified by playing back the stick figure recording.
  • Most users purchase three or two sensors, and many of them install the sensors in bedrooms and bathrooms.
  • User-friendly design of the system is extremely important, because many of the mid-aged adult children that purchase Sentinare are not tech-savvy. The app and the sensor have to be well designed to reduce the efforts of learning how to set up and use the system.
  • Since many customers are not good at new technologies, providing good customer support is also very important to help them learn how to use the system.
  • Similar to long-term care facilities, some homes also do not have good WiFi signals. Making the system work reliably in these environments is also very critical that could affect the user experience.
  • Our Amazon page and positive reviews also attract some business customers to us.

5. Future Directions

5.1. More Advanced Algorithms

The Sentinare system is a joint edge/cloud computing platform. Currently our algorithms mainly run on the sensor, and only detect simple actions such as standing, sitting, lying, and falling in the sensor.
In the future, we plan to develop more advanced deep learning algorithms in the cloud to recognize more complex actions. We will first detect instrumental activities of daily living (ADL) such as eating, drinking, cooking, and making phone calls [15,16,17,18,19,20]. Recognizing these activities is very useful to evaluate the health of seniors.
Our long-term goal is to develop algorithms to identify some behavioral diseases earlier, such as Parkinson’s disease, dementia, depression, and autism, by collaborating with physicians and hospitals [22,23,24,25,26,27,28]. It is also possible to employ the fast evolving large language model (LLM) and large vision model (LVM)-based AI technologies in our system.
These features will greatly enhance healthcare decisions while supporting service providers in their health interventions.

5.2. Applications in Remote Patient Monitoring (RPM) and Remote Therapeutic Monitoring (RTM)

The Sentinare system has great potential in medical applications, especially RPM and the new field of RTM.
In current RPM applications, hospitals and health systems mainly use devices that automatically record and transmit patients’ vital signals, such as weight, temperature, blood pressure, and pulse oximetry. As the advancement of new sensors, it is possible to monitor more data, including the activity data provided by the Sentinare sensor.
In 2021, the Centers for Medicare and Medicaid Services (CMS) approved a new category of service, the Remote Therapeutic Monitoring (RTM) [21], which refers to outpatient rehabili-tation therapy services that include monitoring the musculoskeletal disorders (MSKDs) and respiratory system conditions, as well as therapy adherence and therapy response in patients, through automatic or self-reported metrics. In US, over half of the adults suffering from MSKDs, and the annual costs have been estimated to be $874 billion, or 5.7 % of the annual GDP [29]. The Sentinare sensor is very suitable for RTM.
RTM services can be ordered and billed to CMS codes by physical therapists (PTs), occupational therapists (OTs), and speech-language pathologists, without a physician or nurse practitioner.

5.3. Integration with More Third-Party Systems

We can also integrate with more third-party systems to make the new system more powerful and useful. For example, once the Sentinare sensor identifies signs of some behavioral diseases, it can refer the patients to some patient navigation systems [30].

6. Conclusions

This paper introduces AltumView’s privacy-preserving Sentinare smart activity sensor for senior care and patient monitoring. We discuss the main features and performances of the system, and its applications in different senior care settings in various countries. The outcomes from these applications show that Sentinare is an affordable and versatile tool to protect the safety of seniors, maximize their autonomy and independence, while minimizing the risks to their privacy. We also summarize the lessons learned from the practical deployments of Sentinare. Finally, we outline its great potential in some future directions, including recognition of activities of daily living, early identification of some behavioral diseases, and applications in the new field of Remote Therapeutic Monitoring. All of these show that the Sentinare sensor can greatly enhance healthcare decisions while supporting service providers in their health interventions.

7. Patents

The patents resulting from the work reported in this manuscript are included in [31,32,33,34,35,36,37,38,39,40,41,42,43].

Author Contributions

Conceptualization, J.L.; writing—original draft preparation, J.L., J.Z.; writing—review and editing, J.L.; software, A.A, M.C., C.C., J.Z., Z.D.; validation: Y.X., P.D; All authors have read and agreed to the published version of the manuscript.

Funding

AltumView has been funded by various private investors, National Research Council of Canada, MITACS, AGE-WELL NCE of Canada, and Circle Innovation.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Data sharing is not applicable to this article.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Amazon Alexa Together. Available online: https://www.amazon.com/Alexa-Together/b?ie=UTF8$&$node=21390531011.
  2. Y. Zhang, and H. Neumann. An empirical study towards understanding how deep convolutional nets recognize falls. Proceedings of the European Conference on Computer Vision (ECCV). 2018.
  3. E. Geertsema, G. Visser, M. Viergever, and S. Kalitzin, Automated remote fall detection using impact features from video and audio. Journal of Biomechanics, 2019 May 9;88:25-32. [CrossRef]
  4. Q. Feng, C. Gao, L. Wang, and Y. Zhao, Spatio-temporal fall event detection in complex scenes using attention guided LSTM. Pattern Recognition Letters, Vol. 130, Aug. 2018. [CrossRef]
  5. K. Wang, G. Cao, D. Meng, W. Chen, W. Cao, Automatic fall detection of human in video using combination of features, in 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2016, pp. 1228–1233.
  6. C. Ge, I. Gu, and J. Yang. Human fall detection using segment-level CNN features and sparse dictionary learning. In 2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP). Tokyo, Japan, 2017, pp. 1-6.
  7. E. Auvinet, F. Multon, A. Saint-Arnaud, J. Rousseau, and J. Meunier, Fall detection with multiple cameras: An occlusion-resistant method based on 3-d silhouette vertical distribution, IEEE Transactions on Information Technology in Biomedicine, vol. 15, no. 2, pp. 290–300, 2011. [CrossRef]
  8. C. Rougier, J. Meunier, A. St-Arnaud, and J. Rousseau, Robust video surveillance for fall detection based on human shape deformation, IEEE Transactions on Circuits and Systems for Video Technology, vol. 21, no. 5, pp.611–622, 2011. [CrossRef]
  9. Y. Yun, and I. Gu, Human fall detection via shape analysis on Riemannian manifolds with applications to elderly care, in 2015 IEEE International Conference on Image Processing (ICIP), 2015, pp. 3280–3284.
  10. CNRS fall detection dataset. Available online: http://le2i.cnrs.fr/Fall-detection-Dataset.
  11. University of Montreal fall detection dataset. Available online: http://www.iro.umontreal.ca/ labimage/Dataset/.
  12. Perell KL, Nelson A, Goldman RL, Luther SL, Prieto-Lewis N, Rubenstein LZ. Fall risk assessment measures: an analytic review. J Gerontol A Biol Sci Med Sci. 2001 Dec;56(12):M761-6. [CrossRef]
  13. Morse JM, Morse R, Tylko S. Development of a scale to identify the fall-prone patient. Can J Aging. 1989;8:366–377. [CrossRef]
  14. Shumway-Cook A, Brauer S, Woollacott MH. Predicting the probability for falls in community-dwelling older adults using the Timed Up & Go Test. Phys Ther. 2000;80:896–903. [CrossRef]
  15. S. Yan, Y. Xiong, D. Lin, Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition, In 2018 AAAI Conference on Artificial Intelligence, pp. 7444-7452, 2018.
  16. C. Wang, Y. Wang, Z. Huang, Z. Chen, Simple Baseline for Single Human Motion Forecasting, in 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), BC, Canada, 2021 pp. 2260-2265.
  17. W. Peng, X. Hong, H. Chen, G. Zhao, Learning graph convolutional network for skeleton-based human action recognition by neural searching, In Proceedings of the AAAI Conference on Artificial Intelligence, 34(03), 2669-2676, 2020. [CrossRef]
  18. W. Peng, J. Shi, T. Varanka, and G. Zhao, Rethinking the ST-GCNs for 3D skeleton-based human action recognition, Neurocomputing, Volume 454, 24 September 2021, Pages 45-53. [CrossRef]
  19. Y. Zhou, Z. Cheng, C. Li, Y. Geng, X. Xie, and M. Keuper, Hypergraph Transformer for Skeleton-based Action Recognition, arXiv preprint arXiv:2211.09590, 2022. arXiv:2211.09590, 2022. [CrossRef]
  20. L. Wang, and P. Koniusz, 3Mformer: Multi-order Multi-mode Transformer for Skeletal Action Recognition, CVPR, 2023, arXiv preprint arXiv:2303.14474, 2023.
  21. CMS Manual System, Transmittal 11118. Available online: https://www.cms.gov/files/document/r11118cp.pdf.
  22. Mccurry S., Gibbons L., Logsdon R., Teri L. Anxiety and nighttime behavioral disturbances: Awakenings in patients with Alzheimer’s disease. J. Gerontol. Nurs. 2004; 30: 12–20. [CrossRef]
  23. Peres K., Chrysostome V., Fabrigoule C., Orgogozo J.M., Dartigues J.F., Barberger-Gateau P. Restriction in complex activities of daily living in MCI. Neurology. 2006;67:461–466. [CrossRef]
  24. Gallo J.L., Schmidt K.S., Libon D.J. Behavioral and psychological symptoms, neurocognitive performance, and functional independence in mild dementia. Dementia. 2008; 7:397–413. [CrossRef]
  25. Lee D., Heo S., Yoon S.-S., Chang D., Lee S., Rhee H.-Y., Ku B., Park K.-C. Sleep disturbance and predictive factors in caregivers of patients with mild cognitive impairment and dementia. J. Clin. Neurol. 2014; 10: 303–331. [CrossRef]
  26. Jekel K., Damian M., Wattmo C., Hausner L., Bullock R., Connelly J.P., Dubois B., Eriksdotter M., Ewers M., Graessel E., Mild cognitive impairment and deficits in instrumental activities of daily living: A systematic review. Alzheimer’s Res. Ther. 2015;7:17. [CrossRef]
  27. Ikeda Y, Han G, Maruta M, Hotta M, Ueno E, Tabira T. Association between Daily Activities and Behavioral and Psychological Symptoms of Dementia in Community-Dwelling Older Adults with Memory Complaints by Their Families. Int J Environ Res Public Health. 2020 Sep 18;17 (18): 6831. [CrossRef]
  28. H. Ding, B. Wang, A. P. Hamel, M. Melkonyan, T. F. A. Ang, R. Au, H. Lin, “Prediction of progression from mild cognitive impairment to Alzheimer’s disease with longitudinal and multimodal data,” Front. Dement., Sec. Aging and Risk Factors for Dementia, Nov. 2023. [CrossRef]
  29. Malik KM, Beckerly R, Imani F. Musculoskeletal Disorders a Universal Source of Pain and Disability Misunderstood and Mismanaged: A Critical Analysis Based on the U.S. Model of Care. Anesth Pain Med. 2018 Dec 15;8(6): e85532. [CrossRef] [PubMed]
  30. Chen, F., et al. "Using AI for Patient Navigation during COVID-19 Pandemic: A Case Report," EC Nursing and Healthcare, 5.7 (2023): 01-07. [CrossRef]
  31. H. Ng, X. Wang, J. Zheng, A. Au, C. Chan, K. Lin, D. Zhang, E. Honsch, K.-K. Chan, M. Chen, Y. Gao, A. Auk, K. Ly-Ma, A. Fettes, J. Wu, Y. Lu, Method and System for Privacy-Preserving Fall Detection, US Patent Number 16/672,432, issued, Nov. 23, 2021.
  32. A. Au, D. Zhang, C. Chan, J. Zheng, Deep Learning-based Fall Detection based on Human Keypoints, US patent application 17/534,448, allowed, Oct. 24, 2023.
  33. X. Wang, M. Seyfi, M. Chen, H. W. Ng, and J. Liang, Face Detection Using Small-Scale Convolutional Neural Network (CNN) Modules for Embedded Systems, US Patent number 10,268,947, Issued, Apr 23, 2019.
  34. X. Wang, H. W. Ng, and J. Liang, Convolutional Neural Network (CNN) System based on Resolution-Limited Small-Scale CNN Modules, US Patent Number 10,360,494, Issued, Jul 23, 2019.
  35. X. Wang, M. Seyfi, M. Chen, H. W. Ng, and J. Liang, Joint Face-Detection and Head-Pose-Angle-Estimation Using Small-Scale Convolutional Neural Network (CNN) Modules foe Embedded Systems, US Patent No. 10,467,458, Issued, Nov. 5, 2019.
  36. M. Seyfi, X. Wang, M. Chen, K. Wang, W. Wang, H. W. Ng, J. Zheng, and J. Liang, Method and Apparatus for Real-Time Face-Tracking and Face-Pose-Selection on Embedded Vision Systems, US Patent No. 10,510,157, Issued, Dec 17, 2019.
  37. H. Ng, X. Wang, Y. Gao, R. Ma, and Y. Lu, Enhanced Face-Detection and Face-Tracking for Resource-Limited Embedded Vision Systems, US Patent No. 10,691,925, Issued: Jun. 23, 2020.
  38. X. Wang, M. Seyfi, M. Chen, H. W. Ng, Jiannan Zheng, and J. Liang, Age and Gender Estimation Using Small-Scale Convolutional Neural Network (CNN) Modules for Embedded Systems, US Patent No. 10,558,908, Issued, Feb. 11, 2020.
  39. Z. Yi, X. Wang, H. W. Ng, S. Ma, and J. Liang, High-Quality Training Data Preparation for High-Performance Face Recognition Systems, US Patent No. 10,943,096, Issued, Mar. 9, 2021.
  40. Y. Gao, E. Honsch, R. Ma, C. Shen, M. Chen, Y. Lu, J. Liang, and J. Wu, High-Performance Visual Object Tracking for Embedded Vision Systems, US Patent No. 11,205,274, issued, Dec. 21, 2021.
  41. J. Zheng, C. Shen, D. Zhang, and J. Liang, Video-based Fall Risk Assessment System, US patent application 16/731,025, Dec. 30, 2019. Publication Date: Jul 2, 2020.
  42. A. Au, D. Zhang, C. Chan, and J. Zheng, Method and System for Privacy-Preserving Health Monitoring, US patent ap-plication 17/408,490, Aug. 23, 2021.
  43. C. Chan, D. Zhang, Y. Gao, A. Au, Z. DeVries, J. Liang, Privacy-preserving human action recognition, storage, and retrieval via joint edge and cloud computing, US Patent Application 17522901, Nov. 9, 2021.
Preprints 95159 g001
Figure 1. The system architecture of the AltumView Sentinare smart activity sensor.
Figure 1. The system architecture of the AltumView Sentinare smart activity sensor.
Preprints 95159 g001
Preprints 95159 g002
Figure 2. The main block diagram of the visual sensor’s algorithms.
Figure 2. The main block diagram of the visual sensor’s algorithms.
Preprints 95159 g002
Preprints 95159 g003
Figure 3. Human keypoints and skeleton.
Figure 3. Human keypoints and skeleton.
Preprints 95159 g003
Preprints 95159 g004
Figure 4. Two-level CNN approach for action recognition.
Figure 4. Two-level CNN approach for action recognition.
Preprints 95159 g004
Preprints 95159 g005
Figure 5. Integration between the Sentinare sensor and Amazon Alexa Together.
Figure 5. Integration between the Sentinare sensor and Amazon Alexa Together.
Preprints 95159 g005
Table 1. Fall detection accuracy on CNRS dataset, Setup 1.
Table 1. Fall detection accuracy on CNRS dataset, Setup 1.
[2] Ours
2-fold / 4-fold 2-fold / 4-fold
Frame-level accuracy 75 % / 88 % 83.53% / 86.61%
Event-level accuracy Not reported 100% / 100%
Table 2. Fall detection accuracy on CNRS dataset, Setup 2.
Table 2. Fall detection accuracy on CNRS dataset, Setup 2.
[3] Ours
Sensitivity 90% 97.92%
Specificity 89.6% 100%
Table 3. Fall detection accuracy on UMontreal dataset.
Table 3. Fall detection accuracy on UMontreal dataset.
[4] [5] [6] [7] [8] [9] Ours
Sensitivity 91.60% 93.7% 89.40% 80.60% 95.40% 91.30% 93.48%
Specificity 93.5% 92.0% 93.23% 100% 95.80% 91.67% 98.59%
Accuracy 90.6% 89.7% 90.1% 96.76% 93.07% 89.06% 97.74%
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
Prerpints.org logo

Preprints.org is a free preprint server supported by MDPI in Basel, Switzerland.

facebook logotwitter logolinkedin logo
MDPI Initiatives
Important Links
Subscribe

Disclaimer

Terms of Use

Privacy Policy

© 2024 MDPI (Basel, Switzerland) unless otherwise stated