1. Introduction

2. Related Studies

3. Pilot System

3.1. System Configuration

As illustrated in Figure 1, the hardware portion of the system is comprised of several crucial components: a computer, a projector (wheelchair-compatible), an eye-tracking module (Tobii Eye Tracker 5), a control module (Arduino Nano, IoT module of ESP-8266, HX1838 infrared receiver, and infrared emitter, etc.).

The system employs the eye tracker for real-time capturing of eye movement data, specifically targeting individuals affected by head tremors. The captured data undergo virtual stabilization processing, accurately identifying the stable gaze position of the eyes. Furthermore, the system has designed interactive interfaces for various tasks, such as the virtual hand and bubble. These features enable individual suffering from tremors to interact with distant targets solely through their eye movements.

Figure 1. Configuration of SAR system.

Figure 1. Configuration of SAR system.

3.2. Assistive User Interface

3.2.1. Virtual Hand

Using fingers for direct interaction is an intuitive and natural human behavior. However, individuals with tremor often struggle with manual dexterity due to involuntary bodily tremors. These tremors can significantly hinder their ability to engage with objects at a distance. In response, exploring the use of alternative limbs for interaction can help overcome these limitations, enabling stable and remote interactions.

To realize this objective, a system incorporating a projected virtual hand has been meticulously designed to act as a surrogate for physical limbs, thereby aiding individuals with tremors in conducting remote interactions with objects. As depicted in the Figure 2, the individual with trembling is seated in a wheelchair equipped with both a projector and an eye-tracking device. By adjusting the wheelchair's orientation and deploying the eye-tracker, he/she can effectuate control over a virtual hand through ocular movements, thereby enabling nuanced interaction with distant objects.

It is imperative to highlight that the eye-tracker functions by recording the coordinate positions of the eye's pupil, subsequently determining the virtual hand's coordinate positions through spatial transformation techniques. This process facilitates the registration of the virtual hand within the physical space through a mapping mechanism between the virtual hand and the actual environmental space. Nevertheless, tremors in the head region could introduce interference in the signals captured by the eye tracker. This interference, once transmitted to the virtual hand, may be amplified, resulting in visual instability of the virtual hand. To mitigate the adverse effects of head tremors on the stability of the virtual hand, a virtual stabilization algorithm has been incorporated, with its specifics to be elaborated in the later section. Through the implementation of this innovative technique, subjects can seamlessly control a stable virtual hand for interaction purposes. When combined with the wheelchair configuration, this approach markedly extends the interaction radius accessible to individuals with tremors.

3.2.2. Virtual Confirmation

To enhance the precision of daily interactive inputs in our system, we have innovated a ‘bubble confirmation’ mechanism. This feature is particularly useful in standard operations, such as button clicking, where users traditionally confirm their inputs through prolonged pressing. Mirroring this in a virtual environment, our system introduces a bubble prompt that activates when the virtual hand or pointer hovers over a target. The process, illustrated in Figure 3, is straightforward: upon positioning the virtual hand or pointer over a target, a bubble encircles it. Inside this bubble, a blue dial gradually appears, filling the bubble in a sector-filling fashion over a set time period. For instance, when interacting with a television, this process spans approximately 3 seconds. This duration is meticulously calibrated to ensure not only input accuracy but also the fluidity of the interaction. Should the virtual hand or pointer deviate from the target, the bubble reverts to its original state. This bubble mechanism is a key component in our system, reliably ensuring precise confirmation and gaze input for interactions with virtual objects.

3.2.3. Assistive Interfaces

To facilitate easier operation of household appliances for individuals with tremor, some assistive interfaces was designed. This interface suite aims to simplify interaction steps, making the operation process more intuitive and straightforward. Taking television operation as an illustrative example, an assistive interface, as demonstrated in Figure 3, was created. To ease the operation, this interface retains only the frequently used function modules, such as volume adjustment, channel switching, and the confirmation button. The assistive interface is projected around the television through a projector installed on a wheelchair. Individuals with tremors can interact with the assistive interface by controlling virtual hands and bubbles through their eye movements. All interaction information is transmitted in real-time to the television via microcontrollers and IoT modules, thereby controlling the television. This design can replace traditional remote control input methods, providing convenience for individuals with tremor who may find operating traditional controls challenging.

Figure 3. A virtual bubbles and TV assistive interface.

Figure 3. A virtual bubbles and TV assistive interface.

While such assistive interfaces represent a significant advancement, they still require careful customization. This customization must be based on the specific eye movement interaction patterns, environmental contexts, and the functionalities of household appliances, with a particular focus on the needs of individuals with tremors. Such a tailored approach is crucial to enable them to effortlessly control various home appliances, thereby significantly enhancing their daily living experience and independence.

4. Experience and Results

4.1. Evaluating the Virtual Stabilization of SAR System

This study aims to validate the virtual stabilization algorithm, exploring its efficacy in system assisting individuals with trembling to interact stably and efficiently with remote objects through eye movement control, such as virtual hand, pointer, and bubble.

4.1.1. Experimental Setup

As illustrated in Figure 5, participants were tasked with controlling a virtual pointer through eye movement, aiming to rapidly and accurately align the pointer with randomly appearing red balls in physical space. The experimental procedure encompassed several key steps: At the beginning of each trial, a red dot was randomly generated on the interface. Participants were required to maintain the eye-controlled bubble pointer within the collision range of the red dot for three seconds, after which the red dot would disappear and reappear at a new location. Each participant repeated this task five times. Ten participants aged between 20 and 80 years were recruited for the study. Prior to the experiments, the simulator's vibration frequency and amplitude were precisely calibrated to mimic the characteristics of real head tremors.

Two experimental conditions were set up: one with the system equipped with the virtual stabilization algorithm and the other without it, to compare the effects. The system automatically recorded the coordinates of the pointer and the target position, as well as the task completion time during each trial. Upon completion of the experiments, subjective assessments were collected from the participants to measure their personal experiences and perceptions of the experiment. These assessments were rated on a seven-point Likert scale, ranging from -3 (strongly disagree) to +3 (strongly agree), allowing participants to express their level of agreement or disagreement with each statement. The assessment questions included:

Q1: The movement of the virtual pointer was entirely controlled by my eyes.

Q2: I experienced stability while controlling the movement of the virtual pointer with my eyes.

Q3: I was capable of accurately targeting objects with the eye-controlled virtual pointer.

4.1.2. Result

Efficiency evaluation with virtual stabilization: A key finding of our research was the significant reduction in time spent on tasks when the virtual stabilization system was employed. The paired sample t-test demonstrated a statistically significant decrease in time (p=0.000247) with the implementation of the system. Specifically, as illustrated in Figure 6 (a), the mean time spent without the system was 94.31 seconds (SD = 29.21), which was significantly reduced to 36.72 seconds (SD = 8.43) with the system. The effect size, calculated as Cohen's d, was a substantial 2.68, highlighting the system's effectiveness in minimizing task completion time.

Stability evaluation with virtual stabilization: The mean distance between the pointer position and the target position was obtained by randomly sampling of users' process of targeting the red ball. The Wilcoxon signed-rank test revealed a statistically significant reduction when the virtual stabilization system was active (p =0.0039) (Figure 6 (b)). The results indicated that the mean distance covered was notably less when the system was utilizing the virtual stabilization algorithm compared to when it was not in use, further confirming the system's role in enhancing operational precision and stability.

The statistical analysis of participant responses to three key questions further underscored the significant improvement in user experience in terms of control, stability, and accuracy with the virtual stabilization system. For Question 1(Q1), which assessed the extent to which participants felt the movement of the virtual pointer was controlled by their eyes, the mean difference in responses was 3.3 (95% CI: 0.09 to 6.51), with a significant p-value of 0.000043 (Figure 7). This indicates a considerably higher sense of control when using the system with virtual stabilization.

Similarly, for Question 2 (Q2), regarding the stability experienced in controlling the virtual pointer's motion, the mean difference was 4.4 (95% CI: 1.17 to 7.63), with an even lower p-value of 0.0000045 (Figure 7). This result underscores a marked improvement in perceived stability with the stabilization feature.

Question 3 (Q3), which focused on the participants' ability to accurately target objects using the eye-controlled virtual pointer, also showed a significant mean difference of 4.1 (95% CI: 0.82 to 7.38) and a p-value of 0.000009 (Figure 7). This finding suggests that the virtual stabilization system substantially enhanced the accuracy of eye-controlled targeting.

Figure 6. Comparison of (a) mean time spent and (b) mean distance between the pointer and the target in tasks under the condition of system with and without virtual stabilization (** p < 0.01).

Figure 6. Comparison of (a) mean time spent and (b) mean distance between the pointer and the target in tasks under the condition of system with and without virtual stabilization (** p < 0.01).

Figure 7. Comparison of mean score of Q1 to Q3 in the condition of system with and without virtual stabilization (** p < 0.01).

Figure 7. Comparison of mean score of Q1 to Q3 in the condition of system with and without virtual stabilization (** p < 0.01).

4.2. Evaluating the System-assisted Effectiveness in Complex Interactive Tasks

This study aims to validate the effectiveness of our developed AR system in assisting individuals with tremor in operating everyday household appliances, with a particular focus on enhancing the fluidity and stability of operations. While television control may seem a common and straightforward task in daily life, it indeed comprises a variety of complex interactive tasks, such as switching functions, confirming selections, and returning to previous menus. Patients with tremor disorders often face challenges in smoothly completing these tasks using traditional control devices, like TV remotes, due to hand instability. Given the representativeness and ubiquity of television control, it serves as an ideal test case for assessing the efficacy of the AR system in everyday interactive tasks.

4.2.1. Experimental Setup

The design of our experiment was as follows: We selected several common TV control tasks, such as volume adjustment, channel switching, and confirmation. These tasks were arranged in a random sequence to create 10 sets of TV control task sequences. We recruited 10 participants aged between 20 and 65 years to complete these tasks. Each participant's primary task involved using the AR system and the conventional remote control to complete a randomly selected set of TV control tasks. For clarity in subsequent analysis, specific terminologies were designated for the experimental conditions: interactions facilitated by the AR system were termed ‘System-Assisted’, while those involving the conventional remote control were labeled ‘Manual Interaction’. To simulate the trembling finger manipulation typical of tremor patients when using a traditional remote control, we employed a hand tremor simulator, detailed in “a dual channel electrical stimulation instrument for simulating trembling limbs” [22].

Upon completion of the tasks, participants were asked to respond to a series of subjective questions based on their experience. Responses were rated on a 7-point scale, which ranged from -3 (very poorly matched) to 3 (perfectly matched). Through these responses, we aimed to assess the potential effects of the AR system in enhancing operational accuracy and user experience.

Q4. In this condition, channel-switching and volume adjustment could be few mis-operations.

Q5: In this condition, channel-switching and volume adjustment could be easily performed.

Q6: In this condition, channel-switching and volume adjustment were performed stably.

4.2.2. Result

In the investigation of the AR system's ability to provide smooth control and stability for individuals with tremors during complex interactive tasks, our statistical analysis utilized a paired t-test to compare the ‘System-Assisted’ and ‘Manual Interaction’ conditions. Our analysis revealed a statistically significant improvement in the number of mis-operations during channel-switching and volume adjustment tasks when participants used the AR system as opposed to a conventional remote control. This was evident from the responses to Question 4 (Q4), which showed a mean difference of 3.6 (95% CI: 0.19 to 7.01, p = 0.000035) (Figure 8). This underscores the potential of the AR system to enhance precision in control tasks that are typically challenging for individuals with tremor.

However, when evaluating the ease of performing the same tasks, as assessed in Question 5 (Q5), the statistical analysis did not demonstrate a significant difference between the two conditions (mean difference: -0.2, 95% CI: -2.98 to 2.58, p = 0.619) (Figure 8). This suggests that participants did not perceive a change in the difficulty level when using the AR system compared to the manual method.

In regard to the stability of operation, as queried in Question 6 (Q6), while the mean difference in responses was 1.3, suggesting an improvement with the ‘System-Assisted’, the results did not achieve statistical significance (95% CI: -3.23 to 5.83, p = 0.070) (Figure 8). This outcome implies a potential trend towards increased stability with the assistance of the AR system; however, the data does not allow for a definitive conclusion, indicating the need for further research with a larger sample size or refined experimental conditions.

Figure 8. Comparison of mean score of Q4 to Q6, in the condition of ‘System-Assisted’ and ‘Manual Interaction’ (** p < 0.01).

Figure 8. Comparison of mean score of Q4 to Q6, in the condition of ‘System-Assisted’ and ‘Manual Interaction’ (** p < 0.01).

4.3. Evaluating the System-Assisted Effectiveness in Distant Interactive Environments

This experiment aims to explore and validate the potential of AR technology in assisting individuals with tremors in remote communication and interaction. For individuals affected by tremors, particularly those impacting hand stability, accurately indicating and interacting with distant objects poses a significant challenge. Limited by their physical mobility, these patients often rely on the assistance and communication with others for interacting with objects that are out of reach, such as needing help to retrieve items located at a distance. The complexity of real-life environments, where objects may be obscured, closely packed, or small, further complicates this task. The experiment involves a pointing and guessing game, created to test the effectiveness of the AR system in complex situations. The main objectives are to determine if the AR system enhances their ability to accurately and efficiently point at objects, and to assess how the system aids their communication with others.

4.3.1. Experimental Setup

In our experimental setup, we carefully selected 10 objects of various sizes commonly found in households, including items like cardboard boxes, bottled water, apples, and wall paintings (Figure 9). Each object was assigned a unique numerical identifier to facilitate the experiment's various stages. Spatially, five objects were placed on an upper tier, four on a lower tier, and the wall painting was hung on an adjacent wall. To increase the complexity of the task and more closely simulate a real-life environment, smaller objects were positioned in front of the larger ones on both tiers, creating a partial visual obstruction. All objects were strategically positioned 5 meters away from the participants, covering a horizontal range of 0 to 0.5 meters and a vertical range of 0 to 1 meter.

The task involved a sequence of numerically coded objects presented in a randomized order. As illustrated in Figure 9, the experiment was conducted in 10 rounds, each involving two participants. One participant, using a hand tremor simulator or using the AR system, was to point to the distant objects. The other participant's role was to guess and note the numerical codes of the objects indicated. Upon the conclusion of each round, we compared the recorded numerical sequences of both participants, focusing on the discrepancies and the total time taken for each pointing task. Additionally, participants provided feedback on their experience, rating it on a 7-point scale at the experiment's conclusion. The scale ranged from -3 (very poorly matched) to 3 (perfectly matched), offering insights into their subjective experiences. The following questions guided their feedback:

Q7: In this case, I think I can more easily point to distant objects.

Q8: In this situation, I feel that I can stably point to the object.

Q9: In this case, I can easily understand the object being referred to.

Q10: In this situation, I feel that communication is easy.

Figure 9. This is a figure. Evaluation of system assistance in a distant interaction environment: (a) a participant directly indicating distant items; (b) a participant using system assistance to indicate distant items.

Figure 9. This is a figure. Evaluation of system assistance in a distant interaction environment: (a) a participant directly indicating distant items; (b) a participant using system assistance to indicate distant items.

4.3.2. Result

As illustrated in Figure 10, the analysis of the data revealed that under the support of the system, participants were able to achieve a median accuracy rate of 90%, with the first quartile at 80% and the third quartile reaching 100%. This high level of accuracy demonstrates the system’s effectiveness in aiding participants to overcome challenges posed by obscured, closely packed, or small objects. Furthermore, the average time taken to complete the task showed a median of 3.65 seconds, with the bulk of the experiments falling between 3.0 seconds (first quartile) and 5.225 seconds (third quartile). This indicates a relatively quick response time in completing the tasks, highlighting the system’s role in facilitating efficient interactions.

Notably, our analysis did not find a significant direct correlation between the average time spent on tasks and the level of accuracy achieved. This indicates that the current level of accuracy was not attained at the expense of prolonged interaction duration.

In scenarios where the system was not used, participants were unable to successfully complete the tasks. This contrast highlights the critical role of the system in enabling precise and efficient interaction with remote objects. Overall, the experimental results demonstrate the system’s effectiveness in significantly enhancing remote communication for individuals with tremors.

Question 7 (Q7) addressed the ease of performing remote pointing tasks. Participants noted that pointing at distant targets was easier when using an AR assisted system. Statistical analysis revealed a significant improvement in remote pointing ability with system assistance, with a mean difference of 4.4 (95% CI: 1.35 to 7.45, p = 0.00000278) (Figure 11). This underscores the enhanced precision in locating distant objects by individuals with tremor with the aid of the system.

Question 8 (Q8) focused on the stability of participants while pointing at objects. The statistical outcomes indicated a significant aid provided by the system for tremor individuals in performing stable pointing actions (mean difference of 3.2, 95% CI: -1.89 to 8.29, p = 0.0015) (Figure 11). This demonstrates that the virtual hand and pointer interface offered by the system can be controlled steadily by individuals with tremor.

Question 9 (Q9) explored the clarity of the pointing actions. Participants reported that it was easier to understand the indications given by tremor individuals in a system environment (mean difference of 3.2, 95% CI: -1.42 to 7.82, p = 0.00079) (Figure 11). This highlights the effectiveness of using a virtual hand for remote interaction and pointing in the system setting.

Question 10 (Q10) examined the comparative ease of remote communication using the AR system versus traditional methods. Responses indicated that the convenience of communication significantly improved with the use of AR technology (mean difference of 4.0, 95% CI: -0.13 to 8.13, p = 0.0000685) (Figure 11), suggesting that the system facilitates clearer communication in interactive tasks. These results collectively emphasize the important role of AR systems in enhancing interaction precision and convenience in complex environments.

Figure 10. Assessing participants’ performance in complex remote interaction tasks under system support: accuracy and average time spent.

Figure 10. Assessing participants’ performance in complex remote interaction tasks under system support: accuracy and average time spent.

Figure 11. Comparison of mean score of Q7 to Q10 in the condition of ‘System-Assisted’ and ‘Manual Interaction’ (** p < 0.01).

Figure 11. Comparison of mean score of Q7 to Q10 in the condition of ‘System-Assisted’ and ‘Manual Interaction’ (** p < 0.01).

5. Discussion

6. Application

6.3. Accessible Communication

Individuals with tremors often face challenges in movement and reaching items at height due to involuntary body shakes and limited mobility, such as wheelchair use, significantly impacting their daily communication. Discussing items located afar or at elevated positions becomes particularly challenging without assistance. For instance, discussing a painting hung high on a wall or instructing someone to retrieve an item from a distant location can be difficult for those with tremors.

However, these communication barriers are effectively eliminated with the use of our system. As illustrated in Figure 14, an individual with tremor can control a stable virtual hand through eye movements to communicate about a painting located at a height. As shown in Figure 9, they can also direct others to retrieve items by controlling the virtual hand. Additionally, when our system is integrated with wheelchair configurations, it enables individuals with tremors to easily navigate and reach various corners within the home, facilitating barrier-free communication. This spatial prompting method simplifies the communication process and enhances clarity and efficiency.

The application of the system extends beyond the aforementioned scenarios to a wider range of living situations, including air conditioning temperature adjustment, automated curtain control, and kitchen appliance management. The system can be deeply customized and optimized based on users' specific needs, eliminating many interaction barriers in daily life and expanding the interaction range for individuals with tremors. By reducing the difficulty of daily tasks and decreasing reliance on external assistance, this system opens a new path for tremor patients towards a more independent, efficient, and comfortable lifestyle.

Figure 14. An individual with tremor engaging in daily communication with a companion using the SAR System, discussing distant objectives.

Figure 14. An individual with tremor engaging in daily communication with a companion using the SAR System, discussing distant objectives.

7. Conclusion

References

1. Louis E D; Faust P L. Essential tremor: the most common form of cerebellar degeneration. Cerebellum & ataxias, 2020, 7, 1-10.. [CrossRef]
2. Lenka A; Jankovic J. Tremor syndromes: an updated review. Frontiers in Neurology, 2021, 12, 684835. [CrossRef]
3. Bain P G. Parkinsonism & related disorders. Tremor. Parkinsonism & Related Disorders, 2007, 13, S369-74. [CrossRef]
4. Louis E D; McCreary M. How common is essential tremor? Update on the worldwide prevalence of essential tremor. Tremor and Other Hyperkinetic Movements, 2021, 11. [CrossRef]
5. Welton T; Cardoso F; Carr J A, et al. Essential tremor. Nature Reviews Disease Primers, 2021, 7(1), 83. [CrossRef]
6. National Institute of Neurological Disordersand Stroke. Tremor. https://www.ninds.nih.gov/health-information/disorders/tremor. Accessed 10 Dec 2023.
7. Louis E D. Treatment of essential tremor: are there issues we are overlooking. Frontiers in neurology, 2012, 2, 91. [CrossRef]
8. Zesiewicz T A; Elble R J; Louis E D, et al. Evidence-based guideline update: treatment of essential tremor: report of the Quality Standards subcommittee of the American Academy of Neurology. Neurology, 2011, 77(19), 1752-1755. [CrossRef]
9. Dallapiazza R F; Lee D J; De Vloo P, et al. Outcomes from stereotactic surgery for essential tremor. Journal of Neurology, Neurosurgery & Psychiatry, 2019, 90(4), 474-482. [CrossRef]
10. Carmigniani J; Furht B. Augmented reality: an overview. Handbook of augmented reality, 2011, 3-46.
11. Wang K; Takemura N; Iwai D, et al. A typing assist system considering involuntary hand tremor. Transactions of the Virtual Reality Society of Japan, 2016, 21(2), 227-233.
12. Wang K; Iwai D; Sato K. Supporting trembling hand typing using optical see-through mixed reality. IEEE Access, 2017, 5, 10700-10708. [CrossRef]
13. Zhang X; Liu X; Yuan S M, et al. Eye tracking based control system for natural human-computer interaction. Computational intelligence and neuroscience, 2017, 2017. [CrossRef]
14. Donaghy C; Thurtell M J; Pioro E P, et al. Eye movements in amyotrophic lateral sclerosis and its mimics: a review with illustrative cases. Journal of Neurology, Neurosurgery & Psychiatry, 2011, 82(1), 110-116.. [CrossRef]
15. De Gaudenzi E; Porta M. Towards effective eye pointing for gaze-enhanced human-computer interaction. 2013 Science and Information Conference. IEEE, 2013, 22-27.
16. Nehete M; Lokhande M; Ahire K. Design an eye tracking mouse. International Journal of Advanced Research in Computer and Communication Engineering, 2013, 2(2), 1118-1121.
17. Missimer E; Betke M. Blink and wink detection for mouse pointer control. Proceedings of the 3rd international conference on pervasive technologies related to assistive environments. 2010, 1-8.
18. Shahid E; Rehman M Z U; Sheraz M, et al. Eye Monitored Wheelchair Control. 2019 International Conference on Electrical, Communication, and Computer Engineering (ICECCE). IEEE, 2019, 1-6. Date Added to IEEE Xplore: 27 December 2019 ISBN Information:. [CrossRef]
19. Plaumann K; Babic M; Drey T, et al. Improving input accuracy on smartphones for persons who are affected by tremor using motion sensors. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2018, 1(4), 1-30. [CrossRef]
20. Wacharamanotham C; Hurtmanns J; Mertens A, et al. Evaluating swabbing: a touchscreen input method for elderly users with tremor. Proceedings of the SIGCHI conference on human factors in computing systems. 2011, 623-626.
21. Wang K; Tan D, Li Z; et al. Supporting Tremor Rehabilitation Using Optical See-Through Augmented Reality Technology. Sensors, 2023, 23(8), 3924. [CrossRef]
22. Xu C; Zhang X; Guan Q, et al. Analysis of the electrophysiological features of tremor and essential tremor in Parkinson’s disease. Chinese Journal of Practical Nervous Diseases, 2017, 20(21),16-20.