Preprint

Article

AR Coupled With a Virtual Agent for Educating Towards a Sus-Tainable Way of Living

Altmetrics

Downloads

151

Views

Comments

Tina Katika^*

,Spyridon Nektarios Bolierakis,

Thomas Papaioannou,Ioannis Karaseitanidis,

Georgios Tsimiklis,

Angelos Amditis

Tina Katika^*

,Spyridon Nektarios Bolierakis,

Thomas Papaioannou,Ioannis Karaseitanidis,

Georgios Tsimiklis,

Angelos Amditis

This version is not peer-reviewed

Submitted:

24 April 2023

Posted:

25 April 2023

You are already at the latest version

Alerts

Abstract

Augmented reality (AR) offers an accessible, inexpensive, and rich user experience that has the potential to engage end-users in an immersive environment. Along with vivid visualizations coupled with virtual agents, this technology further develops learning interest in end-users, guides them in various tasks, and boosts motivation and productivity. In this study, we leverage the deep penetration of mobile phones in daily lives and their advanced features to design, develop and demonstrate an AR application (CirculAR) that offers unique user-environment interaction in a gamified way. CirculAR combines learning with enjoyment to help end-users understand fundamental sustainability and circular economy principles. The application has been showcased in a controlled environment to heterogenous audiences and has been shown to improve end-user engagement and motivation. First, participants older than 18 were recruited to showcase the technology acceptance and engagement towards circular economy principles through AR. Then, students aged 5 to 15 years old, along with their parents and educators, were invited to a treasure hunt game where our virtual agent ARis guided them through a map full of virtual experiences. Assessment and evaluation were performed through a survey and a questionnaire. The outcome of their analysis showcased an increase in the dedication and enjoyment of the performed activities, engagement and learning attributes given the AR virtual agent supporting functionalities. Observations during showcasing reported a need for more commitment from the younger audience compared to the older one. This application contributes to the discourse on mobile AR as a tool for the education of novel concepts with a high impact on our daily lives and decisions and aims to shed light on the design principles of educative tools.

Keywords:

Subject: Computer Science and Mathematics - Computer Vision and Graphics

1. Introduction

As human beings, we experience our surroundings and environment through plain observation rather than active exploration and participation. Such an experience limits the engagement potential and, thus, the motivation to understand how specific interactions and behaviors may impact our everyday lives. Complex phenomena, such as sustainable living economy concepts (e.g., circular economy), need to be supported by behavioral, social, and cultural changes [1]. Especially for children such changes need the use of simple and vivid language to explain and communicate complex affairs and global issues that will considerably impact their lives [2].

AR’s immersive nature helps its audience perceive evolving details in action and connect events between a virtual story and their own lives. AR used in the educational context, offers an interface between reality and the perception of reality, bridging the gap between real and virtual worlds and thereby offering the necessary means for enabling end-users of all ages to move from observation to immersion. The most robust case for immersive technologies as learning modules arises from their ability to implement contexts and relationships impossible in a traditional learning setting [2].

AR may reduce challenges in education thanks to its potential to improve user’s sensory perception of the real world by presenting phenomena that are difficult to observe in reality [3]. Thereafter, AR has been widely used in education and other fields such as medicine, advertising, defense, military, and entertainment, evolving yearly. In education, it has brought a new dimension as it offers possibilities for learning experiences and the teaching process by creating an interactive and immersive environment. Thus, AR provides a layer of flexibility in learning by providing an interactive environment adaptable to the real-world environment [4].

Since the introduction of AR, researchers have identified its potential to communicate environmental issues taking advantage of its ability to stem education through gamified mechanisms. Klopfer & Squire, [5], exploited the unique educational affordances of AR, including portability, social interactivity, context sensitivity, connectivity, and individuality, to demonstrate its educational potential in the field of environmental sciences. Since then, technology in the field of AR has evolved, and other researchers have explored the possibility of communicating complex social phenomena to children to improve their socialization, communication skills, and emotional intelligence [6]. Overall, in the educational context, AR has contributed to STEM learning by awakening interest, increasing motivation, and, most importantly, capturing the students’ attention [7].

In recent years, researchers have investigated the use of AR technology to support the various learning approaches in education, such as multi-user and collaborative [6,8], game-based [9], and inquiry-based learning [10]. Virtual agents and bots have emerged to assist end-users in performing tasks supported by AR and also improve social interactions and motivation [11]. In children’s education, virtual agents have proven to help a participant in a learning task and promote collaborative learning by enhancing emotional abilities [12]. As Iqbal et al. [12] demonstrated in their latest systematic review, AR agents are new and have not been effectively implemented in any AR studies for education, as discussed in detail.

In the field of virtual education, it has been demonstrated that some functionalities are required to ensure an immersive and engaging environment when developing such education and training systems. These functions include content creation and manipulation, information modeling, visualization and simulation, sensor and real-world representation, and graphical user interaction [13]. At the same time, pedagogical agents embodied in virtual learning experiences can improve end-user’s learning experience, enhance their engagement with learning activities, and improve their knowledge construction and performance [14].

Despite some isolated efforts mentioned above, and even though AR research has evolved, and education and learning has contributed significantly from its progression, an immersive environment in AR has not been proposed that benefits from a virtual agent and advanced gamification features, teaching end-users of all ages complex social and technical matters, such as circular economy and sustainability [1].

This paper describes CirculAR, a mobile AR application, as an innovative way to support learning by helping end-users to understand complex phenomena such as water scarcity and circular economy. CirculAR end-users can navigate through the map, retrieve digital content, alter their environment in real-time through content manipulation, receive guidelines and visual and audio hints, and test their knowledge. Gamification features, including scoring, badges, and awards, ensure end-users remain immersed in the game and learn through joyful tasks.

The paper is organized as follows: the next section presents the system design and development effort, next follow the most dominant educational and gamification features, a description of the showcase, and the evaluation methodology. Before concluding and describing the future research efforts, we present the results and discussion from our analysis.

2. Materials and Methods

Since the seamless interaction of the end-users with their environment is the prerequisite for improving engagement, motivation, and self-efficacy, CirculAR supports triggered AR content in three ways [15]. The location-based AR functionality uses the device’s GPS location as a trigger to pair dynamic location with Points of Interest (PoIs) to reveal the content. The marker-based function requires a marker (QR) to activate an augmentation related to 2D or 3D items. Finally, the object detection AR functionality is a dynamic augmentation process based on trained Deep Learning Models that enable a more robust interaction with the environment [16].

Figure 1 shows the architecture of the mobile AR application that contains three major components, both the input layer and output engine, as well as the database. CirculAR accesses the location of the device, and using marker and object-based triggers, it imports both location and images into the Content Management System to compare with the database, and then the output engine outputs the matching content on the screen of the mobile devices.

2.1. System design

As explained by [17], the overall system architecture is designed to support two different entities, the AR Content Management System (AR CMS) and CirculAR. The AR CMS end-users are the content owners who produce and add content to the platform to create meaningful and educational experiences, namely the AR campaigns. They also decide the connection of the content with the markers, GPS location, and trained machine learning models. The campaigns are later enhanced with various gamification aspects before being visualized by the mobile application end-users. The platform and the two components have been presented by Katika et al., [1] and [16]. The virtual agent (ARis) is present in most functionalities and offers helpful tips and comments upon execution.

2.2. System model

The AR app has been developed on Android and iOS operating systems. Upon downloading and installing CirculAR, the end-users start their journey with login or registration, followed by tutorial and help functions. The app allows profile editing through the selection of an avatar and a username. As the end-users read their mission, they can go through a demo campaign that offers a guide on how to use all the features, including the selection of content, navigation to the nearest AR experience, interaction with the environment through the camera and object manipulation, object recognition, and the gamification and educative features [1,16]. Later, a map is available for exploration, as shown in Figure 3. While on the map, the end-user overlooks the available AR experiences and can navigate on the PoI of their choice. As soon as the end-user reaches the desired PoI the camera lens is activated. The user then can interact with their environment through different means. Figure 4 describes the options for enabling an AR experience through object detection, GPS activation, and QR scanning [16].

Upon enabling an AR experience through their smartphone’s camera, the app gives access to digital content that can be manipulated using finger touches and gestures. The end-users get access to textual information and quiz questions along with the digital content. Their score is improved based on specific actions: visiting a PoI, answering a quiz successfully, and finishing an entire AR campaign. A leaderboard demonstrates their ranking among their peers. Figure 5 shows these features and functionalities.

The complete user flow is shown in the semantic overview of Figure 6. Five phases have been identified, including initiation and profile editing, exploration and AR visualization, scoring and gamification.

2.3. Development in AR

The software specifications have been described in Katika et al., [1]. The object detection algorithm has been described by Katika et al., [16]. For the treasure hunt game addressed to end-users, low-poly 3D models (three dimensional assets with few design polygons that are easier to visualize through smartphone AR technology) were designed to ensure real-time fast rendering, as shown in Figure 7.

3. Results

3.1. CirculAR education and gamification features

Considering the recommendations from Di Natale et al., [13], for the development of virtual systems, and the benefits of education due to pedagogical agents described in Grivokostopoulou et al., [14], among other functionalities developed for CirculAR, described in Katika et al., [18], a virtual agent has been embedded to offer helpful tips and comments upon execution. The virtual agent (ARis) is introduced upon logging in to the mobile AR app where the mission is presented to the end-user. A tutorial page describes the app’s functionality and places where the agent is expected to guide the end-user to ensure that they benefit from its presence. Figure 8 shows four UIs where ARis provides helpful tips to the end-users, urging them to interact with their environment in three ways (GPS tracking, QR scanning, and object recognition).

ARis is activated along with specific AR features of the mobile application (e.g., camera activation for content visualization and manipulation) and uses textual communication cues to interact with the end-user. Upon introducing ARis, it becomes apparent to the end-user that even if the interaction with the app becomes ambiguous, clicking on ARis will enable further steps. The virtual agent is programmed to react differently in all three user-environment interactions (GPS tracking, QR scanning, and object detection), shedding light on follow-up activities to unlock the attached media.

3.2. Designing and showcasing

To demonstrate CirculAR and receive feedback on the design features and functionalities, a treasure hunt game was created for students aged 5 to 15 years old, where end-users were called to enter a map and look for ARis, who aimed to educate about sustainability processes through virtual content. Figure 9 shows the creation of the AR campaign and curation of the virtual content through the AR CMS prepared to ensure an engaging activity for participants of all ages. The educational and gamification features and the object recognition functionality were also utilized to ensure an even more immersive experience. During the 3-day demonstration, feedback was continuously gathered (more than 2000 students from various schools visited the exhibitions). Interviews and discussions during the event ensured that the feedback was collected in a structured way.

Five researchers from our organization during the 3-day activity explained the game to the students, presenting their mission and goal when using CirculAR. The students were told that finding ARis across the room would identify interesting hints about sustainability practices and circular economy principles. The act of improving their score through quizzes was also described. The game was finalized when the end-user found ARis in all “hidden” spots and answered all quizzes associated with the virtual content revealed. As demonstrated by the app analytics, more than 250 students finalized the treasure hunt game. Figure 10 shows the app analytics.

3.3. Evaluation

The evaluation activities in the present study aimed to assess the effectiveness of the presence of the virtual agent. For the assessment activities, we considered the challenges in performing usability evaluations on AR systems described by Dünser & Billinghurst, [19]. As described, the evaluation heavily depends on the end-users’ characteristics (such as novices, experts, casual users, frequent users, children, adults, the elderly, and so on). Following the recruitment and demonstration from Katika et al., [1], we assessed how the presence of a virtual agent affected several engagement attributes of the participants involved. Information regarding the demographics and validation of measures can be found in the same paper. The evaluation activity included participants older than 18 years old. The survey assessing the AR virtual agent consisted of 6 items presented in Table 1. The Likert Scale was used to demonstrate the agreement of the participant with statements from “strongly agree” to “strongly disagree” (with intervals from 1 to 4 accordingly). At the same time, we assessed how the educational level and tech-savviness of the participants, based on their exposure to similar AR tools and mobile games, affected several attributes of the virtual agent (Table 2).

The participants were exposed to two functionalities of the virtual agent: a) helpful tips during GPS-enabled experiences, and b) guidance for scanning a QR code.

Based on the feedback of 127 participants, we assessed that all engagement attributes under investigation were positively affected by the presence of ARis during the entire AR experience. The highest trend was noted in the AR application’s perceived usability, which was significantly improved. Additionally, the participants declared improvement in their dedication to AR due to the presence of ARis. Figure 11 shows all the trends mentioned above.

We proceeded by further showcasing the impact of familiarity with AR and mobile games on the perception of the virtual agent. From our analysis presented in Figure 12, it was made clear that end-users more confident in both AR and mobile gaming were affected by the gender of the male-like virtual agent but also demonstrated higher dedication and enjoyment due to his presence. The preference towards a more human-like virtual agent did not show any significant fluctuation.

4. Discussion

Overall, our analysis showed that the AR application coupled with a virtual agent used to engage end-users towards sustainable practices and circular economy principles, could help its audience understand important concepts more intuitively, deepening their interest and understanding, helping their attention and confidence. Such a statement agrees with previous studies in AR-based motion-sensing teaching software used in experimental teaching of other complex concepts, such as magnetism [3]. Triggering end-users’ learning interest and endurability encourage them to learn more actively and extensively.

Moving on to the social aspects of digital learning environments, our evaluation’s outcome demonstrated that a virtual agent’s presence resulted in higher dedication and enjoyment in this educative setting. As previously demonstrated by Bailenson et al., [2], the presence of virtual others has an impact on learning since social cues significantly influence the act of learning. Similar to the outcomes drawn by Grivokostopoulou et al., [14] ARis demonstrated that the presence and behavior of a pedagogical agent had a substantial impact on students’ learning, considering that their perceived learning improved by almost 20%. At the same time, their overall engagement and learning experience, as described by all the attributes in Figure 11, significantly improved by the agent’s presence.

During the showcasing, we observed how students (below 18 years of age) approached the treasure hunt game. Unfortunately, the young audience found more interest in finding ARis than revealing the virtual content that the agent meant to disclose. Next, in order of importance, came the quiz answering that led to gaining more points upon correct answering, and contributed in becoming competitive with each other through the leaderboard. It became apparent that the students shifted the mission of CirculAR, from finding educative content to finding the virtual agent and earning the highest score. Considering how students approached the augmented reality platform developed for environmental simulations by Klopfer and Squier, [5], it became apparent that young audiences tend to be less emotionally engaged in the problem and thereafter, the design principles of an educative tool using AR technology should be further tailored to the end-users needs and capabilities. Possible introductory video or audio material, lectures or working groups, could familiarize the young end-users to the content and topic discussed through the AR app (in our case, sustainability and circular economy). Supporting this argument, Katika et al., [1] demonstrated that participants older than 18 years old participating in other demonstration activities, connected further to their mission and tasks through the app, since the topics augmented where part of their social and environmental agenda. CirculAR may require further tailoring to become an educative tool for end-users younger than 18 years old or be combined with other educational material to support.

5. Conclusions and Future studies

In this work, we reported designing and developing a user-engagement mobile AR application coupled with a virtual agent supporting AR functionalities and object detection. End-user evaluation of the virtual agent reported an increase in the dedication and enjoyment of the AR app and demonstrated an increase in engagement attributes. CirculAR demonstrated an increase in self-efficacy and perceived learning in the field of circular economy and sustainability, but further investigation requires the evaluation of its performance in other fields of study.

The contextually relevant tasks are recommended from the virtual agent to the user as an augmented message via text. Further implementation is required to enhance accessibility and inclusiveness in this field, as audio or other interfaces may allow the end-user to experience more virtual agent capabilities. Additionally, further research is required to understand whether the gender of the agent and the physical characteristics influence performance.

Considering the development of new educational technology genres, we should consider the young audience’s inability to fully comprehend this application’s mission and end-goal compared to older participants. The important to connect to the overall mission, in our case sustainability and circular economy, emphasized the need to observe learners’ interactions with the software before offering new designs for such an educative tool.

Finally, leveraging artificial intelligence and semantic understanding technology, the agent can transform into a virtual assistant to better understand user input, locate the required resource, and respond in an accurate, timely manner.

Funding

This research is based upon work supported by funding from the EU Horizon 2020 project under grant agreement No. 869474 (water-mining).

Data Availability Statement

No new data were created.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

References

Katika T., Karaseitanidis I., Tsiakou D., Makropoulos C., Amditis A., (2022) Augmented Reality (AR) Supporting Citizen Engagement in Circular Economy, Circular Economy and Sustainability. [CrossRef]
Bailenson, J.N. , Yee, N., Blascovich, J., Beall, A.C., Lundblad, N. and Jin, M., 2008. The use of immersive virtual reality in the learning sciences: Digital transformations of teachers, students, and social context. The Journal of the Learning Sciences, 17(1), pp.102-141.
Cai, S. , Chiang, F.K., Sun, Y., Lin, C. and Lee, J.J., 2017. Applications of augmented reality-based natural interactive learning in magnetic field instruction. Interactive Learning Environments, 25(6), pp.778-791.
Tang, K.S. , Cheng, D.L., Mi, E. and Greenberg, P.B., 2020. Augmented reality in medical education: a systematic review. Canadian medical education journal, 11(1), p.e81.
Klopfer, E. and Squire, K., 2008. Environmental Detectives—the development of an augmented reality platform for environmental simulations. Educational technology research and development, 56(2), pp.203-228.
López-Faican, L. and Jaen, J., 2020. EmoFindAR: Evaluation of a mobile multiplayer augmented reality game for primary school children. Computers & Education, 149, p.103814.
Ibáñez, M.B. and Delgado-Kloos, C., 2018. Augmented reality for STEM learning: A systematic review. Computers & Education, 123, pp.109-123.
Ke, F. and Hsu, Y.C., 2015. Mobile augmented-reality artifact creation as a component of mobile computer-supported collaborative learning. The Internet and Higher Education, 26, pp.33-41.
Hwang, G.J. , Wu, P.H., Chen, C.C. and Tu, N.T., 2016. Effects of an augmented reality-based educational game on students’ learning achievements and attitudes in real-world observations. Interactive Learning Environments, 24(8), pp.1895-1906.
Kyza, E.A. and Georgiou, Y., 2019. Scaffolding augmented reality inquiry learning: The design and investigation of the TraceReaders location-based, augmented reality platform. Interactive Learning Environments, 27(2), pp.211-225.
Miller, M.R. , Jun, H., Herrera, F., Yu Villa, J., Welch, G. and Bailenson, J.N., (2019) Social interaction in augmented reality. PloS one, 14(5), p.e0216290.
Iqbal, M.Z. , Mangina, E. and Campbell, A.G., 2022. Current Challenges and Future Research Directions in Augmented Reality for Education. Multimodal Technologies and Interaction, 6(9), p.75.
Di Natale, A.F. , Repetto, C., Riva, G. and Villani, D., 2020. Immersive virtual reality in K-12 and higher education: A 10-year systematic review of empirical research. British Journal of Educational Technology, 51(6), pp.2006-2033.
Grivokostopoulou, F. , Kovas, K. and Perikos, I., 2020. The effectiveness of embodied pedagogical agents and their impact on students learning in virtual worlds. Applied Sciences, 10(5), p.1739.
Edwards-Stewart, A. , Hoyt, T., Reger, G., (2016) Classifying different types of aug-mented reality technology. Annual Review of CyberTherapy and Telemedicine, 14, pp.199-202.
Katika, T. , Bolierakis S.N., Vasilopoulos E., Antonopoulos M., Tsimiklis G., Karaseitanidis I., Amditis A., (2022) Coupling AR with object detection neural networks for end-user engagement, 19th EuroXR International Conference, Stuttgart, Germany.
Katika, T., Bolierakis, S.N., Tousert, N., Karaseitanidis, I. and Amditis, S.A. (2021) Building a mobile AR engagement tool: evaluation of citizens attitude towards a sustainable future. 18th EuroXR International Conference. [CrossRef]
Katika, T., Karaseitanidis, I. and Amditis, S.A. (2022) Coupling Mobile AR with a Virtual Agent for End- User Engagement. [CrossRef]
Dünser, A. and Billinghurst, M., (2011) Evaluating augmented reality systems. In Handbook of augmented reality (pp. 289-307). Springer, New York, NY.

Figure 1. The overall architecture of CirculAR.

Figure 2. The system design of the AR components adapted from Katika et al., 2022 [1]. CirculAR supports three means of interaction of the end-user with their environment shown to the right side of the image: a) QR scanning that has been generated by the AR CMS during the media attachment, b) GPS localization that have been defined during the creation of AR campaigns, c) Object detection through recognition algorithms using the camera of the AR app.

Figure 3. The navigation functionality of CirculAR. The bottles represent the nearest AR experiences (PoIs). The end-user can follow the “you’re here” sign while moving towards the PoI of their choice.

Figure 4. The 3 ways the CirculAR end-user can interact with their environment; through object detection, GPS-based activation, and QR code scanning.

Figure 5. Educative and gamification features of CirculAR; the scoring card, the leaderboard, the AR enabled content, and the quiz question.

Figure 6. The user flow of a CirculAR end-user.

Figure 7. A low poly 3d item visualized through CirculAR.

Figure 8. a) The agent after login, and b) its functionality shown through tutorial. c) Prompting the end-user to assess an AR experience through QR scanning, and d) object recognition.

Figure 9. The AR CMS UI where the AR campaigns for the treasure hunt game are curated. Each one of the blue and orange bubbles in the middle of the map, contain virtual content (i.e., videos, images, 3D objects, questionnaires).

Figure 10. End-users participating and finalizing the treasure-hunt game showcased via CirculAR.

Figure 11. The effects of the presence of ARis on the a) enjoyment and b) dedication to AR related to the user engagement attributes.

Figure 12. The effects of AR and mobile game familiarity on the existing attributes of the virtual agent.

Table 1. Survey items assessing how the virtual agent (ARis) affected the AR experience.

Nr	Statement
1	I will enjoy AR more due to the presence of ARis
2	ARis will improve my dedication to AR
3	The gender of the virtual agent will make an impression to me
4	The virtual agent will improve my dedication to AR
5	I will enjoy AR more due to the presence of the virtual agent
6	I would prefer a human like agent

Table 2. Summary of survey items assessing the tech-savviness and the educational level of the participants.

Item	Measure
Playing mobile games	Everyday/Sometimes per week/ Sometimes per month/ Never
Using AR	Everyday/Sometimes per week/ Sometimes per month/ Never

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.

MDPI Initiatives

Important Links

Choose an area of interest and we will send you notifications of new preprints at your preferred frequency.

Disclaimer