The results of the case study on the AR content in the EduCITY app and its value for education for sustainability, as perceived by 27 secondary school students, are presented. The findings are divided in two subheadings: the outcomes of the game and the results of the questionnaire survey.
4.1. The Outcomes of the ’EduCIYY at UA Campus’ Game
Figure 5 illustrates the performance of each group in the game. It is important to reiterate that the game comprised 24 questions, seven of which required the exploration of AR content. The analysis of game logs revealed that the average number of correct answers per group was approximately 21. The questions addressed curricular content and cultural aspects of UA, including the following topics: city noise, botany, water footprint, food waste and cultural aspects of UA. The results indicate that the majority of groups achieved a good game performance (
Figure 5). In a game with 24 questions, the students were able to correctly answer an average of 19.9 questions (ranging from 15 to 24) and incorrectly an average of 4.1 (ranging from 3 to 9). The standard deviation for correct answers and for incorrect ones is 2.42. The high average number of correct answers indicates that, in general, the students demonstrated a solid understanding of the game content. However, the equality of the standard deviation suggests that the difficulties encountered by the students were similar for most of the questions. Despite the overall positive performance, it is evident that a small group of students may have faced more challenges. In this regard, the subsequent analysis presents not only the questions that the students answered most correctly, but also those that they answered most incorrectly.
Group 1 demonstrated the highest level of accuracy, with 24 correct responses and no incorrect answers. In contrast, the group with the lowest score was Group 5, which achieved 15 correct answers and 9 incorrect answers. The groups 2, 6 and 9 did not answer correctly three questions (
Figure 5). Group 4 did not correctly answer 4 questions. It can be observed that groups 3, 7 and 8 answered five questions incorrectly.
Although the average number of correct answers is higher than the average number of incorrect answers, the results indicate a modest overall performance in a game in which the multimedia content is designed to support the provision of answers. This suggests that the game could be revised to be more efficacious in promoting learning. To this end, an improvement plan for the game has been developed, which involves not only a comprehensive review of all questions in the game to improve their clarity and coherence, but also the enhancement and development of multimedia resources for the questions with the highest number of incorrect responses, which are analysed in detail below.
To analyse the students' performance in the game, the groups' answers to each question were analysed. To facilitate comprehension, the game was divided into two graphs: one comprising questions without AR (
Figure 6) and another comprising questions associated with AR (
Figure 7).
Appendix A presents the remaining questions in the game, without the use of AR, to facilitate a more comprehensive understanding of the presented results. The
Table A1 presents the question number in the game, the question introduction, the question-and-answer hypotheses, with the correct answer highlighted in bold.
Figure 6 illustrates the results of the questions in the game without the use of AR. The process of analysis began with the questions that yielded the most favourable outcomes and proceeded in a descending order of performance for those that produced the least favourable results.
A review of the results reveals that the questions without AR, which were answered correctly by all students, were questions 1.3, 3.2, 6.3 and 8.1 (
Appendix A). Questions 1.3 and 8.1 required students to observe two distinct locations, each featuring a different sculpture. To respond correctly, the students were required to take a moment to observe the sculptures. This type of question was designed to foster the students' appreciation of art, thereby facilitating their engagement with the cultural and artistic aspects of the location. Simultaneously, it aimed to stimulate their creativity.
Question 3.2 presented a bar graph comparing the water footprint of various foodstuffs. Students had to observe and interpret the graph to determine which food item had the largest water footprint. In question 6.2, students had to watch the provided video in its entirety to determine the amount of food discarded in a 10-minute period in the UA canteen. The fact that all groups answered this question correctly may indicate that videos are an effective educational tool, as proposed by the authors [
3].
In question 1.2, the students used environmental sensors developed by the EduCITY team. These EduCITY sensors facilitate experimental verification of learning outcomes and are designed to collect environmental data that accompanies the learning process using educational games focused on air quality or noise. In this question, the students were required to employ the noise sensor and understand the range of sound pressure levels corresponding to the noise of a quiet conversation. The groups were accompanied by the researcher responsible for the environmental sensors, who checked that all groups had used the sensors as intended. The use was successful, as evidenced by the fact that only one group answered the question incorrectly. Similarly, one group incorrectly answered question 3.1, which required observation of the location in order to provide an accurate response. In this question, the students demonstrated an understanding of the historical context, particularly regarding the instruments of Muslim heritage in the Iberian Peninsula. In question 6.4, only one group chose the wrong option. The results demonstrate that students are aware of the impact of consuming red meat on both the environment and their own health.
Question 4.1 asked students to apply the geometric concepts they had previously learned to identify the geometric solid that best represented the shape of the lake. The two groups that answered incorrectly selected the rectangle option (
Appendix A), which may be indicative of a confusion between solids and geometric figures in a real-life context among the students. Additionally, at the same point of interest, the students were required to answer a question in the field of biology (4.2). In this question, which pertains to the fish that inhabit the lake, the students were required to ascertain whether the fish were typical saltwater or freshwater species, to identify their specific species, and to determine whether they were native or non-native. Both groups incorrectly identified the fish as redheads, rather than carp. One of the students suggested that the colouration of the fish had led to the misidentification.
Questions 6.1 and 6.2 are of an observational and calculative nature. These two questions required students to observe the location and estimate the total number of bicycles that could be stored in the area, as well as calculate the distance between the three UA canteens in a series of steps. Two groups responded incorrectly to question 6.1, selecting the option "between 30 and 40 bicycles." Similarly, two groups responded incorrectly to question 6.2, with one group selecting the option 1170 and another selecting the option 1300. Nevertheless, the correct responses were "between 50 and 60 bicycles" and "1270," respectively.
In question 7.1, the two groups that chose the incorrect option, "used large windows to illuminate the spaces" reviewed the video in the feedback to better understand why this was not the optimal choice. In question 8.2, the two groups that selected the incorrect option were the two groups that did not approach the UA's 'Frog' sculpture but observed it from a distance. Consequently, they did not read the phrase that represents the search engine 'Frog'. This illustrates the importance of clearly observing the point of interest in order to perform well in this game.
It was crucial to include an introductory question (1.1) at the beginning of the game, welcoming the students and promoting their connection with the mascot, Flamingo, who would accompany them throughout the game. This approach, as evidenced in the study by Paixão and Jorge [
41], enhanced the students' willingness to complete the game, and subsequently increasing their interest and motivation. This is because one of the key factors in the learning process is the students' willingness to learn [
41]. Moreover, the use of the flamingo as a mascot was intended to foster a sense of community among the students, as there is evidence that exposure to wildlife can contribute to the development of environmentally conscious citizens who demonstrate greater awareness of species conservation and environmental stewardship. This question required them to listen to the sound and identify that it was the sound of the flamingo, linking it to the project's mascot. However, three groups incorrectly identified the sound as that of a duck, due to its similarity to the flamingo call.
Question 2.1 required students to consult the introductory part of the question to find the correct answer. This was the first question in the game, where students were required to select two correct answer options, a format that may have potentially contributed to some degree of confusion. In an informal discussion at the end of the activity, the monitors of each group reported that the students were confused when they were required to select more than one correct option, due to uncertainty regarding the number of options that should be selected. It is thus evident that the game can be enhanced to provide greater clarity and efficacy in facilitating learning.
In question 7.2, they were directed to a video which provided the necessary information to enable them to identify the correct response. The three groups that selected the incorrect option selected the option 'he was inspired by the waves of the sea'. The students indicated that they had acted precipitously, having only observed the undulating façade and made their selection without watching the video or associating it with the Ria de Aveiro. The remaining groups (x) watched the video, where the correct answer was 'it was inspired by the natural channels of the Ria'.
By observing cultural aspects of the city, students are able to connect theoretical knowledge with real experience, thereby enhancing the engagement of the learning process [
17]. Moreover, students develop a deeper appreciation for the cultural diversity of their city [
17,
32]. Concurrently, students learn to observe and analyse their surroundings, thereby developing critical thinking skills [
32].
As expected, question 4.3 (
Appendix A) produced the worst results, with five groups giving the correct answer and four giving an incorrect answer. The mechanism of osmoregulation in fish is a challenging topic for students due to the complexity of the biological process involved. The image included in this question, which illustrates the differences between the osmoregulation of freshwater and saltwater fish, proved to be helpful to students in answering the question, although it is a challenging diagram to interpret. Students encountered difficulties in applying their knowledge of biology to the observations made in the field.
In addition to the game analysis, it is also essential to identify and evaluate the challenges associated with the exploration of AR content.
Figure 7 presents a comparison of the number of groups that correctly or incorrectly answered each question relating to AR.
Figure 7 demonstrates that the two questions with AR that were answered correctly by all students were questions 2.2 and 9.3. In question 2.2, students were required to explore the maidenhair tree ARBook and select the “curiosities” button to find the answer. The geological period of fossils is new content for the students as it is not included in the Biology and Geology or Science curricula. Despite the new content of this question, all groups got it right, showing that AR, and the ARBook in particular, helped them to answer the question. In question 9.3, students were asked to observe the tile panel and identify in Augmented Marker the elements of Biology, Chemistry and Geosciences (
Figure 8). The fact that all groups answered this question correctly demonstrates that the students had correctly used the full functionality of the Augmented Marker. This was evidenced by their successful exploration of the marker and their ability to locate the requisite information to answer the question correctly.
Figure 7 also indicates that in questions 5.1, 5.3, 9.1 and 9.2, only one group selected the incorrect answer.
In question 5.1, group 5 selected option "Gables are the seeds of the mediterranean cypress", while in question 5.3, group 2 selected option "Angiospermae". In these two questions, students were had to use the ARBook to find the information they needed to give a correct answer.
In question 9.1, group 5 incorrectly identified the tile panel and selected the correct option, 1993. The group did not explore the Augmented Marker; but focused their attention on the panel that could potentially induce errors. As illustrated in
Figure 8, the digit 9 (the second digit displayed on the panel) is strikingly similar to the digit 7 (the third digit shown on the panel).
In question 9.2, group 6 chose the option “painting of Leonardo da Vinci and cello”. The students' response to this question indicates a misunderstanding of the instructions. They were asked to identify the elements of music and art, respectively, but answered in the reverse order.
Regarding question 5.2, four groups selected the incorrect option "Gables are used to make tea”. In order to answer this question correctly, the students had to consult the ARBook, where they found the information they needed on the "curiosities" button on page 2. This research question is more challenging to answer, as the information is not readily apparent. It is notable that all four groups gave the same incorrect answer, which lends credibility to this hypothesis. This is probably because the students' initial response was influenced by their preconceived notion that tea comes from plants.
The data collected through the game integrated into the EduCITY app suggests that the students performed well. However, it would be wrong to claim that AR is 100% effective, as some groups gave incorrect answers to questions that required the use of AR content. Furthermore, this project is still in the data collection phase, and the results are still limited by the small sample size, which may affect the generalisability of the results.
As referred, after the ‘EduCITY at the UA Campus’ game, students were invited to complete an anonymous questionnaire survey, to ascertain their views on the contribution of EduCITY to the fields of education and sustainability.
4.2. Results of the Questionnaire Survey
An overall view about the answers to those topics indicates that students considered the EduCITY app has a positive potential on education for sustainability. The students indicated that the content they found most valuable was related to the cultural aspects of the UA and the trees at campus. It is noteworthy that all students responded to the question "For me, sustainability is..." which indicates their interest in the subject. All students responded to the questionnaire voluntarily and successfully completed the ‘EduCITY at the UA campus’ game.
Students in the first question of the questionnaire had to identify two or three key insights learned from the game (
Table 2).
The feedback was analysed using inductive categorical content analysis [
42]. The unit of analysis was the main themes of the questions, and all answers were analysed as no sampling was conducted. The coding scheme was developed based on the questions themes: noise, botany, food waste, curricular content, cultural aspects of UA and other (students' answers not aligned to the questions in the game).
Table 2 summarises the students' answers, with a total of 62 statements regarding the knowledge and skills they gained from the game.
The majority of students (33 answers) indicated that they had acquired knowledge about a range of topics and curiosities related to the UA. The most common was the date of the UA's birth. This question (9.1.) was associated with an Augmented Marker, and upon exploring this AR, the students noted the highlighted date. The students enjoyed the opportunity to learn about the curiosities of the UA, as many of them had previously expressed an interest in studying there. This activity gave them an insight into some of the cultural aspects of the area. As evidenced by the results of the game, only one group gave an incorrect answer to this question, indicating that the students made effective use of the UA Panel Augmented Marker.
The second most frequently mentioned category was botany (10 responses). The students indicated that they had acquired knowledge about the mediterranean cypress tree, particularly the characteristics of the galbula, through observation in both the real world and in the 3D model associated with ARBook. Furthermore, the students indicated that they had generally acquired knowledge about plants. This finding is supported by the results of the 'EduCITY at UA campus' game, where all groups chose the correct answer to question 2.2.
Although the results of the game did not demonstrate the absolute effectiveness of AR, when triangulated with the responses to this survey question, it can be concluded that the two subjects most frequently mentioned by students were those with AR questions, which may indicate effective learning. Furthermore, the students' willingness to engage with the game and their active exploration of AR to answer the questions was a significant contributing factor in their successful completion of the game with dedication and concentration.
The category that received the third highest number of citations was "Curricular content", with eight mentions across two dimensions of analysis. Students indicated that they had acquired knowledge about science, which was mentioned on seven occasions. Additionally, one student additionally noted that they had gained insights into mathematics. These findings are in line with those of D'Ambrósio [
43], who asserts that examining local mathematical understandings and integrating them into the school curriculum can facilitate more meaningful and contextualised learning, promote appreciation of cultural diversity and stimulate students' interest in mathematics.
Table 2 illustrates that the students also acquired knowledge about the flamingo’s sounds, food waste, other aspects and curiosities about the UA (“Sapo” sculpture, library, solar panels and the well), and emphasised that they had benefited from collaborative learning as they engaged in team-based activities.
In the second question, students were required to complete the sentence, "Sustainability for me is...". In this question, the students opted to answer collectively, stating: i) "…thinking and being responsible"; ii) "…preserving the environment"; iii) "…guaranteeing the needs of this generation without compromising future ones"; iv) "…fulfilling the 3 Rs"; v) "…being responsible"; vi) "…a way of living" and vii) "…balance between humanity and nature". Students' answers demonstrate a broad understanding of the fundamental principles of sustainability. They recognise the significance of environmental protection, intergenerational equity, individual responsibility and a holistic approach to sustainable living. These findings are encouraging and suggest that the students have a solid comprehension of sustainable practices and the capacity to promote them effectively.
The third question of the questionnaire comprised 25 lines, subdivided into two categories: valuing sustainability and supporting equity. As previously stated, only those questions pertaining to the valuing sustainability category were considered for this paper.
To respond to the question "This activity allowed me to...", the students were required to complete the table and use a Likert scale, with responses ranging from 1 (strongly disagree) to 6 (strongly agree). The findings of the survey presented in
Figure 9 suggest that students perceived the EduCITY app to be a valuable educational tool for sustainability. It is notable that no student indicated that they strongly disagreed with any of the statements.
As illustrated in
Figure 9, the students' most highly valued outcome of the activity was the awareness that values and principles can influence actions that may cause damage, harm, restoration, or regeneration of the environment. This was indicated by 15 students who identified level 6, 7 students who identified level 5, 4 students who identified level 4, and only 1 student who identified level 2.
Furthermore, the considerable number of students (15) who selected option 6 (strongly agree) in the statement "be able to evaluate issues and actions based on sustainability values and principles" indicates that the game is an effective tool for students to evaluate sustainability issues. In the statement "be more inclined to act in accordance with the values and principles of sustainability", 15 students also selected option 6, which is defined as "strongly agree".
In statement “be able to apply equity and justice to current and future generations as criteria for preserving the environment and using natural sources”, 14 students indicated a strong agreement with Option 6, 8 students selected Option 5, 3 students selected Option 4, and 2 students selected Option 3. No students expressed disagreement with the statement (options 1 and 2).
With 13 students selecting option 6, the following statements stand out as particularly noteworthy: “become more willing to criticise and value various cultural contexts, depending on their impact on sustainability”, “be able to respect, understand and appreciate various cultures in relation to sustainability, including minority cultures, local and indigenous traditions (natural to a region) and knowledge systems”, “be more willing to share and clarify points of view on sustainability values” and “being able to make personal choices and actions in line with the values and principles of sustainability”.
With a reduction in the number of students selecting option 6, the following statements are particularly noteworthy “being able to articulate and negotiate sustainability values, principles and objectives, while recognising different points of view” and “be able to help create a consensus on sustainability in an inclusive way” with 12 answers.
With 11 answers in option 6 (strongly agree), the statements are as follows “being able to assess and question personal needs in order to carefully manage resources in pursuit of longer-term goals and common interests” and “becoming more familiar with environmental justice, particularly taking into account the interests and capacities of other species and environmental ecosystems”.
The statements with the worst results were “be able to identify and include community values, including minorities, in framing problems and making decisions about sustainability” and “know that individuals and communities differ in the way and degree to which they can promote sustainability” with a total of 9 students answering strongly agree.
The results of the study suggest that the ‘EduCITY at the UA campus’ game was well received by the student cohort, with the majority of students expressing strong agreement on its contribution to understanding and acting on sustainability principles. The game is particularly effective in enabling students to evaluate sustainability issues, understand the influence of values, and motivate sustainable actions.
Nevertheless, there is a clear need to enhance the skills required to negotiate sustainability values and to develop the ability to critique and value different cultural contexts. Furthermore, while there is a strong grasp of concepts, such as environmental justice and anthropocentrism, these areas could benefit from further reinforcement.