Preprint
Article
An AI-Powered Chatbot to Support University Life
This version is not peer-reviewed.
Submitted:
19 February 2025
Posted:
20 February 2025
You are already at the latest version
Abstract
This article aims to explore the potential of artificial intelligence in the university context, analyzing how these technologies can be used to improve the quality of life of students. The adoption of artificial intelligence in this context presents a series of potential advantages. First, AI can facilitate the management and processing of massive amounts of data, allowing for a more precise and timely analysis of students' needs. Furthermore, AI can support personalization, offering content and resources suited to individual needs and helping to identify any difficulties early on. In support of this theory, the authors have created a chatbot, called UNISALERNOBOT, which is able to provide support to university students in Salerno on various issues of university life. In this article, the focus will be on the evaluation of UNISALERNOBOT. To this end, an experiment was conducted involving over one hundred students and, after this, a questionnaire was administered to collect quantitative and qualitative elements on various aspects of the interaction with the chatbot, including accessibility, usability, relevance and completeness of the answers, as well as overall satisfaction.
Keywords:
artificial intelligence
; evaluation
; chatbot
; university
; lifelong learning
1. Introduction
“AI has enormous potential in higher education, from personalizing study plans to creating adaptive tests to supporting academic writing. Students and researchers are already benefiting from these technologies.” [1].
EdTech Magazine states that chatbots enable 24/7 assistance, quickly responding to questions about registrations, deadlines, study plans and campus services, ensuring a continuous and immediate support experience. In this way, they meet the expectations of so-called “digital natives,” who expect quick and accessible answers on multiple digital platforms, such as Facebook and WhatsApp [2]. Many prototypes and tools as AI-powered chatbots have been developed and kept available to university students to enhance their experiences by providing them instant access to information and communication, or giving answers to university-related questions [3,4]. Integrating chatbot interventions with life-crafting interventions can prevent interrelated academic and mental health problems, improving student retention and performance [5]. AI-powered chatbots in higher education may improve student performance, but may negatively impact social support, wellbeing, loneliness, and sense of belonging [6].
In this perspective, a recent project “UNISALERNO BOT” developed at the University of Salerno aims to explore the contribution that AI can offer to university life. The project focuses on a chatbot designed to support students by answering questions on various aspects of academic life, with the aim of making student interaction with university services more fluid and immediate.
«The inherent complexity of sociocultural phenomena therefore pushes us to satisfy the need to orient ourselves towards methods that better allow the enrichment of knowledge around a phenomenon. And this is even more true in the current historical phase in which we are witnessing the transition from the network society [...] to the platform society.» [7].
To measure the effectiveness of UNISALERNO BOT, an experiment was conducted involving a large sample of students and future students at the University of Salerno. The chatbot was subjected to a thorough evaluation through a questionnaire, which is «A technique for collecting information, used to answer research questions, in the field of social sciences, based on a dyadic relationship.» [8].
The carefully planned questionnaire is structured to cover various aspects of the chatbot’s functioning, including the relevance of the answers provided, the coherence and accuracy of the information, and the ability to handle different types of interactions. Each question has been designed to identify potential areas for improvement, thus ensuring that the chatbot not only meets practical needs, but also high-quality criteria, reflecting academic standards and ensuring high-level user interaction.
In addition, the questionnaire has been modulated to collect qualitative and quantitative data on user experiences, with the aim of adapting the chatbot’s functionalities based on specific feedback. This careful process allows for the constant refinement of the chatbot’s algorithm, making it more effective in responding to student requests and providing academic support.
Through this survey tool, students had the opportunity to indicate the areas in which the chatbot excels and those that could benefit from further improvements.
The results of the experimentation guided optimization interventions, improving the accuracy and relevance of the answers provided by the chatbot. In this article, the various steps adopted in the evaluation, the analysis methodologies applied, and the improvement interventions carried out thanks to the feedback collected will be described.
2. Materials and Methods
During the preparation phase, students were asked to interact with the bot, so that they could subsequently provide informed feedback on their experience using the bot itself.
Chatbots are evaluated to measure their impact on student interaction. This evaluation process allows us to collect useful feedback to optimize the design, functionality and effectiveness of digital tools.
«Analysing data from a chatbot could potentially monitor its development, strengths, weaknesses to provide personalized content and suggestions. This personal touch creates a personalized journey for each learner, potentially improving their overall learning experience.» [9].
The chatbot was distributed through different communication channels to ensure broad participation and representative data collection. In particular, the promotion took place via WhatsApp, Facebook and through awareness-raising activities carried out in person at the university. The choice of these communication channels was strategic to effectively reach students, taking advantage of both digital platforms widely used by young people and direct interaction that allows for more personal and immediate contact [10].
This multifocal approach facilitated access to the questionnaire, allowing students to provide valuable feedback on their experience with the chatbot and contributing significantly to the evaluation of its performance.
3. Results
The activities carried out in the evaluation of the UNISALERNO BOT chatbot were divided into distinct phases.
In a first phase, the chatbot was distributed to give students the opportunity to interact with it and actually use it. In a second phase, the evaluation questionnaire was prepared.
The questionnaire is a widely used tool in research because it is effective in collecting specific information on a given situation or problem.
«Since the questionnaire is a potential source of non-sampling errors, it is necessary to design it as an important part of the program for their prevention and measurement.
To do this, the drafting phase of the questionnaire becomes crucial: from the definition of the objectives, we move on to the actual preparation of the questionnaire.» [11].
Its formulation was carefully studied to ensure the clarity of the questions, using terms with unambiguous meaning in order to avoid ambiguities of interpretation by the interviewees. Before administration, particular attention was paid to the layout of the questions to ensure that they were easily understandable and did not generate confusion, as well as to their coherence in grouping questions aimed at exploring the same aspects. The questionnaire is structured in two distinctive parts:
- A section composed of nine closed-ended questions, aimed at evaluating the performance of the chatbot.
- Five open-ended questions, aimed at gathering personal opinions and suggestions.
Kathryn J. Roulston reveals her definitions of open-ended and closed-ended questions in qualitative interviews in the SAGE Encyclopaedia on Qualitative Research Methods and states that “a closed-ended question allows respondents to choose from a predefined response option, while an open-ended question asks respondents to provide feedback in their own words.” [12]. Figure 1; Figure 2; and Figure 3 show the first three questions framed according to these principles.
The questions following the first three are open-ended questions aimed at collecting qualitative data such as opinions and suggestions on the students’ interaction with the bot (Figure 4 and Figure 5).
The fourth and final phase of the research conducted was the data analysis during which tables, graphic representations and advanced statistics were developed to summarize and interpret the results.
4. Discussion
The students who participated (101 in total) in the survey are mostly women, although the ratio between the presence of female and male users is almost similar. In particular, as shown in Figure 6, 56.4% are female, while 35.6% are male, the remaining 7.9% preferred not to specify.
Each respondent was asked to specify whether they were enrolled students or future students, in order to highlight the target audience of the bot’s users (Figure 7).
It was found that 79.2% of respondents were enrolled students, while the remaining 20.8% intended to enroll in university.
The evaluation of the chatbot revealed several significant issues and limitations. One of the main issues encountered was the quality and relevance of the responses, with many users reporting inaccurate or irrelevant responses to their questions. The chatbot’s ability to understand and handle complex requests was limited, suggesting the need for improved machine learning and natural language understanding (NLP) models (Figure 8).
Furthermore, issues with response speed and accessibility were highlighted, which compromised the user experience due to the use of the free version of the chatbot. The evaluation also found that the bot’s knowledge base needs regular updates to maintain the relevance of the information provided. Finally, the need to further optimize the user interface to make it more user-friendly and to continue monitoring the performance of the chatbot through constant feedback and the use of KPIs to ensure continuous improvement emerged (Figure 9).
The detailed analysis allowed us to identify the chatbot’s limitations and, consequently, to address any changes to refine its functionality, better respond to the needs expressed by students and improve the user experience.
«The body of knowledge and ideas which is the product of work, is the result of a method which has been followed by a much larger number of people, who have interacted intelligently and with open-mindedness with the objects and events of the common environment.» [13].
Finally, the presentation of the results made it clear how the integration and continuous improvement of the chatbot can significantly innovate university educational services.
Therefore, the analysis of the answers to the questionnaire allowed us to identify the aspects to be addressed and identify the changes to be made to the files and documents that constitute the chatbot’s knowledge base with the aim of ensuring continuous improvement of its performance. This process involved the inclusion of specific and relevant terms for each service category, thus improving the bot’s ability to understand and respond accurately to user requests.
The fact that the bot provided more correct answers when asked the same questions again highlights a process of improvement and learning of the system. This suggests that the bot is able to adapt and learn from the inputs received, refining its capabilities over time. The observation that the bot provides more accurate answers when asked the same questions again highlights its process of improvement and adaptation over time, thus confirming the effectiveness of the training process and the importance of continuous feedback to optimize the performance of artificial intelligence.
5. Conclusions
Artificial intelligence is increasingly emerging as a crucial resource for addressing the challenges of contemporary society, particularly in the university context. As highlighted by our analysis, its correct use can represent a strategic lever for supporting students, improving access to knowledge and personalizing training paths. An accurate analysis of this data can provide valuable insights that help companies make chatbots more effective and offer an exceptional customer experience.» [14].
The ability to evaluate their functioning, for example in the context of chatbots, allows these technologies to progressively adapt to user needs, offering increasingly relevant and effective responses.
In the scientific landscape of reference, there are numerous studies that explore the use and impact of artificial intelligence in the educational and technological fields.
In fact, let us consider that during June 2024, the European Parliament set its negotiating position on the AI Act (or AI legislation), the first set of rules in the world on artificial intelligence. «Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 establishes harmonized rules on artificial intelligence and amends Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828.» [15].
Fatima Ezzeddine had participated, when she was still in Lebanon, in a study conducted in interdisciplinary collaboration with Swiss researchers to evaluate artificial intelligence in such a way as to «explain how artificial intelligence works, how it behaves, how it is implemented and how it is used by people.» [16].
Andrea Barni during the “Action Day 2024”, organized on 17 September at the Palazzo dei Congressi in Lugano by the Agire Foundation states: «Our way of living and working will change, because AGI (Artificial General Intelligence) will have creativity, abstract thinking, understanding of cause and effect, transferable learning. For the moment, no one is yet able to predict exactly what will happen, but it will certainly be an absolute revolution, which we will have to learn to govern, without leaving the knowledge of this new world only to the technicians.» [17].
Many studies have shown how AI-based chatbots can progressively improve through machine learning techniques. These improvements are based on the analysis of user-system interactions, which allows to identify weak points, optimize responses and better predict user needs. Other studies have investigated user perception, highlighting that chatbots that are able to adapt their responses over time generate greater trust and satisfaction.
A recent study conducted by Justus Henke of the Martin Luther University of Halle-Wittenberg, published in the Journal of Science Communication, Navigating the AI Era: University Communication Strategies and Perspectives on Generative AI Tools, offers an insight into the situation in Germany. This study conducts a pioneering empirical analysis of generative artificial intelligence tools, such as ChatGPT, in the context of university communication in German universities [18].
However, part of the debate focuses on the importance of avoiding excessive reliance on AI technologies, highlighting the limitations related to the lack of empathy and the ability to handle situations that require human judgment.
To further explore this issue, UNISALERNOBOT was designed, an AI-based chatbot capable of providing detailed answers to students at the University of Salerno throughout their academic career.
To evaluate the effectiveness of UNISALERNO BOT, a trial was conducted on a large group of students and prospective students at the University of Salerno. The chatbot was analyzed through a questionnaire, whose questions were carefully structured and aimed at identifying areas for improvement, ensuring that the chatbot met high standards both in terms of practical usefulness and overall quality of the user experience.
Furthermore, the questionnaire was designed to collect both quantitative and qualitative data, allowing the chatbot to be adapted based on specific feedback received. This approach enabled constant refinement of the algorithm, improving the chatbot’s ability to meet student needs and provide targeted academic support.
The data collected highlighted both the chatbot’s strengths and areas where changes could be made, thus guiding targeted interventions to increase the accuracy and relevance of the responses. The article details the phases of the evaluation process, the methodologies used, and the improvements implemented thanks to user contributions.
As a final step of the research, the chatbot was refined and optimized in the areas indicated by the student feedback collected through the questionnaire.
The results obtained from the experimentation of the UNISALERNO BOT chatbot were significant and encouraging. Thanks to student feedback, the system was improved in several key areas, including the relevance of the responses, the clarity of the information, and the ability to manage a wide range of interactions.
This process led to the creation of an effective tool, capable of supporting students along their academic path, answering their questions in a precise and timely manner.
The chatbot not only represents a concrete example of how artificial intelligence can be applied in the educational context but also stands out for its ability to evolve and improve over time, responding to the real needs of users. This makes it a practical and accessible tool, capable of simplifying students’ lives and enriching the university experience.
Future developments for UNISALERNO BOT and similar chatbots look promising and are geared towards further customization and expansion of functionality. Some possible developments include, for example, the implementation of systems that can adapt to the specific profile of each user, offering personalized responses based on the course of study, level of preparation and individual needs, or connections with academic platforms such as student management systems (for example, Moodle or exam registration systems), allowing students to access information and services directly through the chatbot.
Furthermore, in addition to text interactions, the integration of voice responses, explanatory videos and interactive tutorials, improving accessibility for students with special needs or different preferences.
The extension of the chatbot to offer not only academic support, but also orientation and administrative support, such as the management of scholarships, housing or post-graduate counselling.
The fundamental imprint of artificial intelligence with CHAT GPT has spread quickly and this is also confirmed by the studies carried out by USI vice-rector Luca M. Gambardella, who explained at what artificial intelligence means, what ChatGPT is, what it does, why it amazes us: «Thinking of ChatGPT the adjective δεινός comes to mind (deinòs), which in ancient Greek means terrible and wonderful, extraordinary and frightening at the same time. The great Sophocles uses it when the chorus of his Antigone sings: “Many are the things of God, but nothing is more so than man.” Only now, perhaps for the first time, instead of man there is a machine.» [19].
These future developments aim to make the chatbot an intelligent guide that accompanies students throughout their university experience.
Author Contributions
Conceptualization, S.M. and G.I.; methodology, S.M.; formal analysis, G.I.; data curation, G.I.; investigation, G.I.; supervision, S.M. All authors have read and agreed to the published version of the manuscript.
In accordance with the guidelines of the American Psychological Association (APA), participants were asked to give informed consent regarding the nature of the survey and its objectives exclusively for research purposes. Their participation was voluntary and was carried out by guaranteeing confidentiality and anonymity, since the data were collected in digital form without requesting the identity of the participants. Therefore, the data were collected in compliance with the European Regulation on Data Protection (GDPR n.679/2016) since they involve EU citizens anonymously and do not identify the participants in any way and irreversibly. No conflict of interest is connected to the survey conducted.
References
- Maceri, M. (2024). AI at University: The Silent Revolution of Academic Education. AgendaDigitale.eu.
- Brereton E. (2021). Higher Education’s Increasingly Nuanced, AI-Powered Chatbots. EdTech Magazine.
- Alabbas, A., & Alomar, K. (2024). Tayseer: A Novel AI-Powered Arabic Chatbot Framework for Technical and Vocational Student Helpdesk Services and Enhancing Student Interactions. Applied Sciences. [CrossRef]
- Dinh, H., & Tran, T. (2023). EduChat: An AI-Based Chatbot for University-Related Information Using a Hybrid Approach. Applied Sciences. [CrossRef]
- Dekker, I., De Jong, E., Schippers, M., De Bruijn-Smolders, M., Alexiou, A., & Giesbers, B. (2020). Optimizing Students’ Mental Health and Academic Performance: AI-Enhanced Life Crafting. Frontiers in Psychology, 11. [CrossRef]
- Crawford, J., Allen, K., Pani, B., & Cowling, M. (2024). When artificial intelligence substitutes humans in higher education: the cost of loneliness, student success, and retention. Studies in Higher Education, 49, 883 - 897. [CrossRef]
- Masullo, G., Addeo, F., Delli Paoli, A. (2020). Ethnography and Netnography. Theoretical reflections, methodological challenges and research experiences. Paolo Loffredo Ed., Napoli.
- Corrao S. (2005). The Interview in Social Research. Open Edition Journals.
- LXD Expert (2024). Case Studies on Successful Implementation of Chatbots in eLearning. Clue labs.
- Baghal, T., Serôdio, P., Liu, S., Sloan, L., & Jessop, C. (2024). Using social media metrics and linked survey data to understand survey behaviors. International Journal of Population Data Science. [CrossRef]
- Sabbadini, L. L. (1989). Handbook of survey techniques. The questionnaire: design, drafting and verification. ISTAT National Institute of Statistics.
- Roulston, K. (2014). Interactional problems in research interviews. Qualitative Research, 14(3), 277-293. [CrossRef]
- Dewey J. (1938). The unity of science as a social problem. University of Chicago Press, 1938.
- Ranieri, A., Di Bernardo, I., & Mele, C. (2024). Serving customers through chatbots: positive and negative effects on customer experience. Journal of Service Theory and Practice. [CrossRef]
- European Parliament (2024). What are the risks and benefits of artificial intelligence?
- Ezzeddine, F. (2024). Privacy Implications of Explainable AI in Data-Driven Systems. Late-breaking work, Demos and Doctoral Consortium, co-located with The 2nd World Conference on eXplainable Artificial Intelligence: July 17–19, 2024, Valletta, Malta.
- Barni A. (2024). Action day 2024. https://actiondayagire.ch/.
- Henke, J. (2024). Navigating the AI era: university communication strategies and perspectives on generative AI tools. JCOM 23(03), A05. [CrossRef]
- Gambardella, L.M. (2021), ARTIFICIAL INTELLIGENCE AND MEDICINE: PAST AND FUTURE. Hematological Oncology, 39. [CrossRef]
Figure 1.
How useful did you find the bot?
Figure 2.
How well did it meet your needs?
Figure 3.
Do you think it needs improvement?
Figure 4.
If you answered yes to the previous question, please justify your answer.
Figure 5.
For which questions was it not effective?
Figure 6.
Distribution of respondents by gender.
Figure 7.
Distribution of enrolled students and prospective students.
Figure 8.
Personal considerations.
Figure 9.
Personal considerations.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
Downloads
63
Views
37
Comments
0
Subscription
Notify me about updates to this article or when a peer-reviewed version is published.
MDPI Initiatives
Important Links
© 2025 MDPI (Basel, Switzerland) unless otherwise stated