Prerpints.org logo
Preprint
Article

Chatbot Design and Implementation: Towards an Operational Model for Chatbots

This version is not peer-reviewed.

Submitted:

29 March 2024

Posted:

31 March 2024

You are already at the latest version

A peer-reviewed article of this preprint also exists.

Abstract
The recent past has witnessed a growing interest in technologies for creating chatbots. Advances in Large Language Models for natural language processing are underpinning rapid progress in chatbot development, and experts predict revolutionary changes in the labour market as many manual tasks are replaced by virtual assistants in a range of business functions. As the new technology becomes more accessible and advanced, more companies are exploring the possibilities of implementing virtual assistants to automate routine tasks and improve service. This article reports on qualitative inductive research undertaken within a chatbot development team operating in a major international enterprise. The findings identify critical success factors for chatbot projects, and, based on the in-depth case study, a model is put forward to aid the manageability of chatbot implementation. The presented model can serve as an exemplar guide for researchers and practitioners working on the design, development and implementation of virtual assistants. It is flexible and applicable in a wide range of business contexts, linking strategic business goals with execution steps. It is particularly applicable for teams with no experience of chatbot implementation, reducing uncertainty and managing decisions and risks throughout the project lifecycle, thereby increasing the likelihood of project success.
Keywords: 
Subject: 
Business, Economics and Management  -   Business and Management

1. Introduction

The chatbot is often referred to as a “conversational interface” or “conversational user interface”, indicating a technology that enables communication between people and information systems (IS), using a human language [1]. While chatbots are usually associated with a text-based communication in a messenger or chat window [2], conversational interfaces can also include voice assistants integrated in smart or wearable devices, smartphones, social robots, autonomous vehicles, and other devices. The Cambridge Dictionary [3] defines a chatbot as a “computer program designed to have a conversation with a human being, especially over the internet”. In practice, modern chatbot platforms vary in behaviour and complexity, and can be delivered as standalone applications or as a service based upon another technology platform.
Over the past decade, the growth in the successful use of chatbots has often been linked to advances in the development of AI technology. Virtual assistants have been developed for a range of different functions, but all use AI technologies and natural language processing (NLP) algorithms [4]. For the past decade, new approaches based on non-linear neural networks have become increasingly evident, and the use of recurrent neural networks (RNNs) has demonstrated significant success in solving typical NLP problems [5]. Bernardini et al. [6] have shown how competitive pressures in global markets and rapid advances in hardware performance - making new technologies increasingly accessible - have led to a significant increase in interest in the potential of AI. In 2018 Devlin et al. [7] introduced a BERT (Bidirectional Encoder Representations from Transformers). This model significantly improved conversational interface performance, and for chatbot developers, BERT provided new opportunities to solve applied NLP tasks typical of most chatbots: user intent recognition, question answering, classification of what? [8].
The subsequent improvement of Large Language Models (LLMs) and related text generative models have had a significant impact on the development of chatbots. The latest and most significant development in the field of NLP is the creation of the GPT (Generative Pre-trained Transformer) model by OpenAI in 2018. It shows excellent results in tasks involving the processing of human language text. The GPT model is trained on a wide range of open text data, including Wikipedia, and is able to generate coherent text of almost indistinguishable quality from human-written text [9]. GPT-4, released in March 2023, is the latest and most advanced version of the GPT family of models to date. It is a multimodal model that works not only with textual information, but also with images, and has pioneering abilities to handle a wide range of real-world tasks performed by humans. Benchmarks show that the model scores higher than the average human on many popular standardised exams in maths, physics, chemistry, statistics, biology and other fields [10]. Today, chatbots and generative AI are widely used by large enterprises as well as small and medium-sized enterprises (SMEs) in many industries. They commonly automate processes in marketing and sales, customer support, human resources, and other areas [11].
Chatbots are attractive to businesses as they facilitate customer service through a conversational style that is natural and intuitive for humans [12]. Despite the rapid development of technology in this area, the increasing capabilities of AI algorithms and the growing number of successful real-world applications, chatbot implementation projects face a number of difficulties. The expectation of quick and immediate success from chatbot implementation that many companies had in the middle of the last decade has been replaced by a more sceptical attitude towards the potential of virtual assistants [13]. Practical experience has shown that chatbots cannot be used effectively in all situations, whilst creating a high-quality service requires significant investment. The technological complexity and unpredictability of project outcomes present additional difficulties for implementation teams, who often lack practical experience and theoretical knowledge [14,15,16].
This article explores the range of issues that confront those involved in chatbot implementation projects. Building upon earlier research by the authors [17], a new model is formulated and presented, which can support IT managers (from large and medium-sized companies), IT integrators and other industry practitioners and specialists involved in the creation and deployment of chatbots. The research identifies common patterns and rules that can be useful for a wide range of organisations whose activities are related to customer service, which can be transformed into a client-chatbot interaction, benefiting both the company and the client. More specifically, this article addresses the following research questions (RQs):
RQ1. What are the main challenges which companies face when implementing chatbot projects?
RQ2. What are the critical success factors (CSFs) that contribute to the successful implementation of chatbots in companies?
RQ3. What operational model can support the chatbot implementation process and help responsible managers to deliver expected chatbot project outcomes?
Following this introduction, the article comprises five further sections. In section 2, the different aspects of the research method are discussed. This is followed by a review of relevant literature and development of the provisional conceptual framework for the study in Section 3. Section 4 then sets out and discusses the research results and addresses the research questions. Finally, the conclusion section considers the limitations of the study, and points up possible future avenues of research in this field.

2. Research Method

2.1. The Case Study Approach and Research Philosophy

The research centres on a case study of a chatbot implementation in one of the world's leading steel producers (presented here as an alias - ESteelCo) that took place in the two-year period March 2021 to March 2023. The ESteelCo case study provided an in-depth inquiry into the complexity of chatbot implementation and was the basis for generation of new knowledge in this subject area. The company’s global strategy has a strong client focus and the company and its clients employ popular instant messengers (WhatsApp, Telegram) as the preferred means of communication. In addition, the company launched a chatbot service, which offers a new modern way to solve clients’ problems. ESteelCo offers its products to a variety of different customer groups, and the launch of the chatbot service provided an immediate response to client enquiries.
The overall research aim was to develop a model based on findings obtained from analysing the chatbot implementation process. The authors adopted an interpretivist philosophy to observe the chatbot integration process within the selected organisation. New perceptions of the key issues were obtained through the recording and analysis of the unique experiences of the people involved in the project. The selected paradigm aligns with the application of qualitative methods of analysis to answer the RQs. The experience of practitioners, personal views, criticisms, and beliefs of individuals involved in the study represent essential data that was recorded and analysed to explore the phenomenon and develop rules for the new chatbot implementation model. An interpretivist philosophy is also appropriate for an in-depth case study of the social phenomenon in an existing company environment [18]. The research process comprised four main phases as depicted in Figure 1. The key aspects of data collection and results validation, particularly important in a qualitative interpretivist case study, are now discussed in more detail below.

2.2. Data Collection

The study adopted a multi-method approach to data collection, employing both survey and observation to collect various sets of qualitative data in the case study company. Yin [18] (p. 99) argues that use of various data sources within a case study can significantly improve the quality of the research data, and here the personal experience of specialists participating in the chatbot implementation project made a significant contribution in terms of variety, completeness, and quality of data. This was complemented by observations by one of the authors who played a role within the project team and had the opportunity to observe the implementation of the chatbot project.
Semi-structured interviews were used, and the list of topics and structure of questions were designed to address RQ1 and RQ2 and validate and develop the provisional conceptual framework, discussed below. This was an exploratory study in which chatbot experts and practitioners responded to a pre-defined list of questions, the answers to which formed the basis for formulating a new model for implementing chatbots. At the same time, respondents had the opportunity in the interviews for wider discussion, to recall their recent experiences, think about why certain events took place, and suggest what could have been done better to integrate the chatbot more effectively with other company systems.
There were 15 respondents/interviewees who took part in the study. Eight participated directly in the chatbot implementation project in ESteelCo, and the other seven were external experts from various banking, IT, retail and pharmaceutical companies in Russia, China, and Germany (Table 1). The interviewees were selected “because they have particular features or characteristics which will enable detailed exploration and understanding of the central themes and puzzles which the researcher wishes to study” [19] (p. 78). They were a combination of experienced internal and external specialists with various tech-nical, industry and cultural backgrounds, which would support the generalisability of the research results and the formulation of a model for managing chatbots projects in other industry contexts. Guest et al. [20] (p. 78) suggested that “a sample of six interviews may have been sufficient to enable development of meaningful themes and useful interpretations”, and Kuzel [21] similarly recommended that, for qualitative research, six to eight interviews is enough for a homogenous sample, but 12 to 20 would be preferable when “trying to achieve maximum variation” (p.41).
In terms of the observations made during the study, a narrative approach was adopted for the recording and presentation of research records, which is characteristic of a qualitative study [22]. One of the authors was involved as a project team member, and had the opportunity to observe the issues relating to chatbot integration and operation through the various project stages, from initial design to product launch and service support. Significant daily events were recorded in the form of short notes, which were subsequently expanded and presented in a narrative form. Although this approach proved effective in providing relevant information and insights, bias in the perspectives taken is inevitable. For this reason, observation is used as a secondary data collection technique, which supplements the findings obtained from the semi-structured interviews.

2.3. Results Validation

Validity and reliability of data and results are important criteria that directly impact the quality of the study. Trochim et al. [23] assume that participants are the ones who can best review research findings and evaluate how much they accurately match their opinions. The level of trust in the results heavily depends on how correctly researchers record and document the views of the people who take part in a study. Participant validation is thus a technique that allows researchers to achieve credibility of research data by involving participants in the process of reviewing results [24]. Here, considerable attention was paid to establishing procedures for verifying the data obtained with the direct participation of the interviewees. Interviews were conducted remotely via videoconferencing using screen sharing functionality, and interviews were recorded. During the interview conversations, the key thoughts and statements of the interviewee were noted down by the interviewer, and in case of doubt, clarification was requested. In addition, the transcript was emailed to the interviewee for final review and corrections. Only after confirming that all opinions were recorded and formulated correctly, was the data used for processing and analysis, thereby minimising possible bias in the interpretation of the findings.
Guba et al. [25] identify dependability and confirmability as two important criteria which measure research reliability. They argue that to some extent generalisation can be achieved by a careful and exhaustive description of all stages of a study (dependability). Description of the research methodology, justification of the chosen methods, a detailed description of the ESteelCo case study, and the process of collecting and analysing data are all integral parts of this research work. All these activities aim to establish the quality of the research, and advance the possibility of reproduction by others. The second criterion, confirmability, is associated with the possibility that research findings and conclusions stated by a researcher can be confirmed by others. To this end five experts were contacted to check the accuracy and validity of the research findings. Of these five, two were from the pool of interviewed participants, whilst the other three were experts in this field of research, but not previously involved in the case study (Table 2).
Triangulation is a well-proven approach which helps to improve the validity and credibility of a study. One of the methods proposed by Patton [26] is associated with multi-method studies and the use of two or more data collection techniques. Use of different independent data sources and application of various methods to analyse them and find the answer to the same research question help to verify and validate qualitative analysis. In this context, the results from the interviews, which directly addressed the RQs, are supplemented by observation. Using two different data collection techniques to answer the same questions helps to examine the problem from different perspectives.

3. Relevant Literature and Provisional Conceptual Framework

This section reports on the findings from an initial scoping literature review that Bell et al. [27] have observed can provide “a means of gaining an initial impression” of relevant themes and that “the narrative review may be more suitable for qualitative or inductive researchers, whose research strategies are based on an interpretative epistemology” (p. 97). The scoping literature review aimed at identifying the key themes from the literature that could provide the basis for the development of a provisional conceptual framework for the research, and identify some of the key literature that would support the generation of responses to the RQs. The scan of relevant literature allowed the discovery of “topical relationships, research trends, and complementary capabilities” [28] (p.351), and the following three sub-sections report on this literature as it relates to the three RQs. Finally, sub-section 3.4 sets out the provisional conceptual framework derived from this literature review.

3.1. Chatbot Implementation Challenges

Despite the potential benefits of chatbots, many companies still find the prospect of chatbot implementation challenging. According to CB Insights [13], high user expectations are not always justified as the technologies and methods for creating chatbots are far from perfect. Srinivasan et al. [29] surveyed more than 300 business executives and IT directors in 12 countries and identified the most prevalent challenges companies face when implementing chatbots: scepticism from users in accepting a new way of communicating; the limited capabilities of chatbots and their inability to provide an acceptably personalised service; and the high number of errors in handling user requests. In addition, the complexity of the technical component of chatbots, and organisational barriers are other hurdles that companies must overcome. Srinivasan et al. [29] also point to the difficulty of addressing data security issues and compliance with local data protection laws, the lack of skilled labour, high integration costs, and the difficulty of platform selection due to high vendor fragmentation. This is confirmed by Kaushal et al. [30], who argue that companies have difficulties with platform selection and integration when implementing chatbots. The majority of chatbot platforms do not provide the necessary functionality for testing and customisation, and the resulting behaviour of the virtual assistant may fail to meet user expectations regarding quality, responsiveness and ability to correctly process a request formulated in human language.
Despite justified criticism from businesses regarding the conversational capabilities of modern chatbots, the results of research groups and demonstrations of the capabilities of chatbots from major technology companies show that the quality of a virtual assistants’ dialogue is almost indistinguishable from that of a human. In practice, however, companies are severely limited in what they can do because many NLP related tasks are challenging [31]. The creation of a perfect interactive assistant requires significant computing and financial resources to collect and process large amounts of data to train the language models, as well as the availability of appropriate specialists [32].
Challenges for chatbot developers are not always directly related to technology, organisational issues, or costs. Even a perfectly implemented chatbot that exploits all the technologies currently available may not be popular with users, some of which may be conservative in attitude and prefer the current status quo. Previous negative experiences with virtual assistants may also discourage potential new users [33].
Fach et al. [14], based on feedback from more than 90 leaders from different organisations worldwide, categorised the main challenges as follows:
  • Setup challenges. Activities related to obtaining, preparing and processing of big data, bot training, implementation and launch, support and maintenance.
  • User/customer acceptance. Not all users are ready to interact with virtual agents. This situation is expected to gradually improve, so in the long term, such difficulties should not prevent the introduction of new technology.
  • Language challenges. Effective communicating with users can be hampered by technical limitations, difficulties in understanding various accents, as well as the ability of developers to deal with natural language processing (NLP) tasks.
  • Regulatory restrictions and data security. Both internal company guidelines and national or international regulations (for example, GDPR) can impact system architecture and methods of receiving and storing customer data. This complicates the process of developing virtual assistants in practice and increases the cost of implementation.
  • Technology-related challenges need to be addressed. It is advisable to implement technologies from strong vendors that have mature solutions, innovative features, and flexible APIs for integration with other company systems.
  • This categorisation overlaps considerably with the perspectives given by Srinivasan et al. [29] and other authors.

3.2. Critical Success Factors for Chatbot Projects

The project management triangle, sometimes called the “iron triangle”, has been used for over 50 years to estimate project success. Time, quality and budget are the three parameters, which determine project objectives and allow the definition of measurable goals [34]. However, the above review of the challenges faced by companies implementing chatbots showed that most of the difficulties and failures are related not only to aspects of project management, but to a wide variety of other factors, these being external or internal to the organisation, and having a technological, social, or other nature. This suggests that the traditional approach to determining the success of a project, exemplified in the parameters of the “iron triangle”, could usefully be complemented by the identification of critical success factors (CSFs) for chatbot projects.
CSFs are well established as an approach to project management. Rockart [35] used CSFs as a method for planning information systems in organisations, and Bullen et al. [36] characterised CSFs as project related areas in which sufficient results will guarantee successful achievement of the project goals. Existing literature, documented case studies and IT integrator guidance available on the internet were analysed to find appropriate CSFs for chatbot projects. These CSFs and their sources are presented in Table 3 and the variety of factors suggests that a range of different subject areas impact the outcome of chatbot projects. These include new technology adoption, resource availability, vendor and platform selection, software licensing and implementation costs, as well as organisational and project management aspects.

3.3. Relevant Models, Methods and Frameworks

A number of technology adoption models are of relevance to chatbot projects. DePietro et al. [52] developed the Technology Organization Environment (TOE) framework, which has been adapted and adopted in a range of technology studies [53,54,55,56,57]. The framework suggests that three main factors influence technology innovation in the corporate environment (Figure 2). First, the technology factor concerns the IT infrastructure, hardware, network communication, software packages and services, available to the company, and the processes that facilitate its deployment. Second, organisational culture and structure, core business processes, and resource availability are seen as key elements in successful transformation. Third, the environmental factor refers to the external market within which the business operates, and the activities of other organisations – competitors, service providers, and government regulators.
Another model from the same era, but equally still of relevance today, is that developed by Earl [58], who set out a multiple methodology for IS strategy formulation. In fact, Earl envisaged three interlocking strategies covering IT, IS and IM (Information Management), but IS is considered here. Earl identified three approaches to developing strategy – top-down, bottom-up and inside-out. The top-down method is an analytical deductive approach that identifies business plans and IS needs. The analysis of CSFs is used via interviews, workshops, or document analysis to establish a connection between business goals and IS requirements. The bottom-up approach assumes that some existing IS have strategic potential and their functionality can support other business activities that provide a competitive advantage. Thus, an audit of current systems is required to recognize which ones need to be updated, what kind of add-ons or extension modules exist, what is the potential for enhancement and further development. Inside-out identifies innovation opportunities and discovers the new strategic potential for the company: investments in R&D, human talent and development of technological culture across the organisation will foster innovation and speed up new technology adoption.
There are a number of stage models used by IS researchers or IT practitioners to aid the planning and analysis of projects. These include the approach developed by the Project Management Institute (PMI), which distinguishes between initiation, planning, execution, and closeout phases [59], and the waterfall model [60] in which each phase is validated by stakeholders before moving on to the next phase. It is irreversible and does not allow the previous phases to be reviewed once they have been completed. The advantage of the waterfall model is its comprehensive approach, which helps to develop a well-integrated and documented system. The involvement of stakeholders at each stage ensures their participation and helps to minimise further complaints. The model is often criticised for its long system development process and the risk that requirements may change during development. This can lead to an implemented system that is out of date by the time it is completed [61].
Nicholas et al. [62] propose a stage model for IS implementation that illustrates, in a simplified form, typical project phases that are also found in other stage models. The four stages are: 1. Conception, in which the drivers of change are identified, feasibility is assessed, and external providers are short-listed; 2. Definition, which includes detailed analysis of the concept, system and functional requirements specification and detailed project planning; 3. Execution, when the IS is implemented, deployed and necessary training and process adjustment is undertaken; and 4. Operation, when system handover to the end-users is finalised, and systems maintenance procedures and service levels are put in place.
Agile methodologies for software development are also of relevance to chatbot projects. This way of developing new software is based on self-organised teams that can swiftly develop and deliver new software features, sometimes termed “product increments” [63]. There are a number of methodologies that provide guidelines for undertaking such development, including Scrum, Kanban, lean software development, and feature-driven development (FDD) [64,65]. The MVP (Minimum Viable Product) approach is based on agile methodology. It can be defined as a “version of a new product and allows teams to collect the maximum amount of validated learning about customers with the least effort” [66]. The main idea of this approach is to build the core functionality of the product, deploy it and get it used by early adopters as soon as possible. Given the complexity involved in AI related projects, agile methodologies are of particular value in that they are based on adaptive processes and strong communication amongst project participants [67].
Chatbot projects are characterised by a high degree of uncertainty in the early stages, unpredictability of the outcome, and the need to study user feedback in order to define new requirements. Many IT integrators support the use of the MVP method to address this uncertainty in their projects. Bansal et al. [68] argue that companies should grow their service from the basic form continuously improving its functionality by adding new features, integrations and skills moving towards the advanced level. This approach is supported by others who argue that the MVP method should encourage teams to design, develop and launch a quick pilot version of a chatbot to understand and assess customer reactions [37,41,52,69]. Once the first version with the core functionality is in operation, continuous improvement and optimisation activities can commence. They usually include customer feedback and behaviour research, AI models training and improvement, planning and adding new features incrementally, and additional user training.

3.5. Provisional Conceptual Framework

The provisional conceptual framework (PCF) for the research draws upon aspects derived from the literature discussed above. The CSF approach is adopted, but the list of factors presented in Table 3 is not seen as a definitive set of tools for the research. Further organization and prioritisation are required, and to this end the TOE framework is used, albeit in a modified manner. The Technology and Organisation categories of the framework are retained, but the Environmental element is excluded from the PCF, as there is little evidence of such factors in the CSFs listed in Table 3, and the likelihood of such aspects influencing chatbot projects is relatively low. However, “User Needs” is seen as a logical third category, which aligns with Brandtzaeg et al.’s [45] assertion that productivity is the most significant chatbot criterion and that this relates to a positive user experience.
Chatbot projects, like many other new IS implementations, undergo certain similar phases. Every project stage can be characterised by a certain set of the most important events and key actions that the team takes. Thus, for each stage there are corresponding CSFs, that positively influence the success of a particular stage, as well as create the prerequisites for subsequently successfully solving these problems. An analysis of the preliminary list of CSFs (Table 3) suggests that many of them are relevant to a particular project stage of a project. Classifying the CSFs according to a particular project stage brings a project-oriented perspective to the PCF, and to this end Nicholas et al.’s [62] four-step project life cycle model is utilised to provide further structure to the PCF.
In summary, the PCF comprises the following elements in an X-Y axis used to categorise CSFs: a modified TOE framework which classifies the various sources that influence chatbot projects using three change categories (Technology, Organisation and User Needs); and project stages, based on Nicholas et al.’s model [62]. The CSFs listed in Table 3 are provisionally allocated to the appropriate cell in the x-y axis as depicted in Table 4. This represents the researchers’ assessment of these CSFs based on existing literature sources. The resultant framework indicates that the CSFs are distributed more or less evenly over the cells of the matrix. This suggests that each of the change categories (Technology, Organisation, User Needs) has an impact on the project at each project stage, and that the identification of new CSFs and the refinement of existing ones can be effectively represented within this framework, which provides the basis for the primary case study research, through which its structure, elements and listed success factors are analysed, refined, expanded and validated.

4. Results and Discussion

This section directly addresses the three RQs drawing on the literature review presented in section 3 and on the primary interview material, using interviewee codes noted in Table 1. Documented notes from researcher observations are also cited, as appropriate, as a further data source.

4.1. RQ1. What Are the Main Challenges Which Companies Face When Implementing Chatbot Projects?

The three data sources noted above were analysed in conjunction to ascertain the key challenges. The challenges derived from an analysis of the literature were categorised and allocated to the appropriate cell in the PCF matrix. Then, the interview findings were analysed to produce a summary list of challenges raised in the interviews, and the individual statements of each interviewee were grouped into more general and universal formulations. In addition, the notes from the observations were similarly scanned and challenges were identified in the narrative text. The challenges identified from these three sources were then combined, via a process of aggregation and re-formulation. The results contain items from three sources, grouped by challenge “category”, as defined in the PCF, and ranked by a “score” value (Table 5).
Kaushal et al. [30], who explore different platforms for creating chatbots, point to the lack of out-of-the-box integration with popular software products, as well as the amount of work required to customise platforms and integrate with enterprise IS. This view was confirmed by many interviewees. Interviewee 1, for example, states that "the amount of work involved in integrating a chatbot with data sources is often underestimated during planning". Interviewee 4 adds that "integrating a chatbot with other business systems is particularly difficult and unpredictable". These statements are also supported by an analysis of meeting minutes and lessons learned during the implementation of the chatbot at ESteelCo, which indicates that the team underestimated the amount of work required to integrate the chatbot with the CRM system, resulting in a significant shift in the project start date. Dealing with complexity and amount of system integration work is thus one of the key challenges (Table 5).
The Score parameter reflects the result of a quantitative assessment of the results obtained. It is calculated as follows: each interviewee is assigned an experience level from 1 to 3, measured in points. Junior employees with little chatbot experience are assigned 1 point, senior specialists or project managers with more chatbot experience get 2 points, and chatbot experts with solid experience in integrating chatbots receive 3 points. The score is calculated as the sum of the points of the interviewee who mentioned the challenge. This provides a way of ranking the challenges so that chatbot implementation teams can better assess project risks and plan appropriate activities.

4.2. RQ2. What Are the Critical Success Factors (CSFs) That Contribute to the Successful Implementation of Chatbots in Companies?

The process of identifying CSFs was similarly based on the principle of triangulation, using three sources of data: the extant literature, the interview data and the observation notes. The critical success factors identified in the literature review were noted in section 3.2 and presented in Table 3.
During the interview sessions interviewees were asked to identify critical success factors for successful chatbot implementation. The interview script structure was entirely consistent with the PCF depicted in Table 4. The respondents named success factors concerning the four project stages (Conception, Definition, Execution, and Operation) and three change categories (Technology, Organisation, and User Needs) and discussed respective actions which help companies to meet their objectives. The process of merging CSFs from the three sources was carried out by finding identical or similar formulations, and by aggregating similar or complementary ideas into more general and universal statements to ensure that one CSF is not repeated in similar formats. This process is shown schematically in Figure 3.
As an illustrative example of this process, a number of literature sources maintained that the use of an agile, iterative approach, and the MVP method, are recommended when implementing chatbots [38,40,41]. This was discussed more broadly by several interviewees. Interviewee 12, for example, recommends minimising the size of the first MVP by "implementing 2-3 of the simplest but most popular scenarios in the first iteration". Interviewee 13 supports this, explaining that "launching a minimum viable product version quickly allows testing of the hypothesis and adjustment of the project roadmap". This view is confirmed by the observation findings and analysis of the Lessons Learned meeting minutes, which took place after the launch of the chatbot. Specifically, the team concluded that "an iterative approach and MVP method can be an effective way to manage chatbot development projects, especially when the expected results and outcomes are not clearly defined". In this way, after processing the information obtained from the analysis of publications, interview transcripts and observations, more general formulations of the CSFs were developed and located in the appropriate cell of the PCF.
This process of combining statements and calculating the Score value uses a similar approach to that described in sub-section 4.1. Here, items that received a score of 3 and below are excluded from the result, as in most cases they represent individual judgements of single respondents that have not been confirmed by other sources. The results are presented in Table 6, using the structure of the PCF. The identified CSFs allow implementation teams to strategically plan and execute chatbot projects and increase the chances of a successful chatbot launch.

4.3. RQ3. What Operational Model can Support the Chatbot Implementation Process and Help Responsible Managers to Deliver Expected Chatbot Project Outcomes?

The PCF played a key role during the practical and analytical phases of the study, helping to shape the data collection and processing activities. The CSFs provide a foundation for building the broader operational model for chatbot project development and implementation. The PCF structure facilitates the allocation of the CSFs into the appropriate cell in the matrix of change categories (Technology, Organisation and User Needs) and project phases (Conception, Definition, Execution, Operation), as depicted in Table 6. This represented an initial operational model, but further reflection and adjustment produced an enhanced, more organised and dynamic operating model with a more compact presentation.
First, a critical evaluation of the initial model suggested that such a large amount of information may prove difficult to work with in practice. The total number of CSFs was 59, far exceeding the typical number of 5-8 items suggested by Rockart [35]. Researchers have used different approaches to organise CSFs into different hierarchical structures and groups, and here grouping and aggregation were used to identify seven main CSFs, while maintaining a longer list of secondary factors.
Second, a number of CSFs that apply across several of the categories and stages of the model were identified, and these were taken out of the tabular structure of the matrix to sit above the others as more generally applicable. This also reflected the view of some interviewees that relating certain CSFs to specific cells in the PCF was problematic, suggesting more general factors to be followed across the stages of the project.
For example, the CSF - 'Focus on chatbot productivity' is listed in the definition phase of the User Needs category, although this factor is more global in nature. Interviewees 2, 3, 4, 6, 7, 8, 9 mentioned this factor as important but were unsure of the exact location of this factor in the PCF. Interviewee 8 explained that “he would like to attribute this CSF to the definition stage because it is important to think about chatbot productivity already when you plan your project and create specification. However, this is not really limited by the Definition stage, but rather a general rule which should be considered by the teams at each stage”. Focusing on chatbot productivity can be justified in almost any action to solve a variety of problems, such as choosing a technology and platform, setting goals, formulating technical requirements, defining functionality, planning the development backlog, testing, etc. So this CSF was removed from within the PCF matrix and relocated outside of it (Figure 4 - transition A).
Similarly, some CSFs were deemed relevant to a particular category, but across all project stages. However, at each stage, this CSF is characterised by specific actions that are relevant to that particular stage. For example, "Use the MVP method and iterative approach" is assigned to the Definition phase, but there are other entries associated with it in all the other project stages (Figure 4 – transition B). The use of MVP and agile methodologies is a recommendation that is relevant throughout the project. As the CSF factor is formulated in a more general way, it no longer corresponds to a specific cell in the two-dimensional matrix and can be taken out of it. At the same time, at each stage in the Organisation line, following this recommendation can lead to specific actions that correspond to the current stage of the project. In this way, these entities can be linked hierarchically - the CSF, which exists in the context of the whole project, and the associated activities, which are derivatives of this CSF and are recommended for execution according to the project stage. This dependency is shown schematically in Figure 5.
This process of aggregating model elements resulted in a reduced list of more general CSFs that fall outside the scope of the tabular structure of the PCF. The remaining CSFs are left within the cell matrix but have been reformulated as local and specific tasks or “Key Actions”. These can be seen as derivatives of the CSFs and respond to the key issues that teams need to address within the specified project stages. They are of value in moving the model from a mainly strategic level to facilitate project execution. Seven CSFs are placed outside the matrix, organised horizontally, and are more general entities which provide a broad conceptual and operational platform upon which the Key Actions can be grounded.
Third, existing literature suggests CSFs can be used to link strategic objectives to operational activities. In the context of IS strategy, Earl [58] put forward a three-level structure in which the CSFs are the element that links the company's strategic goals to tactical planning activities. A similar construct is presented by Bullen et al. [36], in which CSFs link an objective to key activities.
The incorporation of a strategy level based on objectives enhances the operational model, making it more applicable to project planning and control activities in chatbot integration projects. This can be achieved by moving the CSF - "Focus on one business objective" - to a higher level and adjusting its wording. Trying to implement a chatbot in several different business processes at the same time can lead to a lack of focus and insufficient investment in the implementation of each of the planned functions, which significantly reduces the quality of the final result and most likely leads to the failure of the chatbot. Solving a single but important task will significantly improve the likelihood of a successful project launch. In project management, the key metrics that measure the success of the project are usually associated with the project goal or objective. This is reflected in the model, which assumes that the activities to define the project aim, the success indicators and the corresponding metrics are carried out in the Definition phase and fall under the Organisation category. The updated construct resulting from the creation of the new Objectives layer is shown in Table 7.
The new model has thus been developed from the PCF to comprise three conceptual and operational layers: the Objectives layer, the CSFs, and Recommended Key Actions. The top Objectives layer concerns information about strategic objectives of the project and specifies appropriate metrics for assessing achievement of these objectives. The model calls for limiting the functionality of the first version of the chatbot, strictly focusing on one business objective and choosing one or more metrics that match that objective.
The central part of the model is the CSFs layer which contains 6 CSFs that are applicable to different projects and should be discussed by the team in strategic meetings. The Recommended Key Actions (the bottom layer) are derived from the CSFs and should be viewed as a flexible set of tasks that can be modified, removed or added to, depending on the organisational context and specific project requirements. They are allocated to the three change categories that affect chatbot projects: those of a technological nature, organisational aspects, and user needs; and to the four phases, which are typical for most projects: Conception, Definition, Execution, and Operation.
The application of the model facilitates a holistic and comprehensive approach to planning and managing virtual chatbot implementation projects. By linking the strategic level of business objectives, critical success factors and key actions, the new model creates the foundation for successfully building and launching virtual assistants. This was confirmed by experts during the validation process, who confirmed their view that this model can make chatbot projects more manageable and increase the chances of success.
The validation process involving five experts consisted of three steps. First, the experts were provided with a new model for chatbot implementation, along with a detailed description. Next, a discussion of the model took place between the researcher and the experts, focusing on the model's structure, content, and application recommendations. Finally, the experts independently completed a questionnaire evaluating four aspects of the model: evaluation of the model structure, evaluation of the CSFs, evaluation of the Key Actions, and evaluation of the new model in general. Overall, the experts were positive in their support of the model, albeit with some suggested minor modifications which have been incorporated in Table 7.

4.4. Model Application and Guidance

The model is best seen as a flexible tool that can be used alongside established planning methods and techniques. It is most effectively deployed by adopting a top-down approach. The preliminary stage requires the development of the concept and formulation of the project idea. Then the project team are best served by focusing on one business objective, after which time the model can be utilised. In addition to agreeing the key business objective, one or more metrics that directly measure the achievement of the objective should be specified. If this proves difficult to achieve at the outset of the project, metrics can be added or refined later in the Conception stage.
After defining objectives and selecting success metrics, the CSFs and Key Actions layers of the model should be explored sequentially. The six CSFs are general formulations that can be applied in different phases across the duration of projects, so teams are encouraged to discuss them during planning and review sessions to establish a link with the real context of the organisation. By exploring the relevance of each CSF to a particular project, project stakeholders gradually prioritise their requirements and gain a comprehensive understanding of the challenges, issues, risks and opportunities associated with the upcoming chatbot implementation.
The CSFs are reflected and interpreted in the Key Actions layer in the form of recommended tasks and actions. The Key Actions layer is the most detailed yet flexible part of the model, and teams may find it necessary to adapt them to particular project circumstances. For example, the task "Find an expert to join your team" may not be relevant for all teams. At the same time, the specifics of the organisation's business, IT environment, internal rules and procedures may require important specific tasks to be solved that are not listed in the operational model. For example, there may be a number of important tasks related to integration with specific IS, creating new micro-services to implement specific planned bot functions, and preparing necessary data.
From a project lifecycle point of view, it is advisable to use the operating model as early as possible - as soon as the idea of potentially using a chatbot in a company has been raised. The model is designed to help teams with strategic planning and to shape the project's action plan, so its role is most important at the very beginning. However, the project manager and other team members can refer to the model periodically throughout the course of a project. This can help to check how well the current tasks are aligned with the CSFs. If there are significant gaps or problems, or significant changes to the project plan are needed, the operating model can be a useful tool for re-planning. Repeating the previous planning activities, from validating the key objective and reviewing the CSFs to defining the Key Actions, will facilitate the review and update of the project plan.
The model can also be used post-implementation as the final stage in the model (Operation) includes the list of actions and factors that are relevant to project servicing and maintenance. An iteratively evolving chatbot service will return periodically to the Execution phase to develop and release new versions with improved capabilities and features. Using the parts of the model associated with these stages, teams can be given guidance on how to adjust their work and improve their approaches. At the same time, the model can support the on-going management of chatbots as a check-list against current management approaches and project outcomes can be assessed and adjusted as appropriate. A review of the overall objective, metrics, CSFs and Key Actions can facilitate adjustments that improve project organisation and outcomes. The model can help identify and address gaps, solve known problems or improve the performance of an already successful project.

5. Conclusions

5.1. Contribution to the Theory and Practice

The main contribution of this research is the new model for successful chatbot implementation, which is designed to support the development process and deliver the expected outcomes of such projects. This will be of interest to researchers and practitioners involved in the implementation of virtual assistants and other projects related to new technology adoption. It provides answers to common questions about chatbot integration and offers a holistic, universal approach that links strategic business objectives to the practical steps needed to achieve successful chatbot projects.
The design of the model builds upon several widely used models and frameworks, the elements of which are combined into a new structure. This research confirms the universality and wide applicability of the TOE framework [52], as well as the possibility and suitability of making modifications to its structure depending on the context of application. The stage model by Nicholas et al. [62] became a part of the design of the new model, adding a project management perspective to the implementation of chatbots, and illustrating how such models can be used effectively in combination with other models, complementing the design with a process-oriented approach. The top-down approach proposed by Earl [58] has proven its flexibility and applicability in the context of addressing contemporary project planning issues. Finally, the popular CSF method [36] was used as a tool to investigate the factors that determine the success of chatbot implementation projects. This study thus extends the existing knowledge of the models and frameworks mentioned above, and demonstrates the possibilities of their application in a new context, as well as in combination with each other.
Flexibility, versatility and its generic nature are the characteristics of this model, which is designed to support teams from different countries and regions in chatbot development projects. It is relevant to projects across a range of industries, especially those aimed at automating communication between companies and their customers. It is designed for large and medium-sized companies that want to automate routine processes, although the model can also be used in small company projects, in which case managers applying the model are advised to omit redundant elements. The model does not specifically address voice assistants, but teams building voice bots may still find it valuable in considering the specific aspects of voice development.
Although the model may be of most use to project managers responsible for implementing chatbots in organisations, it can also be useful for other team members and stakeholders, including the project sponsor, IT directors and representatives from the contractor companies. IT integrator teams that provide chatbot implementation services can also benefit from its use by improving their processes, and understanding the needs of client organisations and meeting their expectations.

6.2. Limitations and Future Research

Despite the contribution of this research outlined above, it clearly has its limitations. From a methodological viewpoint, the selection of interviewees was limited by the availability of appropriate personnel. However, fifteen respondents took part in the study, representing six industries, three countries and speaking two languages. This diversity allowed the identification of characteristics of chatbot projects that are applicable cross-industry and internationally. Nevertheless, additional interviews could enhance and improve the possibilities for generalisation and wider applicability of the model.
Although the research focused on just one case study, the use of a multi-method and two data collection techniques - interviews and observation - maximised the value of the available data. Further research could include parallel cases studies in different companies in various industry sectors that would provide a richer and more diverse set of research data. The model is generic enough to be configurable and customizable. The Key Actions layer, in particular, should be considered as a preliminary list of broad recommendations that can be significantly expanded and adapted to the particular circumstances of each project. Practical testing of the model in real project implementations can help evolve the model, and the use of an action research strategy would provide an opportunity to study successive cycles of chatbot development, thereby enabling iterative improvement and optimisation. A focus on specific issues such as approaches and methodologies for platform selection and the impact on project outcomes would also be of value.
Finally, it is essential that the framework is regularly reviewed and updated in line with the latest technological advances in the industry to ensure the relevance of the model and its potential application in practice. In recent years, there has been a rapid development of LLMs and their application to solving practical problems, including the creation of chatbots. OpenAI's ChatGPT, for example has shown impressive results since the end of 2022, and many experts predict that this technology will transform entire industries in the near future. The rapid evolution of this technology will require assessment and incorporation of updates and revisions to the model put forward here. ChatGPT can be considered as one of the most advanced NLP tools, but nevertheless, the issues addressed in this research, as well as the new model to aid teams in solving chatbot problems in the context of an organisation, remain relevant today.

Author Contributions

Conceptualization, A.S. and M.W.; methodology, A.S. and M.W.; validation, A.S.; formal analysis, A.S.; investigation, A.S.; data curation, A.S.; writing—original draft preparation, A.S.; writing—review and editing, M.W.; visualization, A.S.; supervision, A.S. and M.W.; project administration, A.S. and M.W. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Data Availability Statement

The data presented in the article derives from confidential interview transcripts and observation notes. Further inquiries can be directed to the corresponding author.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. McTear, M. Conversational AI, 1st ed.; Springer, 2022. [Google Scholar]
  2. Jurafsky, D.; Martin, J.H. Speech and Language Processing. In Speech and Language Processing; Taylor & Francis, 2010; pp. 16–28. [Google Scholar] [CrossRef]
  3. Cambridge Dictionary. https://dictionary.cambridge.org/dictionary/english/chatbot (accessed on 2 July 2022).
  4. Hoy, M. Alexa, Siri, Cortana, and More: An Introduction to Voice Assistants. Medical Reference Services Quarterly 2018, 37, 81–88. [Google Scholar] [CrossRef] [PubMed]
  5. Goldberg, Y. Neural Network Methods for Natural Language Processing. Synthesis Lectures on Human Language Technologies 2017, 10, 1–309. [Google Scholar] [CrossRef]
  6. Bernardini, A.; Sônego, A.; Pozzebon, E. Chatbots: An Analysis of the State of Art of Literature. In Proceedings of the 1st Workshop on Advanced Virtual Environments and Education; 2018; pp. 1–6. [Google Scholar] [CrossRef]
  7. Devlin, J.; Chang, M.; Lee, K.; Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv 2018, arXiv:1810.04805v2. [Google Scholar]
  8. Horev, R. BERT Explained: State of the art language model for NLP. Medium. November 17, 2018. https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270.
  9. Floridi, L.; Chiriatti, M. GPT-3: Its Nature, Scope, Limits, and Consequences. Minds and Machines 2020, 30, 681–694. [Google Scholar] [CrossRef]
  10. OpenAI. GPT-4. 2023, March 14. https://openai.com/research/gpt-4.
  11. Chui, M. The state of AI in 2023: Generative AI’s breakout year. McKinsey & Company. https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2023-generative-ais-breakout-year (accessed on 01 March 2024).
  12. Williams, S. Hands-On Chatbot Development with Alexa Skills and Amazon Lex: Create Custom Conversational and Voice Interfaces for Your Amazon Echo Devices and Web Platforms. 2018. [Google Scholar]
  13. CB Insights. Lessons from the Failed Chatbot Revolution — And 7 Industries Where the Tech Is Making A Comeback. CB Insights Research 2021, January 14. https://www.cbinsights.com/research/report/most-successful-chatbots/.
  14. Fach, P.; Tranker, N.; Schüssler, S. Deloitte Survey | Chat Talk Touch. Deloitte 2019, December 20. https://www2.deloitte.com/content/dam/Deloitte/de/Documents/financial-services/deloitte-survey-chat-talk-touch.pdf.
  15. Gwendal, D.; Cabot, J.; Deruelle, L.; Derras, M. Xatkit: A Multimodal Low-Code Chatbot Development Framework. IEEE Access 2020, 8, 15332–15346. [Google Scholar] [CrossRef]
  16. Janssen, A.; Grützner, L.; Breitner, M.H. Why do Chatbots fail? A Critical Success Factors Analysis. ICIS 2021 Proceedings.
  17. Skuridin, A. Chatbot Implementation in a Steel Company in Russia. In Handbook of Research on Digital Transformation, Industry Use Cases, and the Impact of Disruptive Technologies; Wynn, M., Ed.; IGI-Global: Hershey, PA, USA, 2022; pp. 268–290. [Google Scholar] [CrossRef]
  18. Yin, R. Case Study Research. Design and Methods, 3rd ed.; Sage Publications Inc.: London, 2009. [Google Scholar]
  19. Ritchie, J.; Lewis, J. Qualitative Research Practice—A Guide for Social Science Students and Researchers; Sage Publications Ltd: London, Thousand Oaks, CA, 2003. [Google Scholar]
  20. Guest, G.; Bunce, A.; Johnson, L. How Many Interviews Are Enough? An Experiment with Data Saturation and Variability. Field Methods. 2006, 18, pp. 59–82. Available online: http://fmx.sagepub.com/cgi/content/abstract/18/1/59 (accessed on 18 June 2023).
  21. Kuzel, A. Sampling in qualitative inquiry. In Doing qualitative research; Crabtree, B., Miller, W., Eds.; SAGE Publications Ltd: Newbury Park, CA, USA, 1992; pp. 31–44. [Google Scholar]
  22. Kumar, R. Research Methodology: A Step-by-Step Guide for Beginners, 4th ed.; Sage, 2014. [Google Scholar]
  23. Trochim, W.M.K.; Donnelly, J.P. Research Methods Knowledge Base, 3rd ed.; Thomson Custom Pub, 2007. [Google Scholar]
  24. Saunders, M.; Lewis, P.; Thornhill, A. Research Method for Business Students; Pearson Education Limited: Harlow, United Kingdom, 2016. [Google Scholar]
  25. Guba, E.G.; Lincoln, Y.S. Competing Paradigms in Qualitative Research. In Handbook of Qualitative Research; Denzin, N.K., Lincoln, Y.S., Eds.; Sage Publications, Inc., 1994; pp. 105–117. [Google Scholar]
  26. Patton, M.Q. Enhancing the Quality and Credibility of Qualitative Analysis. Health Serv. Res. 1999, 34 5 Pt 2, 1189–1208. [Google Scholar]
  27. Bell, E.; Bryman, A.; Harley, B. Business Research Methods; Oxford University Press: Oxford, England, 2018. [Google Scholar]
  28. Porter, A.L.; Kongthon, A.; Lu, J.C. Research Profiling: Improving the Literature Review. Scientometrics 2002, 53, 351–370. [Google Scholar] [CrossRef]
  29. Srinivasan, K.; Nguyen, C.; Tanguturi, P. Chatbots Are Here to Stay. Accenture, February 6, 2021. https://www.accenture.com/_acnmedia/pdf-77/accenture-research-conversational-ai-platforms.pdf.
  30. Kaushal, V.; Yadav, R. Exploring B2B Chatbots Adoption Experiences: Lessons for Successful Implementation in Businesses. Research Square. 2022. [Google Scholar] [CrossRef]
  31. Dilmegani, C. An Assessment in 2022 For Why Chatbots Fail. AIMultiple, June 14, 2022. https://research.aimultiple.com/why-chatbots-fail/.
  32. Varitimiadis, S.; Kotis, K.; Pittou, D.; Konstantakis, G. Graph-Based Conversational AI: Towards a Distributed and Collaborative Multi-Chatbot Approach for Museums. Appl. Sci. 2021, 11, 9160. [Google Scholar] [CrossRef]
  33. Drift. The State of Chatbots Report: How Chatbots Are Reshaping Online Experiences. Drift, January 23, 2018. https://www.drift.com/blog/chatbots-report/.
  34. Atkinson, R. Project Management: Cost, Time, and Quality, Two Best Guesses and a Phenomenon, It’s Time to Accept Other Success Criteria. Int. J. Proj. Manag. 1999, 17, 337–342. [Google Scholar] [CrossRef]
  35. Rockart, J.F. Chief Executives Define Their Own Data Needs. Harv. Bus. Rev. 1979, 57, 81–93. [Google Scholar] [PubMed]
  36. Bullen, C.; Rockart, J. A Primer on Critical Success Factors. Sloan School of Management, 1981.
  37. Franke, A. How May A.I. Assist You? KPMG, January 10, 2018. https://advisory.kpmg.us/articles/2018/how-may-ai-assist-you.html.
  38. Henke, N.; Bughin, J.; Chui, M.; Manyika, J.; Saleh, T.; Wiseman, B.; Sethupathy, G. The Age of Analytics: Competing in a Data-Driven World. McKinsey & Company, December 20, 2016. https://www.mckinsey.com/~/media/McKinsey/Industries/Public%20and%20Social%20Sector/Our%20Insights/The%20age%20of%20analytics%20Competing%20in%20a%20data%20driven%20world/MGI-The-Age-of-Analytics-Full-report.pdf.
  39. Sengupta, R.; Lakshman, S. Conversational Chatbots – Let's Chat. Deloitte, February 6, 2017. https://www2.deloitte.com/content/dam/Deloitte/in/Documents/strategy/in-strategy-innovation-conversational-chatbots-lets-chat-final-report-noexp.pdf.
  40. Bakhshi, N.; van den Berg, H.; Broersen, S.; de Vries, D.; El Bouazzaoui, H.; Michels, B. Chatbots Point of View. Deloitte, 2018. Available at: https://www2.deloitte.com/content/dam/Deloitte/nl/Documents/deloitte-analytics/deloitte-nl-chatbots-moving-beyond-the-hype.pdf (accessed on 15 January 2021).
  41. Robinson, M.; Gray, J.; Cowley, A.; Tan, R. Adopting the Power of Conversational UX. Deloitte, 2017. Available at: https://www2.deloitte.com/content/dam/Deloitte/nl/Documents/financial-services/deloitte-nl-fsi-chatbots-adopting-the-power-of-conversational-ux.pdf (accessed on 26 March 2021).
  42. Dasgupta, A.; Wendler, S. AI Adoption Strategies; Working Paper Series, No. 9; 2019. [Google Scholar]
  43. Bughin, J.; Hazan, E.; Ramaswamy, S.; Chui, M.; Allas, T.; Dahlström, P.; Henke, N.; Trench, M. Artificial Intelligence: The Next Digital Frontier? McKinsey Global Institute, 2017. Available at: https://www.mckinsey.com/~/media/mckinsey/industries/advanced%20electronics/our%20insights/how%20artificial%20intelligence%20can%20deliver%20real%20value%20to%20companies/mgi-artificial-intelligence-discussion-paper.ashx (accessed on 01 March 2024).
  44. Fokina, M. The Future of Chatbots: 80+ Chatbot Statistics for 2023. Tidio, January 18, 2023. https://www.tidio.com/blog/chatbot-statistics/.
  45. Brandtzaeg, P.B.; Folstad, A. Why People Use Chatbots. In Internet Science; 2017. [Google Scholar]
  46. Isbister, K.; Nass, C. Consistency of personality in interactive characters: verbal cues, non-verbal cues, and user characteristics. International Journal of Human-Computer Studies 2000, 53, 251–267. [Google Scholar] [CrossRef]
  47. Katz, S. The Ultimate Guide to Chatbot Personality. Chatbots Magazine, November 20, 2019. Available at: https://chatbotsmagazine.com/the-ultimate-guide-to-chatbot-personality-b9665ab5e99d (accessed on 20 March, 2021).
  48. Liew, T.W.; Tan, S. Virtual Agents with Personality: Adaptation of Learner-Agent Personality in a Virtual Learning Environment. In Proceedings of the 2016 Eleventh International Conference on Digital Information Management (ICDIM); pp. 157–162.
  49. Ruane, E.; Farrell, S.; Ventresque, A. User Perception of Text-Based Chatbot Personality. In Lecture Notes in Computer Science; 2021; pp. 32–47. [Google Scholar] [CrossRef]
  50. Smestad, T.L. Personality Matters! Improving the User Experience of Chatbot Interfaces - Personality Provides a Stable Pattern to Guide the Design and Behaviour of Conversational Agents. 2018.
  51. Overend, R.; Stefanoff, S.; Jackson, P.; Rayment, S.; Cooper, S.; Taylor, D. Conversational AI. The Next Wave of Customer and Employee Experiences. Deloitte, 2019. Available at: https://www2.deloitte.com/content/dam/Deloitte/au/Documents/strategy/au-deloitte-conversational-ai.pdf (accessed on 12 March 2023).
  52. DePietro, D.; Wiarda, E.; Fleischer, M. Processes of Technological Innovation; Lexington Books: Massachusetts, MA, 1990. [Google Scholar]
  53. Awa, H.O.; Ukoha, O.; Emecheta, B.C. Using T-O-E Theoretical Framework to Study the Adoption of ERP Solution. Cogent Business & Management 2016, 3, 1196571. [Google Scholar] [CrossRef]
  54. Racherla, P.; Hu, C. ECRM System Adoption by Hospitality Organizations: A Technology-Organization-Environment (TOE) Framework. J. Hosp. & Leis. Mark. 2008, 17, 30–58. [Google Scholar] [CrossRef]
  55. Oliveira, T.; Martins, M.F.O. Understanding the Determinant Factors of Internet Business Solutions Adoption: The Case of Portuguese Firms. Appl. Econ. Lett. 2011. [Google Scholar] [CrossRef]
  56. Il’in, V.I.; Ivetić, J.; Simić, D. Understanding the Determinants of E-Business Adoption in ERP-Enabled Firms and Non-ERP-Enabled Firms: A Case Study of the Western Balkan Peninsula. Technol. Forecast. Soc. Change 2017, 125, 206–223. [Google Scholar] [CrossRef]
  57. Wynn, M.; Bakeer, A.; Forti, Y. E-Government and Digital Transformation in Libyan Local Authorities. Int. J. Teach. and Case Stud. 2021, 12, 119–139. [Google Scholar] [CrossRef]
  58. Earl, M.J. Management Strategies for Information Technology, 1st ed.; Prentice Hall, 1989. [Google Scholar]
  59. Blake, J. Project Managing the SDLC: Using Milestones to Align Project Management and System Development Lifecycles and Report Project Success. Paper presented at PMI® Global Congress 2004, North America, Anaheim, CA; Project Management Institute: Newtown Square, PA, 2004. [Google Scholar]
  60. Buede, D.M.; Miller, W.D. The Engineering Design of Systems: Models and Methods; Wiley, 2016. [Google Scholar]
  61. Foster, E.C. Software Engineering: A Methodical Approach; Apress, 2014. [Google Scholar] [CrossRef]
  62. Nicholas, J.M.; Steyn, H. Project Management for Engineering, Business, and Technology, 5th ed.; Routledge: London, 2017. [Google Scholar]
  63. Atlassian. What is the Agile methodology? Atlassian. https://www.atlassian.com/agile (accessed on 1 March 2024).
  64. Dingsoyr, T.; Nerur, S.; Balijepally, V.; Moe, N.B. A Decade of Agile Methodologies: Towards Explaining Agile Software Development. J. Syst. Softw. 2012, 85, 1213–1221. [Google Scholar] [CrossRef]
  65. Sutherland, J. Scrum: A Revolutionary Approach to Building Teams, Beating Deadlines, and Boosting Productivity; Random House, 2014. [Google Scholar]
  66. Ries, E. Minimum Viable Product: A Guide. Startup Lessons Learned, August 3, 2009. Available at: http://www.startuplessonslearned.com/2009/08/minimum-viable-product-guide.html (accessed on 10 June 2023).
  67. Abrahamsson, P.; Conboy, K.; Wang, X.; Baskerville, R.; Fitzgerald, B.; Morgan, L. Agile Processes in Software Engineering and Extreme Programming; Springer: Berlin Heidelberg, Germany, 2008. [Google Scholar]
  68. Bansal, A.; Salwan, S.; Thakker, V. Chatbots. Riding the Next-Gen Technology Wave to Operational Success. Deloitte, 2019. Available at: https://www2.deloitte.com/content/dam/Deloitte/in/Documents/technology-media-telecommunications/in-tmt-chatboats-noexp.pdf (accessed on 15 February 2024).
  69. Guenole, N.; Feinzig, S. The Business Case for AI in HR. IBM Watson Talent, 2018. Available at: https://www.ibm.com/downloads/cas/AGKXJX6M (accessed on 2 March 2024).
Preprints 102603 g001
Figure 1. The four phases of the research process.
Figure 1. The four phases of the research process.
Preprints 102603 g001
Preprints 102603 g002
Figure 2. The TOE framework (Based on DePietro et al. [52]).
Figure 2. The TOE framework (Based on DePietro et al. [52]).
Preprints 102603 g002
Preprints 102603 g003
Figure 3. Data analysis using the principle of triangulation to identify CSFs.
Figure 3. Data analysis using the principle of triangulation to identify CSFs.
Preprints 102603 g003
Preprints 102603 g004
Figure 4. Reviewing CSFs: adjusting the PCF matrix.
Figure 4. Reviewing CSFs: adjusting the PCF matrix.
Preprints 102603 g004
Preprints 102603 g005
Figure 5. Link formation between CSFs and derivative actions.
Figure 5. Link formation between CSFs and derivative actions.
Preprints 102603 g005
Table 1. List of interviewees.
Table 1. List of interviewees.
Interviewee code Job title Industry Country Interview language Internal or external
01 Project manager Metallurgy Russia Russian Internal
02 Project manager IT integration Russia Russian Internal
03 Analyst Metallurgy Russia Russian Internal
04 Analyst Metallurgy Russia Russian Internal
05 Senior Marketing Manager Pharmaceutical Germany English External
06 Business Processes Director Pharmaceutical Germany English External
07 Project manager Banking Russia Russian External
08 Product owner Metallurgy Russia Russian Internal
09 Analyst Metallurgy Russia Russian Internal
10 Marketing director Metallurgy Russia Russian Internal
11 CRM director Online retail Russia/China Russian External
12 Director Metallurgy Russia Russian Internal
13 IT Director Online retail Russia Russian External
14 IT Director Fintech Russia Russian External
15 IT developer Online retail Russia Russian External
Table 2. Validation experts consulted in the study.
Table 2. Validation experts consulted in the study.
Expert No. Was involved as an interviewee? Characteristics
01 Yes Senior IT manager and department head in a large company. Responsible for managing a programme of new IS integration projects including CRM, e-commerce platforms, AI application and chatbots. Experienced methodologist who organised the production of IT projects in an international IT integrator. Participated in several chatbot implementation projects as project leader.
02 No Head of project implementation department in a large IT integrator. Leads the development of multiple projects including new CRM integration, website development and chatbot integration.
03 No Professor in Digital Skills with experience in AI, cyber security, and data science.
04 Yes Director of digital transformation in a large enterprise. Responsible for managing a programme of IT projects including ERP, CRM, websites and various AI application projects including chatbots. Previously involved in several chatbot implementation projects, either as project lead or key stakeholder. Holds a PhD.
05 No IT Director in a medium-sized company. Experienced in managing IS integration projects that include ERP, CRM, eCommerce platforms, AI applications and chatbots.
Table 3. CSFs identified from the extant literature.
Table 3. CSFs identified from the extant literature.
No. Critical success factor
1 Identify use cases and assess suitability of using a conversational User Interface (UI) [37,38,39]
2 Focus on one business objective [40,41]
3 Analyse integration capabilities of the selected platform [37,40]
4 Define business value metrics aligned to the organisation strategy [37]
5 Consider security and compliance requirements [29]
6 Explore vendor options in selected industry field and locality [40]
7 Analyse build and buy approaches to define the strategy [42]
8 Use the buy (not build) approach and explore your local vendors [42]
9 Choose a platform that can provide detailed reports about chatbot performance [40]
10 Offer users an option to chat in their favourite messenger and provide multiple messenger support [39]
11 Apply agile iterative approach and MVP method [38,40,41]
12 Design an experience that provides value for both to the customer and business [41]
13 Use information about the customers to offer personalised service [43,44]
14 Focus on chatbot productivity. The ease, speed, and convenience of using chatbots is vital. Bots should solve users’ problems in a more efficient way than any other communication channel [41,45]
15 Use entertainment elements and social interaction elements for additional motivation [45]
16 Design chatbot personality [37,39,44,46,47,48,49,50]
17 Evaluate performance and identify which requests were not processed properly [40,41]
18 Invest time and resources in bot training [38,41,51]
19 Blended communication enables excellent service. Switch dialog to a human when a bot can't handle a user's request [34,39,44]
20 Create dashboard that displays key chatbot performance metrics [41]
21 Study the users' needs by collecting feedback and plan new features accordingly [37,41]
22 Continuously improve the chatbot by applying recent AI technologies [41]
23 Advertise bot’s capabilities to its users [37]
Table 4. Provisional conceptual framework for researching chatbot implementation.
Table 4. Provisional conceptual framework for researching chatbot implementation.
Project Stages/Change Categories Conception Definition Execution Operation
Technology Analyse build and buy approaches to define the strategy
Explore vendor options in selected industry field and locality
Analyse integration capabilities of the selected platform
Choose a platform that can provide detailed reports about chatbot performance		 Consider security and compliance requirements
Offer users an option to chat in their favourite messenger and provide multiple messenger support
 Create dashboard that displays key chatbot performance metrics Continuously improve the chatbot by applying recent AI technologies
Organisation Identify use cases and assess suitability of using conversational User Interface
Focus on one business objective
Define business value metrics aligned to the organisation strategy		 Apply agile iterative approach and MVP method Invest time and resources in bot training
Advertise bot’s capabilities to its users
User Needs Design an experience that provides value for both to the customer and business
Use information about the customers to offer personalised service
Apply blended communication approach		 Focus on chatbot productivity. The ease, speed, and convenience of using chatbots is vital
Use entertainment elements and social interaction elements for additional motivation
Design chatbot personality		 Evaluate performance and identify which requests were not processed properly
Study the users' needs by collecting feedback and plan new features accordingly
Table 5. Challenges when implementing chatbot projects.
Table 5. Challenges when implementing chatbot projects.
No. Challenge Score
TECHNOLOGY CATEGORY
Challenge 1 Complexity and amount of system integration work. 23
Challenge 2 Difficulty in finding the best product and vendor which can solve your problem and will offer its service for a long time 15
Challenge 3 Overall complexity of chatbot projects and necessity to deal with many software systems and IT technologies 13
Challenge 4 Chatbots require much calculation resources. Many companies are not ready to purchase required hardware 10
Challenge 5 Few companies are able to collect, analyse and use their internal data to develop appropriate chatbot behaviour 9
Challenge 6 The complexity of regulation and information security tasks 9
Challenge 7 Current chatbot technologies are complex but not yet perfect and bots often fail to process users' requests properly 9
Challenge 8 The license and support costs of a chatbot systems from leading vendors is too high, what forces teams to create their own solution 6
ORGANISATION CATEGORY
Challenge 9 Difficulty in formulating project objectives, success criteria and measuring KPIs 24
Challenge 10 Lack of in-house chatbot expertise and limited availability of relevant specialists in the market 21
Challenge 11 Top managers are often underestimate chatbot integration projects complexity and costs and not ready to invest significant budget in a new technology adoption 17
Challenge 12 Planning and managing chatbot projects is challenging due to their unpredictability and high uncertainty 11
Challenge 13 Low quality of integration services provided by vendors and IT integrators 10
USER NEEDS CATEGORY
Challenge 14 User scepticism about the capabilities of chatbots and unwillingness to interact with them 27
Challenge 15 User behaviour in chats is very different from other channels like phone, website, mobile app. High-quality UX design skills and user training are required. 11
Challenge 16 The scenario-based approach for creating bots prevails, but limits their capabilities 8
Challenge 17 Difficulty in creating a chatbot which solves the company’s problem and provides users with an excellent service the same time. 4
Table 6. Critical success factors for chatbot implementation.
Table 6. Critical success factors for chatbot implementation.
Project Stages/Change Categories Conception Definition Execution Operation
Technology Identify and choose the most suitable technology and vendor which meet your business requirements. (Score=26)
Explore vendor options and external service availability in your field and locality. (Score=24)
Choose a platform that can provide detailed reports about your chatbot performance. (Score=19)
Analyse build and buy approaches to define your strategy. (Score=17)
Analyse API of the selected platform and how to integrate it with the company's IS. (Score=10)
Study project portfolio of vendor candidates, request recommendations from their clients. (Score=9)
The selected platform should have a powerful dialog editor. (Score=6)		 Invest in analysis and planning of system integration and data migration tasks. (Score=23)
Define scalability requirements and realistically estimate the required hardware. (Score=13)
Obtain necessary access to IS and solve information security issues. (Score=13)
Strategically plan the compatibility of the bot platform with the company's internal IT systems. (Score=12)
Make sure you know exactly how you can develop the bot on your own or change the integrator company, if necessary. (Score=8)
Formulate comprehensive functional requirements that determine the result. (Score=5)
 Create dashboard for tracking chatbot behavioural metrics and success indicators after the launch. (Score=11)
Perform load testing, plan and prepare infrastructure for high load. (Score=9)
Prepare and use development, testing and production environment. (Score=9)
Use SLAs (service level agreements) to set performance requirements for involved IS. (Score=6)		 Monitor a chatbot availability and usage. (Score=19)
Continuously improve the chatbot by applying recent AI technologies, updating language models, and installing the platform updates. (Score=14)
Organisation Identify your aim, key success indicators and associated improvement metrics. (Score=24)
Identify use cases and assess suitability of using a conversational UI. (Score=19)
Find an expert to join your team. (Score=10)
Prepare the ground for agile working. (Score=9)		 Form a product team on the company's side which includes a product owner, analyst or dialog designer, data scientist, UX-designer, technical leader. (Score=23)
Use the MVP method and an iterative approach. (Score=18)
Find a responsible project leader from the business team who has the necessary expertise. (Score=9)
Involve top management into the process. (Score=6)
Communicate your aim to your team, contractors, and top management. (Score=5)		 Apply an agile iterative approach and MVP method. (Score=23)
Define the chatbot maintenance team, their responsibilities and train the key users. (Score=9)
Test your bot internally with your team, and with business experts in the company. (Score=7)
Pay attention to project control, reporting and communication with the contractor. (Score=4)		 Collect dialog data and invest time and resources in bot training. (Score=13)
Develop the chatbot expertise and grow your team in-house. (Score=9)
Continue to use the agile development approach. Turn it into a routine. (Score=9)
Make sure that the chatbot systems are accepted by IT support technical specialists. (Score=7)
Advertise bot features and share success stories within your company. (Score=5)
User Needs Focus on one business objective. (Score=20)
Analyse alternatives to the chatbot service. Specify why the chatbot is the best way to solve your problem. (Score=13)
Make sure the chatbot is not only good for the company but designed to make the customer’s life better. (Score=10)
Make sure the chatbot is the most convenient way for users to solve their problems. (Score=5)		 Focus on chatbot productivity. The ease, speed, and convenience of using chatbots are vital. (Score=16)
Use CJM and Jobs-To-Be-Done methods to capture user experience and specify chatbot requirements. (Score=16)
Choose the “easy-to-develop” functionality. Publish limited but well-developed functionality. (Score=14)
Design a chatbot personality which fits your brand communication style. (Score=13)
Focus on intuitive and simple UX. (Score=9)
Apply blended communication approach. (Score=9)
Plan methods and tools for collecting user feedback. (Score=8)		 Test every product iteration on a small pilot group of real customers using real data. (Score=23)
Test UX and employ measurement tools which your platform offers. (Score=13)
Determine your bot personality which fits your brand communication style. (Score=10)		 Monitor chatbot performance and identify which requests were not processed properly. (Score=23)
Study your user needs by collecting feedback and plan new features accordingly. (Score=20)
Monitor chatbot behavioural metrics and success indicators. Ensure that the chatbot achieves its objectives. (Score=20)
Advertise bot capabilities to its users and create respective promotion plan. (Score=15)
Collect qualitative data about chatbot performance to improve functionality. (Score=12)
Table 7. New operational model for chatbot implementation.
Table 7. New operational model for chatbot implementation.

Objective 		Preprints 102603 i001
Preprints 102603 i002
CSFs Preprints 102603 i003
Preprints 102603 i004
Conception ► Definition ► Execution ► Operation ►
Recommended Key Actions Technology Identify and choose the most suitable technology and vendor which meet your business requirements. (Score=26)
Explore vendor options and external service availability in your field and locality. (Score=24)
Choose a platform that can provide detailed reports about your chatbot performance. (Score=19)
Analyse build and buy approaches to define your strategy. (Score=17)
Analyse API of the selected platform and how to integrate it with the company's IS. (Score=10)
Study project portfolio of vendor candidates, request recommendations from their clients. (Score=9)
The selected platform should have a powerful dialog editor. (Score=6)		 Invest in analysis and planning of system integration and data migration tasks. (Score=23)
Define scalability requirements and realistically estimate the required hardware. (Score=13)
Obtain necessary access to IS and solve information security issues. (Score=13)
Strategically plan the compatibility of the bot platform with the company's internal IT systems. (Score=12)
Make sure you know exactly how you can develop the bot on your own or change the integrator company, if necessary. (Score=8)
Formulate comprehensive functional requirements that determine the result. (Score=5)
 Create dashboard for tracking chatbot behavioural metrics and success indicators after the launch. (Score=11)
Perform load testing, plan and prepare infrastructure for high load. (Score=9)
Prepare and use development, testing and production environment. (Score=9)
Use SLAs (service level agreements) to set performance requirements for involved IS. (Score=6)		 Monitor a chatbot availability and usage. (Score=19)
Continuously improve the chatbot by applying recent AI technologies, updating language models, and installing the platform updates. (Score=14)
Organisation Identify your aim, key success indicators and associated improvement metrics. (Score=24)
Identify use cases and assess suitability of using a conversational UI. (Score=19)
Find an expert to join your team. (Score=10)
Prepare the ground for agile working. (Score=9)		 Form a product team on the company's side which includes a product owner, analyst or dialog designer, data scientist, UX-designer, technical leader. (Score=23)
Use the MVP method and an iterative approach. (Score=18)
Find a responsible project leader from the business team who has the necessary expertise. (Score=9)
Involve top management into the process. (Score=6)
Communicate your aim to your team, contractors, and top management. (Score=5)		 Apply an agile iterative approach and MVP method. (Score=23)
Define the chatbot maintenance team, their responsibilities and train the key users. (Score=9)
Test your bot internally with your team, and with business experts in the company. (Score=7)
Pay attention to project control, reporting and communication with the contractor. (Score=4)		 Collect dialog data and invest time and resources in bot training. (Score=13)
Develop the chatbot expertise and grow your team in-house. (Score=9)
Continue to use the agile development approach. Turn it into a routine. (Score=9)
Make sure that the chatbot systems are accepted by IT support technical specialists. (Score=7)
Advertise bot features and share success stories within your company. (Score=5)
User Needs Focus on one business objective. (Score=20)
Analyse alternatives to the chatbot service. Specify why the chatbot is the best way to solve your problem. (Score=13)
Make sure the chatbot is not only good for the company but designed to make the customer’s life better. (Score=10)
Make sure the chatbot is the most convenient way for users to solve their problems. (Score=5)		 Focus on chatbot productivity. The ease, speed, and convenience of using chatbots are vital. (Score=16)
Use CJM and Jobs-To-Be-Done methods to capture user experience and specify chatbot requirements. (Score=16)
Choose the “easy-to-develop” functionality. Publish limited but well-developed functionality. (Score=14)
Design a chatbot personality which fits your brand communication style. (Score=13)
Focus on intuitive and simple UX. (Score=9)
Apply blended communication approach. (Score=9)
Plan methods and tools for collecting user feedback. (Score=8)		 Test every product iteration on a small pilot group of real customers using real data. (Score=23)
Test UX and employ measurement tools which your platform offers. (Score=13)
Determine your bot personality which fits your brand communication style. (Score=10)		 Monitor chatbot performance and identify which requests were not processed properly. (Score=23)
Study your user needs by collecting feedback and plan new features accordingly. (Score=20)
Monitor chatbot behavioural metrics and success indicators. Ensure that the chatbot achieves its objectives. (Score=20)
Advertise bot capabilities to its users and create respective promotion plan. (Score=15)
Collect qualitative data about chatbot performance to improve functionality. (Score=12)
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
Alerts
Prerpints.org logo

Preprints.org is a free preprint server supported by MDPI in Basel, Switzerland.

facebook logotwitter logolinkedin logo
MDPI Initiatives
Important Links
Subscribe

Disclaimer

Terms of Use

Privacy Policy

© 2025 MDPI (Basel, Switzerland) unless otherwise stated