1. Introduction

2. Related Work

3. Materials and Methods

3.3. Results: Stage 1

In this section, the responses from the questionnaire are analysed and discussed. Figure 2 provide the numbers of responses for each available response (strongly disagree, disagree, neutral, agree, strongly agree) Figure 3 provides bar charts for each response in relation to the response choices, whilst also showing the response split between different job roles, as provided in Table 2. Furthermore, Figure 4 provides information on how many years of experience the participants have against the responses. The discussion is grouped based on the constructs presented in Table 1, except for biographic which is discussed in Section 3.2.

Figure 2. Response to questions

Figure 2. Response to questions

Figure 3. Responses for each survey by question, illustrating the distribution of answers based on job role

Figure 3. Responses for each survey by question, illustrating the distribution of answers based on job role

Figure 4. Responses for each survey by question, illustrating the distribution of answers based on number of years experience

Figure 4. Responses for each survey by question, illustrating the distribution of answers based on number of years experience

3.3.1. Time

Question 4 asks the participants if they believe an individual’s not having enough time prevents the use of automated testing. The response to the question is well-balanced with only a small majority stating that they agree. As demonstrated in Figure 3b, the distribution of job roles amongst the responses is balanced, with both technical and non-technical roles agreeing and disagreeing. One identified trend is that participants that are strongly disagreeing have identified as performing a technical role, with only 2 of the 10 responses occupying a senior role. This indicates that more junior roles more strongly disagree with the presented statement. It can also be established from Figure 4b that there is an even distribution of years of experience amongst responses.

Question 15 then asked the participants whether the participant believe that automated testing techniques are time-consuming to learn. The responses to this question are also quite evenly distributed. The low percentage of participants strongly agreeing with this statement results in an average between neutral and disagreement. Figure 3m demonstrates the distribution of job roles amongst response categories and it is worth noting that in general non-technical roles are responding more closely with agree and neutral replies, which could perhaps be down to their lack of experience with the technology. Figure 4m demonstrates the years of experience for each response category. For example, all responses from CEOs are in the agreed category. Interestingly, there is an even distribution apart from agree whereby there is the highest quantity of participants with the lowest number of years of experience, which could demonstrate that those with a lower amount of experience could believe that automated testing takes more time to learn, which would most likely originate from the fact that they will have more to learn during earlier years of employment.

3.3.2. Cost

Question 11 asked whether the participant agreed or disagreed that commercial tools are too expensive, thus preventing their use. The majority of the participants agreed with this statement. Figure 3i illustrates the number of responses in relation to each job role. It is observed that the majority of management, senior consultants, and QA and senior QA roles agree with the statement, whereas CEOs are neutral or disagree. Furthermore, Figure 4i where the years of experience are evenly distributed amongst the available responses.

Question 19 asked the participant whether they believe automated test scripts and cases are expensive to generate. The responses to this question are balanced with only a slight emphasis on disagreement. Figure 3q illustrates that the general trend is that it is more likely for managerial roles to agree with this statement. It is also worth noting from Figure 4q that the small number of responses that both strongly agree and disagree have greater than 5 years of experience, whereas the other categories have an even distribution. This could indicate that the views of experienced employees are on average neutral, with a minority having polarised views.

Question 20 asked the participants whether they agree that automated testing requires high maintenance costs. Overall, the participants agreed with this statement, but it is evident that those in non-technical roles are more likely to agree with this statement, which is perhaps to be expected considering their daily interaction with financial operations Figure 3r illustrates that in general non-technical roles are more likely to agree with this statement, which is perhaps to be expected considering their daily interaction with financial operations. There is a slight emphasis on technical staff not agreeing with this statement, which is perhaps down to their lack of involvement with the financial side of their employer’s activities. Furthermore, Figure 4r identifies that those strongly agreeing or disagreeing have a higher number of years of experience.

3.3.3. Tools and techniques

Question 6 asks the participants whether they believe not having the right automation tools and frameworks is preventing use. It is evident that overall the majority of participants do not believe the use of automated testing is prohibited by the inability to identify and use the correct tools. Interestingly, Figure 4d illustrates that participants that are strongly agreeing have 10 to 15 and 30 to 35 years of experience. However, overall there is an even distribution of years of experience among the responses.

Question 16 asked participants whether they believed that automated testing tools and techniques lack the necessary functionality. Overall, the majority of participants disagree with this statement. Figure 4n demonstrates that there is a slight increase in the portion of responses from participants with an increased number of years of experience in the agree and strongly agree categories. This could perhaps indicate that more experienced employees have a stronger belief that current techniques and tools lack functionality, which could be down to the fact that they have in-depth experience and knowledge of missing functionality.

Question 17 asked the participants whether they believe that automated testing tools and techniques are not reliable enough, making them unsuitable for use. The responses overwhelmingly disagreed with this statement. Figure 3o illustrates that there is a diverse distribution of job roles amongst each response category. It is worth noting that only one participant strongly agreed and they are performing a technical role, which could indicate that their dissatisfaction originates from working closely with automated testing tools and techniques. Furthermore, as evident in Figure 4o, there is no relationship between years of experience and response, apart from the observation that there is a higher proportion of participants with a lower number of years of experience either agreeing or strongly agreeing. This could indicate that they have not yet mastered their craft and utilised the full potential of automated tools, or even their dissatisfaction with their chosen career.

Question 18 asked the participants whether they agree that automated testing lacks support for testing non-functional requirements (usability, safety, security.) The responses to this statement are close, but the majority are in agreement with this statement. Figure 3p and Figure 4p presents that there is an even distribution amongst roles and year’s experience within the response categories.

Question 21 asked the participants whether or not they agree that automated testing tools and techniques change too often, introducing problems that need fixing. A majority of the responses agree with this statement. Figure 3s illustrates that non-technical employees are more likely to agree with the statement, with only management roles submitting as strong accept. Interestingly, it also illustrates that QA and senior QA roles only responded as agreed. The majority of technical testing, engineering, and automation roles are either neutral or disagree, and they are also the only roles to strongly disagree. Furthermore, Figure 4s also displays that in general, the participants with a higher number of years of experience are more likely to respond with a neutral or disagreeing response. This indicates a different point-of-view between non-technical and technical roles, as well as the number of years of experience that an individual has. Experienced individuals may have gained sufficient expertise in how to maintain their scripts and keep them updated with new versions of testing tools.

3.3.4. Utilisation

Question 5 asks the participants whether they believe that difficulties in preparing test data and environments are responsible for preventing the use of automated testing. Overall, the majority of the responses agree with this statement. Figure 3c presents the breakdown of responses versus job roles. Interestingly, the results demonstrate that non-technical roles (CEO, Consultant, Management) are mostly agreeing with this statement and there is only 1 response from a Manager that disagrees.

Question 7 asked the participant whether they agreed with the statement that difficulties in integrating different tools/frameworks together are preventing their use. A small majority were in favour of disagreeing with the statement. Figure 3e illustrates that the majority of non-technical roles agree with this statement and the balance based on technical roles is almost even, with a slight emphasis on disagreeing with the statement. Figure 4e demonstrates an even distribution of years of experience among the responses, although the responses with a greater number of years of experience are on balance more in agreement than disagreement.

Question 8 asks the participant whether they agree or not with the statement that frequent requirement changes are often preventing the use of automated testing. The provided responses are overall in agreement with the statement. In Figure 3f it is evident that the job role distribution is mostly even with the majority of respondents operating in Software Tester, Engineering, Analysts and Test Architects, whereas respondents with Quality Assurance roles are majority agreeing. Interestingly non-technical positions, such as CEOs, are either neutral or disagree with this statement, which could indicate a misalignment between both non-technical and technical employee experiences with automated testing when it comes to the impact of changing software requirements.

Question 12 asked the participant whether they think that open-source automation tools are difficult to use and the majority of participants disagree. As illustrated in Figure 3j, the number of non-senior and technical roles is low for both agree and strongly agree. In relation to years of experience, Figure 4j illustrates that a higher number of individuals with a lower number of years of experience disagree with the statement, which could be down to the fact that those with fewer years of experience received dedicated training on the tools that they are using, i.e, they might be recent graduated having been training specifically on the used technology.

Question 22 is the final question and asked the participants whether they agree that it is difficult to reuse test scripts and data across different stages of testing. The responses in general align with disagreeing with this statement. The lower number of neutral responses indicates polarised views on this statement. From analysing the different roles that are presented in Figure 3t, it is evident that non-technical employees are more likely to agree with the statement; however, this is a weak correlation as some non-technical staff do disagree. Furthermore, technical staff are distributed across all categories. However, only those undertaking QA and technical roles strongly disagree. Figure 4t illustrates that of users who strongly agree, they all have a high number of years of experience. The different categories of experience and then evenly distributed among the different response categories, apart from those that strongly disagree that have between 5 and 10 years of experience only.

3.3.5. Organisation and Capabilities

Question 3 asked the participants whether they agreed with the statement that the lack of skilled resources is preventing automated testing from being adopted within an organisation. This demonstrates that the majority of the responses agree that a lack of skilled resources is a problem. Furthermore, as demonstrated in Figure 3a, the majority of people agreeing with this statement are performing non-technical roles, whereas the majority of the people disagreeing are performing more technical roles. This is of significance as it highlights the different viewpoints when considering whether there is a resourcing issue. In addition, Figure 4a highlights that the majority of responses provided by participants with in excess of 20 years of experience either agree or strongly agree. However, it is also worth noting that participants that strongly disagree are only in the 15-20 years of experience category.

Question 9 asked whether people believed that automated testing is often not used due to people not realising and understanding the potential benefits. The overall trend is that a majority disagree with this statement.

Question 10 asked the participant whether they believed a lack of support from senior management is preventing their use. The results from this question are well-balanced with the number of participants agreeing with the statement being slightly higher than those disagreeing. In Figure 3h, it is evident that there is an even distribution of job roles amongst the responses; however, the role of consultant only appears in the neutral, agree, and strongly agree responses, whereas managers and CEOs are on average disagreeing with the statement. The difference with consultants could be due to the fact that they are not directly employed by an organisation and provide independent observation.

Question 13 asked the participant whether they agree or disagree with the statement that test automation requires a high level of expertise, which is often not available. Overall the trend is that the respondents disagree with the majority. Figure 3k illustrates how different roles selected their answers, It is evident that there is an even distribution of roles. It is therefore a fair assumption to state that only those with a good technical understanding disagree with the statement. Figure 4k illustrates that the number of years of experience within each reply category is well distributed; however, strongly disagree has the highest average years of experience when compared to the other categories.

Question 14 asked whether the participants believe that automated testing requires strong programming skills. The responses overall strongly agreed with this statement. Figure 3l demonstrates that there is an even distribution of roles providing responses within each response category, and Figure 4l illustrates that there is an even distribution of years of experience within each response category.

3.4. Results: Stage 2

In this stage, Principle Component Analysis (PCA) is performed using SPSS (version 24). PCA is a statistical analysis technique that uses linear algebra techniques (specifically orthogonal transformation) to convert a data set believed to contain correlations into subsets of correlated data, known as principal components. In this process, the twenty items (questions) were used and these comprised the final attitude scale. In performing PCA, the level of variance (known as the Cronbach’s alpha) is calculated and is used as a measure of how suitable the data is for identifying principal components. Nunnally and Bernstein [39] state that 0.70 is an acceptable minimum for a scale that is newly developed. In our results, reliability for these 20 items of the sample produced a Cronbach’s alpha of 0.86 and is presented in Table 4. It is important to note that the alpha coefficient was not increased by eliminating items. Ferketich [40] recommended that corrected item-total correlations should range between 0.30 and 0.70 for a good scale [41]. In our result, all 20 questions had significant item-total correlations and were retained (ranging from 0.24 to 0.60).

Table 3 presents the results using the Kaiser-Meyer-Olkin (KMO) and Bartlett’s measure the sampling adequacy. As evident in the results, the KMO value is very close to 0.8, which is often determined as the determining that the sample is adequate. A value of below 0.6 determines that the sampling would be inadequate but anything above 0.7 is acceptable [42,43]. Furthermore, Barlett’s test of sphericity is used to measure the correlation between the questions within a theme. In our results, we obtain a value of 553.333 and the further the value is from 0, the higher the correlation between two variables [44].

Table 3. KMO and Bartlett’s Test.

Table 3. KMO and Bartlett’s Test.

Kaiser-Meyer-Olkin Measure of Sampling Adequacy	.778
Bartlett’s Test of Sphericity Approx. Chi-Square	553.333
Significance	.000

Table 4. Reliability Statistics.

Table 4. Reliability Statistics.

Cronbach’s Alpha	Number of Items
.860	20

AtAT with oblique (nonorthogonal) rotation was used to investigate the components that stop people from adopting automated testing. Analysis of the scree plot and eigenvalues led to the extraction of two components, which together accounted for 39% of variance in the data (see Table 5). We termed component one non-software factors, which comprises items relating to finance, expertise, and time. The second component we termed software factors, which comprises of ten items loading on this component related to the effectiveness, efficiency, completeness, and adaptability.

Our analyses of the twenty items reveal a two-component structure (Table 5). The non-software component consists of ten items explaining 29% of the variance and yields an eigenvalue of 5.8. Eigenvalue values are a measure of a component’s magnitude. The non-software factors component is highly correlated with a high internal consistency (Cronbach’s alpha=.813). The software factors consists of ten items explaining 10% of the variance and yields an eigenvalue of 2.0. The software factor also highly correlates (Cronbach’s alpha=.790). The results present the commonality scores, indicating how well each item fits the components. Furthermore, all existing theoretical guarantees for PCA assume that the data and the corrupting (all items grouped together into one theme) components are mutually independent (uncorrelated).

Table 6 presents the average percentage response grouped by role type and also by identified components from the principle component factor analysis. It is evident that based on the identified factor, participants undertaking a technical role more strongly agree that non-software reasons are preventing their use and they more strongly disagree that software reasons are preventing their use. It is also evident that they more strongly agree with the non-software factor being responsible for not adopting automated testing. Participants undertaking a non-technical role are more strongly agreeing that both non-software and software factors are preventing the use of automated testing.

4. Discussion and Findings

5. Conclusion

References

1. Charette, R.N. Why software fails [software failure]. IEEE spectrum 2005, 42, 42–49. [Google Scholar] [CrossRef]
2. Ammann, P.; Offutt, J. Introduction to software testing; Cambridge University Press, 2016.
3. Dustin, E.; Rashka, J.; Paul, J. Automated software testing: introduction, management, and performance; Addison-Wesley Professional, 1999.
4. Elghondakly, R.; Moussa, S.; Badr, N. Waterfall and agile requirements-based model for automated test cases generation. 2015 IEEE Seventh International Conferenc e on Intelligent Computing and Information Systems (ICICIS). IEEE, 2015, pp. 607–612.
5. Rafi, D.M.; Moses, K.R.K.; Petersen, K.; Mäntylä, M.V. Benefits and limitations of automated software testing: Systematic literature review and practitioner survey. 2012 7th International Workshop on Automation of Software Test (AST). IEEE, 2012, pp. 36–42.
6. Asfaw, D. Benefits of Automated Testing Over Manual Testing. International Journal of Innovative Research in Information Security 2015, 2, 5–13. [Google Scholar]
7. Taipale, O.; Kasurinen, J.; Karhu, K.; Smolander, K. Trade-off between automated and manual software testing. International Journal of System Assurance Engineering and Management 2011, 2, 114–125. [Google Scholar] [CrossRef]
8. Vos, T.E.; Marin, B.; Escalona, M.J.; Marchetto, A. A methodological framework for evaluating software testing techniques and tools. 2012 12th international conference on quality software. IEEE, 2012, pp. 230–239.
9. Eldh, S.; Hansson, H.; Punnekkat, S.; Pettersson, A.; Sundmark, D. A framework for comparing efficiency, effectiveness and applicability of software testing techniques. Testing: Academic & Industrial Conference-Practice And Research Techniques (TAIC PART’06). IEEE, 2006, pp. 159–170.
10. 2019.
11. Kumar, D.; Mishra, K. The Impacts of Test Automation on Software’s Cost, Quality and Time to Market. Procedia Computer Science 2016, 79, 8–15. [Google Scholar] [CrossRef]
12. Mittal, V.; Garg, N. Test Automation using Selenium Webdriver 3.0 with C 2018.
13. Vogel-Heuser, B.; Fay, A.; Schaefer, I.; Tichy, M. Evolution of software in automated production systems: Challenges and research directions. Journal of Systems and Software 2015, 110, 54–84. [Google Scholar] [CrossRef]
14. Zhou, Z.Q.; Sinaga, A.; Susilo, W.; Zhao, L.; Cai, K.Y. A cost-effective software testing strategy employing online feedback information. Information Sciences 2018, 422, 318–335. [Google Scholar] [CrossRef]
15. Panichella, S.; Di Sorbo, A.; Guzman, E.; Visaggio, C.A.; Canfora, G.; Gall, H.C. How can i improve my app? classifying user reviews for software maintenance and evolution. 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2015, pp. 281–290.
16. Tracey, N.; Clark, J.; Mander, K.; McDermid, J. An automated framework for structural test-data generation. Proceedings 13th IEEE International Conference on Automated Software Engineering (Cat. No. 98EX239). IEEE, 1998, pp. 285–288.
17. Fewster, M.; Graham, D. Software test automation: effective use of test execution tools; ACM Press/Addison-Wesley Publishing Co., 1999.
18. Graham, D.; Fewster, M. Experiences of test automation: case studies of software test automation; Addison-Wesley Professional, 2012.
19. Böhme, M.; Paul, S. A probabilistic analysis of the efficiency of automated software testing. IEEE Transactions on Software Engineering 2015, 42, 345–360. [Google Scholar] [CrossRef]
20. Rahman, A.A.; Hasim, N. Defect Management Life Cycle Process for Software Quality Improvement. 2015 3rd International Conference on Artificial Intelligence, Modelling and Simulation (AIMS), 2015, pp. 241–244. [CrossRef]
21. Garrett, T. Useful Automated Software Testing Metrics. Software Testing Geek 2011.
22. Rex, B. Managing the Testing Process: Practical Tools and Techniques for Managing Hardware and Software Testing, 2002.
23. Berner, S.; Weber, R.; Keller, R.K. Observations and lessons learned from automated testing. Proceedings of the 27th international conference on Software engineering. ACM, 2005, pp. 571–579.
24. Jansing, D.; Novillo, J.; Cavallo, R.; Spetka, S. ; others. Enhancing the Effectiveness of Software Test Automation. PhD thesis, 2015.
25. Dustin, E.; Garrett, T.; Gauf, B. Implementing automated software testing: How to save time and lower costs while raising quality; Pearson Education, 2009.
26. Melton, J.R. The Hidden Benefits of automated Testing. Proceedings of the 2015 Aerospace Testing Seniar. CVENTS, 2015.
27. Kasurinen, J.; Taipale, O.; Smolander, K. Software test automation in practice: empirical observations. Advances in Software Engineering 2010, 2010. [Google Scholar] [CrossRef]
28. Leitner, A.; Ciupa, I.; Meyer, B.; Howard, M. Reconciling manual and automated testing: The autotest experience. 2007 40th Annual Hawaii International Conference on System Sciences (HICSS’07). IEEE, 2007, pp. 261a–261a.
29. Rafi, D.M.; Moses, K.R.K.; Petersen, K.; Mäntylä, M.V. Benefits and limitations of automated software testing: Systematic literature review and practitioner survey. Proceedings of the 7th International Workshop on Automation of Software Test. IEEE Press, 2012, pp. 36–42.
30. Monier, M.; El-mahdy, M.M. Evaluation of automated web testing tools. International Journal of Computer Applications Technology and Research 2015, 4, 405–408. [Google Scholar] [CrossRef]
31. Garousi, V.; Felderer, M. Worlds Apart: Industrial and Academic Focus Areas in Software Testing. IEEE Software 2017, 34, 38–45. [Google Scholar] [CrossRef]
32. Zou, W.; Lo, D.; Chen, Z.; Xia, X.; Feng, Y.; Xu, B. How practitioners perceive automated bug report management techniques. IEEE Transactions on Software Engineering 2018. [Google Scholar] [CrossRef]
33. Lo, D.; Nagappan, N.; Zimmermann, T. How practitioners perceive the relevance of software engineering research. Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, 2015, pp. 415–425.
34. Meyer, A.N.; Fritz, T.; Murphy, G.C.; Zimmermann, T. Software developers’ perceptions of productivity. Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2014, pp. 19–29.
35. Abdi, H.; Williams, L.J. Principal component analysis. Wiley interdisciplinary reviews: computational statistics 2010, 2, 433–459. [Google Scholar] [CrossRef]
36. Jolliffe, I.T.; Cadima, J. Principal component analysis: a review and recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 2016, 374, 20150202. [Google Scholar] [CrossRef] [PubMed]
37. Boone, H.N.; Boone, D.A. Analyzing likert data. Journal of extension 2012, 50, 1–5. [Google Scholar] [CrossRef]
38. Faraj, S.; Sproull, L. Coordinating expertise in software development teams. Management science 2000, 46, 1554–1568. [Google Scholar] [CrossRef]
39. Nunnally, J.C.; Bernstein, I.H.; Berge, J.M.t. Psychometric theory; Vol. 226, McGraw-hill New York, 1967.
40. Ferketich, S. Focus on psychometrics. Aspects of item analysis. Research in nursing & health 1991, 14, 165–168. [Google Scholar]
41. Cortina, J.M. What is coefficient alpha? An examination of theory and applications. Journal of applied psychology 1993, 78, 98. [Google Scholar] [CrossRef]
42. Ferguson, E.; Cox, T. Exploratory factor analysis: A users’ guide. International journal of selection and assessment 1993, 1, 84–94. [Google Scholar] [CrossRef]
43. Nunnally, J.C. Psychometric theory 3E; Tata McGraw-hill education, 1994.
44. Tobias, S.; Carlson, J.E. Brief report: Bartlett’s test of sphericity and chance findings in factor analysis. Multivariate behavioral research 1969, 4, 375–377. [Google Scholar] [CrossRef] [PubMed]
45. Tabachnick, B.G.; Fidell, L.S.; Ullman, J.B. Using multivariate statistics; Vol. 5, Pearson Boston, MA, 2007.
46. Warren, C.R.; Lumsden, C.; O’Dowd, S.; Birnie, R.V. ‘Green on green’: public perceptions of wind power in Scotland and Ireland. Journal of environmental planning and management 2005, 48, 853–875. [Google Scholar] [CrossRef]