1. Introduction

2. Related Work

2.1. Factor Analysis

When developing a programming environment and evaluate its usability, the validity of the evaluation tools must be ensured. In other words, it is necessary to verify that the tools developed to evaluate usability contain questions that are suitable for evaluation. Validity refers to whether the measurement target can be measured appropriately. Additionally, to evaluate two or more pieces of content, it is important to confirm whether the corresponding constructs can be used to evaluate what they are meant to evaluate. To confirm the validity of a construct, factor analysis can be used, whose purpose is to reveal the covariance structure of the data variables. It is used to create a single construct when one variable changes with another. The process for confirming the validity of the usability evaluation tool and performing factor analysis to create the constructs is as follows:

1)

Create questions, perform a usability analysis, and obtain scores for each evaluation question.

2)

Calculate a matrix of the correlation coefficients between questions.

3)

Extract non-rotated factors.

4)

Rotate the factors.

5)

Interpret and assign names based on the content of questions with high factor loadings related to rotated factors.

Factor rotation is used to obtain a structure wherein the variables of each factor can be clearly interpreted. The goal is to consider factors that are not accurately explained through factor loading, which indicates the degree to which each variable reflects a single factor, and convert them into a simple structure. By rotating to a simple structure, each variable receives a high load from only one factor and relatively low loads from others, simplifying the factor structure. Thereafter, it can be interpreted as a factor structure that is not explained by the initial factor load. The following factor equation can be used to derive factors F₁, F₂, … F_K which include the weight coefficients a₁, a₂, … a_K of multiple variables in the data:

Factor analysis was used to extract constructs for measuring their effectiveness and satisfaction toward education and to ensure their validity. Specifically, 48 evaluation items were developed to determine the relationship between motivation and achievement based on students’ degree of participation in an e-learning environment. Furthermore, six constructs (psychological motivation, peer collaboration, cognitive problem solving, interaction with instructors, community support, and learning management) were extracted to conduct the study [16]. To determine why beginner programmers have low coding skill levels, factor analysis was conducted on the answers to programming problems submitted by 614 university students through a web-based learning system, and four skill-level constructs were extracted: code style, syntactic, logical error-related, and syntax debugging [17]. Factor analysis is a method used to extract underlying constructs from a set of observed variables, categorizing confirmed constructs based on shared variance and extracting content commonly explained by multiple evaluation questions.

Thus, factor analysis ensures the validity of the developed tools and can be utilized to extract factors that commonly explain the questions within an examination tool.

2.2. Regression analysis

Regression analysis is a statistical technique that enables the prediction of values for dependent variables based on the values of independent variables. It accomplishes this by establishing linear equations that represent the relationships between the independent and dependent variables. In other words, it examines the extent to which dependent variables change based on the changes in independent variables, and estimates the predictive power of independent variables with regard to dependent variables. Simple regression analysis assumes linear relationships between independent and dependent variables and is expressed as follows:

In regression analysis, the least squares method is utilized to estimate the intercept (β₀) and regression (β₁) coefficients. These coefficients minimize differences between the actual values of the dependent variable (Y) and the predicted values (Y') obtained using the independent variables. The goal is to find the regression equation that minimizes the overall difference between the observed and predicted values of the dependent variable.

The total change in the Y value is classified into two parts: those that can and cannot be explained by the regression equation.

An analysis of the variance table for simple regression can be explained as follows:

Table 1. Block-based programming environment analysis results.

Table 1. Block-based programming environment analysis results.

	Sum of Squres(SS)	df	Mean Square(MS)	F	Zj
Regression	${SS}_R$	1	${SS}_R/1$	${MS}_R/{MS}_E$	${SS}_R/{SS}_T$
Residual	${SS}_E$	n-2	${SS}_R/n-2$
Total	${SS}_T$	n-1

One criterion for judging the suitability of a regression equation is the determination coefficient(R²). It indicates the explanatory power of an independent variable for a dependent variable and refers to the ratio of the variance, obtained using the regression equation, to the total variance of the dependent variable. The closer the value of the determination coefficient is to 1, the greater the explanatory power of the independent variable.

Simple regression analysis is used for analyzing the predictive power of an independent variable with regard to a dependent variable, whereas multiple regression analysis is a statistical method used to determine the variable that affects the dependent variable among several independent variables. The linear equation for the multiple regression model is as follows:

where ²_K is the unstandardized regression coefficient, i.e., the partial slope of the regression equation. It indicates the change in the Y value when the value of a certain independent variable X_k is increased by 1 while those of the other independent variables are fixed.

There are several methods for selecting the variables that must be included in the regression model to find the optimal regression equation, such as enter, forward selection, backward elimination, and stepwise selection [18]. Stepwise selection determines the optimal regression equation through an appropriate combination of adding and removing independent variables. As the variables are added individually, the significance of the variables already included in the model is reviewed, and those that are insignificant are excluded. This method is helpful for extracting significant variables.

Multiple regression analysis has been used to determine the factors that affect students’ attitudes toward computer programming, which are a sense of achievement during the programming process, self-efficacy with regard to programming, and recognition learning [19]. Additionally, studies have been conducted to predict students’ levels of academic achievement in the early stages [20]. In some cases, path diagrams have been used to visualize the extent to which two or more explanatory variables affect a dependent variable [21]. This is because path diagrams of the effects of three independent variables (X, U₁, and U₂) on a dependent variable (Y) can help users understand the relationships between them and interpret their significance

Figure 1. A path diagram of multiple linear regression.

Figure 1. A path diagram of multiple linear regression.

3. Programming Environments

Figure 2. Programming Activity User Interface(UI).

Figure 2. Programming Activity User Interface(UI).

The third component was the “Programming” area, wherein the learners dragged and dropped commands from the provided command area to complete the program. Unnecessary commands could either be deleted or modified by clicking the Modify button (). The Run () button could be pressed to execute the combined commands and see the results. The Run One Step () button could be used to run a command one line at a time, obtain the results, and view the locations of commands where errors have occurred during the debugging process, which is useful. Clicking the Run or Run One Step buttons showed commands causing errors in yellow color. To initialize with the first command given, the Command Initialization() button must be clicked.

4. Methods

4.2. Programming Course

Additionally, to determine the effects of the proposed block-based programming environment on programming learning, we conducted classes, collected data, performed factor analysis, and then regression analysis, in that order.

Figure 3. Procedure.

Figure 3. Procedure.

4.2.1. Procedure

The classes were held for 2 hours per week for 14 weeks, and the class content included output (print), input, operations (the four arithmetic operations, logical operations, and comparison operations), conditions (IF, ELIF, ELSE), iteration (for, while), lists, Python’s built-in functions, user-defined functions, and individual projects.

4.2.2. Data Collection

Table 2 presents the content of the survey regarding the “usability of programming environment” and “perceptions of programming” answered by the 128 participants. The survey used a 5-point Likert scale, where 5 = "strongly agree” and 1 = "strongly disagree.” The survey was conducted after the classes were finished.

4.2.3. Factor Analysis

To determine the factors of the programming environment that affected beginners’ “perceptions of programming” in this study, factor analysis was conducted through the following steps: data suitability assessment, factor extraction, and factor rotation. SPSS (version 26.0; IBM Corp., Armonk, NY, USA) for Windows was used for factor analysis.

First, to verify the suitability of the factor analysis model, the Kaiser–Meyer–Olkin (KMO) test and Bartlett’s test of sphericity were performed on the survey content “degree of help with programming, ” as presented in Table 3. The KMO value was 0.924, and Bartlett’s test of sphericity significance probability was 0.000. Thus, the factor analysis model was suitable.

Second, the factors were extracted. Principal component analysis and exploratory factor analysis were used in this study. Table 4 presents the total explained variance. There were 23 components before extraction; however, after extraction and rotation, there were 4 components with eigenvalues greater than 1. Apparently, the four extracted factors accounted for 81.06% of the total variance and could be considered to comprise high explanatory power.

Third, a factor rotation was performed. This study used an orthogonal rotation method based on varimax, which uses Kaiser normalization. Table 5 shows the rotated-component matrix. The following Table 5 presents the results of reducing factors using the rotated-component matrix.

The model settings were validated through dimensionality reduction. The following model was used in this study:

Figure 4. Procedure.

Figure 4. Procedure.

The following Table 6 shows the averages and standard deviations for each grade level, according to this proposed model.

4.2.4. Regression Analysis

A multiple regression analysis was performed to determine the effects of the “usability of the programming environment” on “perceptions of programming” at each grade level. Stepwise selection was used to input the independent variables into the analysis, and SPSS for Windows (version 26.0) was used for regression analysis.

5. Results

Table 8. The regression coefficient of the correlation.

6. Discussions

7. Conclusions

References

1. Wong, G. K.; Cheung, H. Y.; Ching, E. C.; Huen, J. M. School perceptions of coding education in K-12: A large scale quantitative study to inform innovative practices. 2015 IEEE international conference on teaching, Assessment, and learning for engineering 2015, 5-10. [CrossRef]
2. Stephens, M. Embedding algorithmic thinking more clearly in the mathematics curriculum. ICME 24 School mathematics curriculum reforms: challenges, changes and opportunities 2018.
3. Intelligence, T. The Global AI Index. Available online: https://www.tortoisemedia.com/intelligence/global-ai/ (accessed on 14 June 2023).
4. Moors, L.; Sheehan, R. Aiding the transition from novice to traditional programming environments. Proceedings of the 2017 Conference on Interaction Design and Children 2017, 509–514. [Google Scholar]
5. Topalli, D.; Cagiltay, N. E. Improving programming skills in engineering education through problem-based game projects with Scratch. Computers & Education 2018, 120, 64–74. [Google Scholar] [CrossRef]
6. Gomes, A.; Mendes, A. J. Learning to Program - Difficulties and Solutions. International Conference on Engineering Education 2007, 283–287. [Google Scholar]
7. Wing, J. M. Computational thinking. Communications of the ACM 2006, 49, 33–35. [Google Scholar] [CrossRef]
8. McCall, D.; Kölling, M. A new look at novice programmer errors. ACM Transactions on Computing Education 2019, 19, 1–30. [Google Scholar] [CrossRef]
9. Xinogalos, S.; Satratzemi, M.; Malliarakis, C. Microworlds, games, animations, mobile apps, puzzle editors and more: What is important for an introductory programming environment? Education and Information Technologies 2017, 22, 145–176. [Google Scholar] [CrossRef]
10. Cárdenas-Cobo, J.; Puris, A.; Novoa-Hernández, P.; Parra-Jiménez, Á.; Moreno-León, J.; Benavides, D. Using scratch to improve learning programming in college students: A positive experience from a non-weird country. Electronics 2021, 10, 1180. [Google Scholar] [CrossRef]
11. Emerson, A.; Rodríguez, F. J.; Mott, B.; Smith, A.; Min, W.; Boyer, K. E.; Lester, J. Predicting Early and Often: Predictive Student Modeling for Block-Based Programming Environments. International Educational Data Mining Society 2019. [Google Scholar]
12. Seraj, M.; Katterfeldt, E. S.; Bub, K.; Autexier, S.; Drechsler, R. Scratch and Google Blockly: How girls' programming skills and attitudes are influenced. Proceedings of the 19th Koli Calling International Conference on Computing Education Research 2019, 1–10. [Google Scholar]
13. Deng, W.; Pi, Z.; Lei, W.; Zhou, Q.; Zhang, W. Pencil Code improves learners' computational thinking and computer learning attitude. Computer Applications in Engineering Education 2020, 28, 90–104. [Google Scholar] [CrossRef]
14. Tavares, P. C.; Henriques, P. R.; Gomes, E. F. A Computer Platform to Increase Motivation in Programming Students-PEP. CSEDU 2017, 1, 284–291. [Google Scholar] [CrossRef]
15. Yong, S. T.; Tiong, K. M. A Blended Learning Approach: Motivation and Difficulties in Learning Programming. International Journal of Information and Communication Technology Education 2022, 18, 1–16. [Google Scholar] [CrossRef]
16. Lee, J.; Song, H. D.; Hong, A. J. Exploring factors, and indicators for measuring students’ sustainable engagement in e-learning. Sustainability 2019, 11, 985. [Google Scholar] [CrossRef]
17. Zhang, Y.; Paquette, L.; Pinto, J. D.; Fan, A. X. Utilizing programming traces to explore and model the dimensions of novices' code-writing skill. Computer Applications in Engineering Education 2023. [Google Scholar] [CrossRef]
18. Paisanwarakiat, R., Na-udom, A., & Rungrattanaubol, J. (2022, April). Combining Logistic Regression Analysis with Data Mining Techniques to Predict Diabetes. Proceedings of the 18th International Conference on Computing and Information Technology 2022, 88-98.
19. Gürer, M. D.; Cetin, I.; Top, E. Factors affecting students' attitudes toward computer programming. Informatics in Education 2019. [Google Scholar] [CrossRef]
20. Alboaneen, D.; Almelihi, M.; Alsubaie, R.; Alghamdi, R.; Alshehri, L.; Alharthi, R. Development of a web-based prediction system for students’ academic performance. Data 2022, 7, 21. [Google Scholar] [CrossRef]
21. Hayes, A. F.; Rockwood, N. J. Regression-based statistical mediation and moderation analysis in clinical research: Observations, recommendations, and implementation. Behaviour research and therapy 2017, 98, 39–57. [Google Scholar] [CrossRef]
22. Hu, Y.; Chen, C. H.; Su, C. Y. Exploring the effectiveness and moderators of block-based visual programming on student learning: A meta-analysis. Journal of Educational Computing Research 2021, 58, 1467–1493. [Google Scholar] [CrossRef]
23. T. Jenkins. On the difficulty of learning to program. Proceedings of the 3rd Annual Conference of the LTSN Centre for Information and Computer Sciences 2002, 53–58. [Google Scholar]
24. Noone, M.; Mooney, A. Visual and textual programming languages: a systematic review of the literature. Journal of Computers in Education 2018, 52, 149–174. [Google Scholar] [CrossRef]
25. Puente, E. L. Effect of the use of block-based languages in programming learning. 2022 International Symposium on Computers in Education 2022, 1–6. [Google Scholar] [CrossRef]
26. Xu, Z.; Ritzhaupt, A. D.; Tian, F.; Umapathy, K. Block-based versus text-based programming environments on novice student learning outcomes: A meta-analysis study. Computer Science Education 2019, 29, 177–204. [Google Scholar] [CrossRef]
27. K-12 Computer Science Framework Steering Committee. K-12 computer science framework. ACM. 2016.
28. Papadakis, S. Evaluating the efficiency of two programming environments in shaping novices' attitudes, perceptions, beliefs and knowledge in programming: a comparison between Scratch and App Inventor. International Journal of Teaching and Case Studies 2019, 10, 31–52. [Google Scholar] [CrossRef]
29. Weintrop, D.; Wilensky, U. Comparing block-based and text-based programming in high school computer science classrooms. ACM Transactions on Computing Education 2017, 18, 1–25. [Google Scholar] [CrossRef]
30. Erol, O.; Çırak, N. S. The effect of a programming tool scratch on the problem-solving skills of middle school students. Education and Information Technologies 2022, 27, 4065–4086. [Google Scholar] [CrossRef]
31. Cheung, J. C.; Ngai, G.; Chan, S. C.; Lau, W. W. Filling the gap in programming instruction: a text-enhanced graphical programming environment for junior high students. ACM SIGCSE Bulletin 2009, 41, 276–280. [Google Scholar] [CrossRef]
32. Mihci, C.; Ozdener Donmez, N. Teaching GUI-Programming Concepts to Prospective K12 ICT Teachers: MIT App Inventor as an Alternative to Text-Based Languages. International Journal of Research in Education and Science 2017, 3, 543–559. [Google Scholar] [CrossRef]
33. Xu, W. W.; Su, C. Y.; Hu, Y.; Chen, C. H. Exploring the effectiveness and moderators of augmented reality on science learning: A meta-analysis. Journal of Science Education and Technology 2022, 31, 621–637. [Google Scholar] [CrossRef]
34. Pérez-Marín, D.; Hijón-Neira, R.; Bacelo, A.; Pizarro, C. Can computational thinking be improved by using a methodology based on metaphors and scratch to teach computer programming to children? Computers in Human Behavior 2020, 105, 105849. [Google Scholar] [CrossRef]
35. Weintrop, D.; Wilensky, U. Comparing block-based and text-based programming in high school computer science classrooms. ACM Transactions on Computing Education 2017, 18, 1–25. [Google Scholar] [CrossRef]
36. Blanchard, J.; Gardner-McCune, C.; Anthony, L. Bridging Educational Programming and Production Languages. Paper for the" Every Child a Coder" workshop, ACM SIGCHI Conference on Interaction Design and Children (IDC 2015), Boston, MA 2015.
37. Powers, K.; Ecott, S.; Hirshfield, L. M. Through the looking glass: teaching CS0 with Alice. Proceedings of the 38th SIGCSE technical symposium on Computer science education 2007, 213–217. [Google Scholar]
38. Alrubaye, H.; Ludi, S.; Mkaouer, M. W. Comparison of block-based and hybrid-based environments in transferring programming skills to text-based environments. arXiv preprint arXiv:1906.03060 2019. arXiv:1906.03060 2019.