In this section, we describe the experimental setup, results, and evaluation.
4.3. Results
As a result of the research, we were able to achieve several deliverables. There were four successful grouping of models derived from implementing this research’s approach. Visual Model Group 1 generated visual models of relevant data under uncorrelated conditions such as histograms and scatter plots. Visual Model Group 2 generated visual models of relevant data under correlated conditions such as multi-year data and bar charts. Visual Model Group 3 generated visual models of relevant data under predicted conditions of next event such as line graphs with and without the estimated predictions. Visual Model Group 4 generated visual models of relevant data under assumed impact of implemented approach such as web deployment tools and the multifunction choropleth maps.
After compiling the data, a typical scenario was run through of how a school may approach this usage if they were to use data-driven decision making. In this dataset, the results from grade level three participants across all the school districts from the state of Texas and Louisiana were used. Then we dove into how that representation looked for district performance, used that to run comparisons at certain proficiency levels, and then tracked the data across time and implemented predictions based upon the data. This allowed for further analysis to be ran on comparison of an individual district’s performance in reference to statewide results. While implementing this research, we wanted to focus on valid use cases for this paper that would be relevant to demonstrate our concept. To that end, we introduce the following storyboard:
“At any given time, a school district will make attempts to evaluate their institutional effectiveness. At times, that can include evaluating themselves against other districts; either in their surrounding area that may be competitors for enrollment or across a geographic area that has similar distributions of demographics within their population(s) served.”
These evaluations not only are analyzed at an organizational level as a whole (district) but also at an institutional level (school). The prevalent analyses often delve further by disaggregating by demographics or special populations. Priorities of doing so are usually to attain insight on whether or not there is improvement in a desired focus area such as enrollment or assessment outcomes. Statewide results are more relevant for an institution when population size is a parameter. It offers the benefit of having the ability to see how many institutions for any comparison made is in the total comparison group (see
Figure 13). Comparing results across groups with like populations is typical in assessing overall academic performance in institutions.
One of the measures for evaluating baselines or successful learning organizations is the ability to review and disaggregate performance in proficiency of state mandated assessment results. As these assessments measure learning of students in content areas, schools use them in decision making in curriculum and content for instruction. Furthermore, the data allows them to review the success of the instructor’s ability to deliver content effectively in order for students to demonstrate satisfactory progress.
Being able to accurately use data to derive insights becomes the most important outcome at the organizational level. At the stage in which the data is the most effective, it also requires quick turn around and accuracy while being delivered in the most efficient manner across multiple audience types. Visualizing the data serves to meet this need as we can quickly view results from any distribution or measured standpoint. It is the creation of images, diagrams, or animations of data for visual representations in order to provide information. Visualization helps to quantify results in the most meaningful and appropriate focus area. Effective, high-impact, practical evidence that can be received in a timely manner is the best way to measure cause and effect of learner outcomes and the relationships between organizational objectives, students and teachers. Providing this through visualization serves to make the information digestible and interpretable while also connecting the material to the audience. Take a look at how simple using visualization to showcase statistical measures or data trends can be implemented in the examples below.
In
Figure 14, we are showcasing the how state assessment results are distributed across all districts in Texas against agencies in Louisiana. The scatter plot shows two performance measures and where our test district lies within the distribution of those performances. This distribution shows the distribution of the overall school population percentage of students tested in grade three reading for the state’s assessment for each performance measure. In Texas the minimum passing standard is the Approaches performance standard (orange) while in Louisiana the minimum standard is Basic (green). Texas Approaches standard does not assume the student is proficient at their grade level. The Texas Meets performance standard (blue) correlates to proficiency at grade level. By comparing these categories with each other, we can understand the difference between assessments and student performances between the two states. The comparisons are often used to determine where an institution falls in the range of performances as seen by the example district’s performance ratings (purple-Approaches, yellow-Meets).
For the case of this paper’s scenario, we use the results for Alief Independent School District in Houston, Texas as our district of interest. In
Figure 15, the district’s percent of students receiving the differing designations across the district for the state assessment in reading for third grade is shown. With these types of visualizations, we can view how certain areas of interest are distributed within the organization of interest as well as make summative comparisons against similar organizations.
In addition to measuring or showing data distributions, often education agencies make plans based on “what-if” scenarios. These scenarios give a sample guide of how situations are addressed or affect an outcome. It is a basic analytic tool that is used quite frequently in education for decision making. They can be further thought of as specific types of trends or projection analysis which can also be modeled through machine learning.
Machine learning is a field of artificial intelligence that allows learning of systems, computers, or other devices through experience with data or algorithms. We can use machine learning to project situational outcomes given the various data inputs or-in other words-to make predictions and/or decisions. Machine learning minimizes the error/biases sometimes created, by allowing the data define rules for predicting the most probable outcome and then modeling that result across different domains. Using different types of machine learning algorithms will have differing effects on the strengths of the relationships in your model. Machine learning practitioners also can make use of correction algorithms combined with their predictor algorithms to increase the effectiveness and accuracy. When including machine learning as a component to supporting data-driven objectives in education, there is a noticeable difference in turn-around time on providing results that are actionable and valuable. We make use of two machine learning algorithms. Although there are other applicable types such as k-means clustering or SVM algorithms, in this paper we are using linear regression followed by logistic regression. This research focuses on simple regression using single variable analysis. It is possible to use multivariate regression techniques but to prove that regression is a viable solution, we use single variable analysis to demonstrate.
Linear regression is a modeling technique that approaches prediction by drawing a relationship from on variable based upon the value of another. What happens is that the data is modeled in such a way that a line approximating the relationship between the variables is used then used as the estimating function for determining a new value if given some test input. Similar to how we estimate the next point on a line by using the slope-intercept formula in mathematics.
As shown in
Figure 16, the variable we focused on for regression analysis is the performance metric for grade level standard. It is the “Meets” category which defines how the student performed on an assessment content area. The value for each student’s assessment score is placed into this category using true/false indicators to denote whether the student has fulfilled the requirements of the exam to demonstrate proficiency in the content on the grade level assessed for that content. As such, not only does the student “pass” the exam, but it is also at the grade level of proficiency as indicated in the subject assessed by the exam.
Previously mentioned, the content and grade level chosen to demonstrate the concept of using MLAs is third grade reading. The dataset given imports all of the STAAR results at the district level for all of the school districts in the state of Texas. This publicly available data found on the Texas Education Agency (TEA) website is comprised of third grade reading performance results from assessment years (where existing) 2017-2022. From there, we use this single variable under the “Meets” category and, following the rules for implementing linear regression, fit a line to the data in order to predict the estimated percentage expected to receive for the example district in 2023. For our test district we used the linear regression package built in the scikit learn library to create our regression model. At the time of the initial version of this report, the expected value for our example district using linear regression was 34 as shown in
Figure 17.
This was of course, assuming no other variables are associated and will have a large enough effect necessitating the need to include them in the regression model. The results schools sometimes calculate themselves tend to be within +/- one percentage point of the official published data usually published sometime later, after the schools are using their data to make administrative decisions. Being able to implement this type of predictive analysis much quicker and to some degree of accuracy allows for better response time for districts to improve their operations* (*note: The official published data for our sample district noted the results to be at 35%, which is within the accepted margin of +/- one percentage point districts accept, proving validity of usefulness of MLAs to enhance decision making).
Figure 18.
Logistic Regression Modeling for Education Assessment Data.
Figure 18.
Logistic Regression Modeling for Education Assessment Data.
Similarly, with linear regression, we used the same variable and conditions to run our test case under logistic regression. In the paper only one variable is used and following the procedure for logic regression, received an estimate for the predicted value. As with linear regression, the values also assumed no other variable was influencing the result in order to show possibility of regression analysis. Using logistic regression, the estimated next score prediction was computed to be 41%, which can be seen in the figure below.
Figure 19.
Logistic Regression Predictions next score predictions.
Figure 19.
Logistic Regression Predictions next score predictions.
The robustness of the experiment is replicable at multiple levels. Often as previously mentioned, districts may want to do comparisons at a demographic or population group level. Similar with district data, states, districts and even schools have the ability to compare across different subpopulations. To date, the most popular comparisons are usually by racial groups, gender, language and economic groups. Following the same procedure as with our other visualizations and regression techniques, you can derive insights based on your criteria of interest for datasets like the one retrieved from TEA website shown below.
Although in our example we use Python for our practical implementation of the concept, there are many other tools that serve this purpose. And with the progressive movement to dashboarding education is expected to slowly join in the collective of stakeholders buying into them. Other means include using R, Tableau, Power BI, Julia and more that vendors and companies are taking advantage of to customize solutions to sell and deploy products. Each method has its own considerations for usage in the education space. As long as the infrastructure and resources are clearly understood, you can effectively duplicate, streamline, improve or further develop your own avenues for showcasing deliverables within the data-driven space.
One of the driving forces for this paper is the availability to make something of this nature web-deployable. To make this research a fully functional product that education institutions and other stakeholders would find useful, we focused on using tools that they would be able to acquire. It should be re-mentioned that in addition to the data being publicly available and therefore no-cost, the other tools that were used to build this solution were no-cost as well; helping to limit the cost to build and can be replicated by those that would invest time and labor into production. With agencies limiting the funding to education institutions in the U.S., this becomes increasingly important to consider in their budgets. The data can be easily retrieved from the state website. After using python to preprocess the data, we created our visualizations using Jupyter as the IDE to see results.
Figure 20.
Special Population Groups Dataset by Performance Level from TEA.
Figure 20.
Special Population Groups Dataset by Performance Level from TEA.
To showcase the scenarios that use comparisons across locales, we made use of Python libraries to incorporate map and mapping functions with geolocations.
Figure 21.
Districts across Texas-Interactive Choropleth.
Figure 21.
Districts across Texas-Interactive Choropleth.
This not only served as a way of providing the ability to compare data across locales, but also interact with the maps to perform useful functions such as searches, radius marking, and multilevel information relay. We can hide features that would be important to have at the ready for users on rollover or click function and embed those qualities into the map so that each location has their own specific detailing according to the information available (see
Figure 22). The web-deployment criteria were successfully implemented using StreamLit, a powerful tool allowing for results to web browser that is typically the preferred method of information display. StreamLit allowed the flexibility to take any visualizations, documents, videos and more to build an interactive site. We tested this by creating in Python html document ready pages for our visualizations and linking them as subpages. Some of the data was output as tables that were directly read from our data files into StreamLit. Even the interactive maps were able to find their home on the site page with full functionality intact.
Figure 22.
Example of Optional Features for Maps Using Folium.
Figure 22.
Example of Optional Features for Maps Using Folium.
Since our approach required the use of command line code to run StreamLit, it also provided an excellent method for packaging as a runnable software package. By creating a simple executable or batch file to run the code(s) required, you have an easy and simple way to deploy this solution as an in-house product for an institution.
Figure 23.
Deploying Creations to Functional Websites Using StreamLit.
Figure 23.
Deploying Creations to Functional Websites Using StreamLit.
The solution can be housed on any server and deployed from any station or imaged into an institution’s endpoint configuration. Python, RStudio, Dash and other tools have the ability to interact and deploy web solutions proving that this is not only useful, but sustainable in the education space. There are numerous avenues this product can live in an environment and what makes it flexible is that it is also able to be delivered through systems such as Altiris, SCCM or ServiceNow, if group policy management determining device management were of consideration. Further improvements would allow the use of automation through Selenium to incorporate action-based scripting. With the completion of this research and paper, this could be used for presentation of modernizing decision making in education for multiple levels of education. While this research was implemented on K-12 data, it also has the versatility to be used in higher-ed. As in the case with other programs such as “My Brother’s Keeper” we could also integrate multiple sources of data to increase the types of information presented to stakeholders the more granular the request is.
One example is local disciplinary data or epidemiological records that would then showcase education results and its level of relationship between them. Since there are very little programs available providing these types of complete and comprehensive visualizations, especially those that are useful as case management tools, if worked to its fullest potential we could actually derive cross-functional insights from several correlational studies.