1. Introduction

2. Literature Review

3. The Data & the Analytic Plan

3.1. About the Data

The data includes parameters such as air temperature, wind speed, wind direction, relative humidity, barometric pressure, Direct Normal Irradiance (DNI), Diffuse Horizontal Irradiance (DHI), and Global Horizontal Irradiance (GHI). The dataset schema is as follows:

Table 1. Data Schema:Attributes of the data.

Table 1. Data Schema:Attributes of the data.

S No	Column Attribute	Data Type
1	Site: Location identifier	Categorical
2	Latitude: Geographical latitude of the site	Numeric
3	Longitude: Geographical longitude of the site	Numeric
4	Date: Timestamp of the recorded data	Numeric
5	Air Temperature (${}^{\circ }$C): Ambient temperature in degrees Celsius	Numeric
6	Wind Direction at 3m (°N): Wind direction at 3 meters height in degrees North	Numeric
7	Wind Speed at 3m (m/s): Wind speed at 3 meters height in meters per second	Numeric
8	Relative Humidity (%): Relative humidity percentage	Numeric
9	Barometric Pressure (mB): Atmospheric pressure in millibars	Numeric
10	DHI (Wh/m²): Diffuse Horizontal Irradiance in watt-hours per square meter	Numeric
11	DNI (Wh/m²): Direct Normal Irradiance in watt-hours per square meter	Numeric
12	GHI (Wh/m²): Global Horizontal Irradiance in watt-hours per square meter	Numeric

Given the data and its schema, the following visualizations and analytical insights are planned. Time Series Plots, Correlation heat maps, Box Plots, Bar charts, Line plots, predictive models, Classification, regression analysis, clustering, anomaly detection, optimization, and impact analysis.

3.2. The methodology

Data ingestion is the collection and importation of data from the Open Data URL. There are multiple data sets each one for a specific geographic location and the collection center. These data sets have the same schema. The schema details are discussed in section 2 above. Each of these files is extracted into one folder/directory. This completes data ingestion. A web scraper with the given URL extracts all the open data files and dumps them into one directory.

Figure 2. Data Analytic Pipeline:The methodology

Figure 2. Data Analytic Pipeline:The methodology

Data cleaning involves the following targeted tasks:

• Removing the top 10 lines of each file
• Then promoting the first line as column header
• Reading the first file and appending all other files to it to form one Combined single CSV file.
• Removing duplicates from this combined file
• Date formatting from “d/m/Y H:M:S” to “d-m-Y” form.
• Missing value substitution by the strategy of the mean of the column.

A data preprocessing script automatically performs these tasks. Finally, manual inspection is carried out to ensure the data is clean.

Data transformation involves, Column removal, row removal, (if any), and defining new measure and quick measure to calculate correlation coefficients. In the prediction event, data transformation involves the addition of blank new columns to be filled by the prediction algorithm with the predicted values.

Data storage is saving the cleaned and transformed data in a directory or database or a cloud location so that it can be accessed by the application for analysis. In this research, the data is stored locally by creating a dynamic path to access it. The dynamic path creation helps to access the files even if the file paths are changed. As on the application testing date, the size of the case study data did not exceed 2MB. Any update to the source files will be dynamically reflected in the visualization dashboard and analytical outputs.

Data visualization involves the initial inspection of the data through relational graphs such as line graphs, bar charts, pie charts, doughnut charts, and scatter plots. Plots drawn for Air temperature, wind speed, barometric pressure, and relative humidity against the date and the radiation data parameters provide interesting glimpses for further inquiry.

3.3. Early Data Visualizations

In this section some of the initial data visualizations are collected to comprehend the data and its trends. In this regard the following visualizations are shown below:  

Time Series & Scatter Plots: Plots of GHI, DHI, and DNI values over time. Figure 3 (a). The plot shows lowest temperature in 2020.

Scatter Plot: A scatter plot of DHI vs Air Temperature shows the linear rise of DHI as the air temperature increases Figure 3 (b).

Figure 3. Time series (a) and Scatter (b) plots

Figure 3. Time series (a) and Scatter (b) plots

Bar Charts: Bar chart showing average GHI for different sites to identify the most productive locations. Monthly Average GHI: Bar chart to compare average GHI values across different months.

Figure 4. GHI over different sites, over the years

Figure 4. GHI over different sites, over the years

Line Plots: Line plot to compare DHI, DNI, and GHI values over time.

Figure 5. Variations of meterological parameters, over the years

Figure 5. Variations of meterological parameters, over the years

The machine learning techniques are then put into practice by writing a Python script. The script provides an integrated data analytic component that is first tested independently on a Jupyter notebook before being integrated into the application architecture. The system is expanded and the visuals and analytical results are reintegrated into visualizations.

At this point, the model is assessed using the case study data set, and metrics like mean average errors, $R^2$ values, confusion matrix, and prediction accuracy. The findings of the empirical tests are then presented in PDF reports and a visual dashboard.

4. The Model Construction

5. The Analytic Dashboard

6. Results & Discussion

6.3. Predictive Analytics

In this section the prediction results of the random forest regressor and its evaluation metrics are presented. Figure 13 shows the actual GHI values vs Predicted GHI values. This plot shows the meteorological coefficients, the regressor metrics and confusion matrix. The metric values are as follows:

$MAE=221.0249287903704$; $RMSE=280.8131826264988$; $R^2=0.9094993852057965$

• The MAE of 221.0294 indicates that, on average, the model’s predictions of Global Horizontal Irradiance (GHI) deviate from the actual values by about 221.03 Wh/$m^2$
• The RMSE of 280.81318 suggests a moderate level of error when predicting GHI. RMSE is more sensitive to larger errors compared to MAE.
• The $R^2$ value of 0.909499385 indicates that approximately 90.95% of the variance in the GHI data is explained by the model.

Figure 13. Actual vs Predicted GHI & Metrics:An instance of Predictive Analysis

Figure 13. Actual vs Predicted GHI & Metrics:An instance of Predictive Analysis

6.4. Model Coefficients & GHI Estimation

The two GHI model estimations are discussed in this section.

• Multi linear equation based GHI estimation
• Nonlinear equation based GHI estimation

Let,

$$\begin{array}{cc}\hfill \mathrm{GHI}& =\mathrm{Global}\mathrm{Horizontal}\mathrm{Irradiance}\hfill \\ \hfill \mathrm{DNI}& =\mathrm{Direct}\mathrm{Normal}\mathrm{Irradiance}\hfill \\ \hfill \mathrm{DHI}& =\mathrm{Diffuse}\mathrm{Horizontal}\mathrm{Irradiance}\hfill \\ \hfill T& =\mathrm{Air}\mathrm{Temperature}(°\mathrm{C})\hfill \\ \hfill H& =\mathrm{Relative}\mathrm{Humidity}(\%)\hfill \\ \hfill P& =\mathrm{Barometric}\mathrm{Pressure}\left(\mathrm{mB}\right)\hfill \\ \hfill W& =\mathrm{Wind}\mathrm{Speed}(\mathrm{m}/\mathrm{s})\hfill \\ \hfill cos\left(\theta \right)& =\mathrm{Cos}\mathrm{ine}\mathrm{of}\mathrm{the}\mathrm{Solar}\mathrm{Zenith}\mathrm{Angle}\hfill \\ \hfill {\alpha }_0,{\alpha }_1,{\alpha }_2,{\alpha }_3,{\alpha }_4,{\alpha }_5,{\alpha }_6& =\mathrm{Model}\mathrm{coefficients}\hfill \\ \hfill {\beta }_1,{\gamma }_1& =\mathrm{Additional}\mathrm{coefficients}\hfill \end{array}$$

In terms of these parameters the multi linear GHI estimator is Equation (2)

$$\mathrm{GHI}={\beta }_0+{\beta }_1·\mathrm{T}+{\beta }_2·\mathrm{H}+{\beta }_3·\mathrm{W}+{\beta }_4·\mathrm{P}+{\beta }_5·\mathrm{DHI}+{\beta }_6·\mathrm{DNI}$$

(2)

Using the multi linear model coefficients of Table 4 & Equation (2), the GHI (multi linear) is computed. The plot in Figure 15 (a) shows the actual GHI vs the computed GHI over the years. It is observed that the multi linear model, though provides actual GHI values at some instances, it fails to follow the pattern of actual GHI. Arguably, since GHI has complex nonlinear dependency on the meteorological parameters a multi linear equation may be too simple to fit to the model. Therefore, a nonlinear model is required to compute GHI.

Figure 14. Scatter plot of Actual vs Predicted GHI after Non-linear modeling

Figure 14. Scatter plot of Actual vs Predicted GHI after Non-linear modeling

Figure 15. Line graphs of actual and computed GHI under multilinear (a) and Nonlinear (b) modeling

Figure 15. Line graphs of actual and computed GHI under multilinear (a) and Nonlinear (b) modeling

The Equation (3) provides a GHI estimation considering the nonlinear dependency on the meteorological paraders.

$$\mathrm{GHI}={\alpha }_0+{\alpha }_1(\mathrm{DNI}·cos\left(\theta \right))+{\alpha }_2·\mathrm{DHI}+{\alpha }_3·e^{{\beta }_1·\mathrm{T}}+{\alpha }_4·{\mathrm{H}}^2+{\alpha }_5·ln\left(\mathrm{P}\right)+{\alpha }_6·{\mathrm{W}}^{{\gamma }_1}$$

(3)

Figure 14, shows the meteorological model coefficients under nonlinear modeling.

Using these model coefficients and a nonlinear Equation (3), GHI (Nonlinear) is computed. The plot in Figure 15 (b) shows the actual GHI vs the computed GHI over the years. It can be noted that the nonlinear model follows the exact pattern as the actual GHI line. The value difference can be compensated by clamping the parameter ${\alpha }_0$. The plot in Figure 16 (d & e), shows the actual and computed GHI plots under nonlinear modeling. In conclusion of this discussion, Equation (3) is a better fit for predicting GHI.

Therefore, to estimate GHI in Saudi Arabia, the nonlinear Equation (3) with the coefficient values given in Table 5 may be used.

7. Conclusion & Future Work

Abbreviations

References

1. Halliday, D.; Resnick, R.; & Walker, J.Fundamentals of Physics; 10^th Eds.; Wiley & Sons: Danvers, Massachusetts, USA, 2010; pp. 195.
2. REN21:Renewable Energy Now. Available online: URL https://www.ren21.net/reports/global-status-report/ (accessed on 07/10/2024).
3. Renewables. Available online: URL https://www.iea.org/reports/renewables(accessed on 07/10/2024).
4. Adnan Ayaz.;Faraz Ahmad.;Mohammad Abdul Aziz Irfan.;Zabdur Rehman.;Krzysztof Rajski.; and Jan Danielewicz. Comparison of Ground-Based Global Horizontal Irradiance and Direct Normal Irradiance with Satellite-Based SUNY Model. Energies 2022, 15(7), 2528. [CrossRef]
5. Breiman, L. Random Forests. Machine Learning 2001, 45, 5–32. [Google Scholar] [CrossRef]
6. Ali Jassim M Lari. Forecasting and Prediction of Solar Energy Generation using Machine Learning Techniques. Doctor of Philosophy, Swansea University, Wales UK, 20/06/2023.
7. Ewa Chodakowska.; Joanicjusz Nazarko.; Lukasz Nazarko.; and Hesham S. Rabayah. Solar Radiation Forecasting: A Systematic Meta-Review of Current Methods and Emerging Trends. Energies 2024, 17(13), 3156. [CrossRef]
8. Yang D.; Kleissl, J.; Gueymard C. A.; Pedro H. T. C.; & Coimbra C. F. M. History and trends in solar irradiance and PV power forecasting: A preliminary assessment and review using text mining. Solar Energy 2018, 168, 160–101. [CrossRef]
9. Huijun Zhang.; Mingjie Zhang.; Ran Yi.; Yaxin Liu.; Qiuzi Han Wen.; and Xin Meng. Growing Importance of Micro-Meteorology in the New Power System: Review, Analysis and Case Study. Energies 2024, 17(6), 1365. [CrossRef]
10. Chen C.; Liaw A.; & Breiman L. (University of California, Berkeley. USA); 2004.
11. Geurts P.; Ernst D.; & Wehenkel L.Extremely randomized trees. Machne Learning 2006, 63, 3–42. [CrossRef]
12. Lingjun He.; Richard A.; Levine.; Juanjuan Fan.; Joshua Beemer.; Jeanne Stronach. Random Forest as a Predictive Analytics Alternative to Regression in Institutional Research Practical Assessment, Research & Evaluation Journal 2018, 23(1), 1–16. ISSN:1531-7714.
13. Rob Kitchin.The Data Revolution: A Critical Analysis of Big Data, Open Data and Data Infrastructures; 2^nd Eds.; SAGE: Oliver’s Street, London, UK, 2022; pp. 43-75. ISBN:978-1529733754.
14. Arie Purwanto.; Anneke Zuiderwijk.; and Marijn Janssen. Authors Info & Claims "Citizens’ Trust in Open Government Data: A Quantitative Study about the Effects of Data Quality, System Quality and Service Quality. In Proceedings of 21st Annual International Conference on Digital Government Research, Seol, Republic of Korea, (15-19/06/2020); Pages 310–318. [CrossRef]
15. Cong Feng.; Dazhi Yang.; Bri-Mathias Hodge.; Jie Zhang. OpenSolar: Promoting the openness and accessibility of diverse public solar datasets. Solar Energy 2019, 188(1), 1369–1379. [CrossRef]
16. Aneela Zameer.; Fatima Jaffar.; Farah Shahid.; Muhammad Muneeb.; Rizwan Khan.; Rubina Nasir. Short-term solar energy forecasting: Integrated computational intelligence of LSTMs and GRU PLoS ONE 2023, 18(10): e0285410. [CrossRef]
17. Stefan Preda.; Simona-Vasilica Opre.; Adela Bâra.; and Anda Belciu (Velicanu). PV Forecasting Using Support Vector Machine Learning in a Big Data Analytics Context. Symmetry 2018, 10(12), 748. [CrossRef]
18. Giheung-gu.; Yongin-si.; Gyenggi-do. A Two-Step Approach to Solar Power Generation Prediction Based on Weather Data Using Machine Learning. Sustainability 2019, 11(5), 1501. [CrossRef]
19. Andreas C. Müller.; and Sarah Guido. Introduction to Machine Learning with Python.; Oreilly publisher: Gravenstein Highway North, Sebastopol, CA, USA. 2017; pp. 378. ISBN:978-1-449-36941-5.
20. Pedregosa F.; Varoquaux G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel O.; & Duchesnay, E. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 2011, 12, 2825–2830.
21. Annie Leung.; & Janni Leung. (University of Queensland, Queensland. UK). "How to use Python in Power BI? Step-by-step tutorial – a case study of creating a correlation heatmap, 2020. [CrossRef]