Complex diseases such as cancer are caused by changes in the Gene Regulatory Networks. Systems that model the complex dynamics of these networks along with adapting to real gene expression data are closer to reality and can help understand the creation and treatment of cancer. In this paper, for the first time, modelling of gene regulatory networks is performed using delayed nonlinear variable order fractional systems in the state space by a new tool called GENAVOS. This tool uses gene expression time series data to identify and optimize system parameters. This software has several tools for analyzing system dynamics. The results show that the nonlinear variable order fractional systems have very good flexibility in adapting to real data. We found that regulatory networks in cancer cells actually have a larger delay parameter than in normal cells. It is also possible to create chaos, periodic and quasi-periodic oscillations by changing the delay, degradation and synthesis rates. Our findings indicate a profound effect of time-varying order on these networks, which may be related to a type of cellular memory due to epigenetic and environmental factors. We showed that by changing the delay parameter and the variable order function for a normal cell system, its behavior changes and becomes quite similar to the behavior of a cancer cell. This work also confirms the effective role of the miR-17-92 cluster in the cancer cell cycle. GENAVOS is available at https://github.com/hanif-y/GENAVOS with its user guide and MATLAB codes.