Machine learning (ML) methodologies have been looked upon recently as a potential candidate for mitigating nonlinearity issues in optical communications. In this paper, we experimentally demonstrate a 40-Gb/s 256-quadrature amplitude modulation (QAM) signal-based Radio over Fiber (RoF) system for 50 km of standard single mode fiber length which utilizes support vector machine (SVM) decision method to indicate an effective nonlinearity mitigation. The influence of different impairments in the system is evaluated that includes the influences of Mach-Zehnder Modulator nonlinearities, in-phase and quadrature phase skew of the modulator. By utilizing SVM, the results demonstrated in terms of bit error rate and eye linearity suggest that impairments are significantly reduced and licit input signal power span of 5dBs is enlarged to 15 dBs.