The Geostationary Environment Monitoring Spectrometer (GEMS) onboard the Geostationary Korean Multi-Purpose Satellite 2B was successfully launched in February 2020. GEMS is a hyperspectral spectrometer measuring solar irradiance and Earth radiance in the range of 300 to 500 nm. This paper introduces the spectral calibration algorithm for GEMS, which uses a nonlinear least-squares approach. To assess the performance of the algorithm, sensitivity tests for a series of spectral parameters such as shift, spectral range for fitting, signal-to-noise ratio, spectral response function (SRF), and reference spectrum have been conducted. To improve the assessment, a synthetic GEMS spectrum using the prelaunch GEMS SRF is adopted here. The test results show that the required accuracy (0.002 nm) is achievable for the expected uncertainties of the parameters except for the SRF and the choice of high-resolution reference spectrum, which degrade the algorithm performance by an order magnitude. To mitigate the sensitivity to SRF, retrieval of in-orbit SRF using an analytic function is suggested. Finally, a few candidates for the high-resolution solar reference spectrum are prepared for testing by the instrument during in-orbit tests.