The effect of stress on task performance is complex, too much or too little negatively affects performance; and there exists an optimal level of stress to drive optimal performance. Task difficulty and external affective factors are distinct stressors that impact cognitive performance. Neuroimaging studies showed that mood affects working memory performance and the correlates are changes in haemodynamic activity in the prefrontal cortex (PFC). We investigate the interactive effects of affective states and working memory load (WML) on working memory task performance and haemodynamic activity using functional near-infrared spectroscopy (fNIRS) neuroimaging on the PFC of healthy participants. We seek to understand if haemodynamic responses could tell apart workload related stress from situational stress arising from external affective distraction. We found that the haemodynamic changes towards affective stressor and workload related stress were more dominant in the medial and lateral PFC respectively. Our study reveals distinct affective state-dependent modulations of haemodynamic activity with increasing WML in n-back tasks, which correlate with decreasing performance. The influence of negative affect on performance is greater at higher WML, and haemodynamic activity showed evident changes in temporal, and both spatial and strength of activation differently with WML.