Disorders of swallowing often lead to pneumonia when material enters the airways (aspiration). Flexible Endoscopic Evaluation of Swallowing (FEES) plays a key role in the diagnostics of aspiration but is prone to human errors. An AI-based tool could facilitate this process. Recent non-endoscopic/non-radiologic attempts to detect aspiration using machine-learning approaches have led to unsatisfying accuracy and show black box characteristics. Hence, for clinical users it is hard to trust in these model decisions. Our aim is to introduce an explainable artificial intelligence (XAI) approach to detect aspiration in FEES. Our approach is to teach the AI about the relevant anatomical structures like the vocal cords and the glottis based on 92 annotated FEES videos. Simultaneously, it is trained to detect bolus that passes the glottis and becomes aspirated. During testing, the AI successfully recognized glottis and vocal cords, but could not yet achieve satisfying aspiration detection quality. Albeit detection performance has to be optimized, our architecture results in a final model that explains its assessment by locating meaningful frames with relevant aspiration events and by highlighting the suspected bolus. In contrast to comparable AI tools, our framework is verifiable, interpretable and therefor accountable for clinical users.