The research field of energy system analysis is dealing with increasingly complex energy systems and their respective challenges. Moreover, the requirement for open science has become a focal point of public interest. Both drivers have triggered the development of a broad range of (open) energy models and frameworks in recent years. However, there are hardly any approaches on how to evaluate these tools in terms of their capabilities to tackle energy system modelling challenges. This paper describes a first step towards a flexible evaluation of software to model energy systems. We propose a qualitative approach as an useful supplementary to existing model fact sheets and transparency checklists. We demonstrate the applicability by evaluating the newly developed “Open Energy Modelling Framework” with respect to existing challenges in energy system modelling. The case study results highlight that challenges related to complexity and scientific standards can be tackled to a large extent while the challenges of model utilization and interdisciplinary modelling are only tackled partially. However, the challenge of uncertainty remains for the most part unaddressed at present. Advantages of the evaluation approach lie in its simplicity, flexibility and transferability to other tools. Disadvantages mostly stem from its qualitative nature. Our analysis reveals that some challenges in the field of energy system modelling cannot be addressed by a software as they are on meta level like model result communication and interdisciplinary modelling.