Automated medical diagnosis is one of the important machine learning applications in the domain of healthcare. In this regard, most of the approaches primarily focus on optimizing the accuracy of classification models. In this research, we argue that unlike general purpose classification problems, medical applications require special treatment. In case of medical diagnosis, apart from model performance other factors such as cost of data acquisition may also be taken into account. Since, models which are relatively expensive to operate would have diminished applicability in the field. Therefore, both performance and cost factors may be considered for designing the automated diagnosis solutions. In this regard, we have proposed two ensemble based cost-sensitive feature ranking techniques which tend to select an optimal feature subset by evaluating the cost-benefit trade-off on benchmarked chronic kidney disease dataset. We have also addressed the issue of automatic threshold selection which is generally faced by feature ranking approaches; hence enhancing the overall applicability of the solution. In this research, the main focus is on the application of decision-tree based classifiers in a cost-effective manner. As a case study, we have selected chronic kidney disease as the application of interest in which we demonstrate that our proposed approaches select features which are both useful and cost-effective. Furthermore, the proposed approaches are also evaluated against a number of comparative techniques and the results are promising.