In this paper, a new approach to retrieve semantic images based on shape and geometric features of image in conjunction with multi-class support vector machine is proposed. Zernike moment as shape feature is to verify the invariance of objects for silhouette image. In addition, a set of geometrical features is to explore the objects shape using two features of rectangularity and circularity. Then the extracted features are normalized and employed for multi-class support vector machine either for learning or retrieving processes. The retrieving process relies on three main tasks which namely Query Engine, Matching Module and Ontology Manger, respectively. Query Engine is to build the input text or image query using SPARQL language. The matching module extracts the shape and geometric features of image’s objects and employ them to Ontology Manger which in turn inserts them in ontology knowledge base. Benchmark mammals have been conducted to empirically conclude the outcome of proposed approach. Our experiment on text and image retrieval yields efficient results to problematic phenomena than previously reported.