Medical ontology has become an important element to enrich semantic description in medical images. The enrichment process can be done with medical image annotation where label or keyword can be added into the image description. Therefore, in this paper, we have proposed an annotation model that contains two components i.e. image processing and annotation component in order to annotate chest X-ray images with relevant medical ontology concepts. The first component helps transforming the original medical image into six regions of interest for the lung area while the second component prepares medical ontology concepts to annotate the derived regions. By annotating the chest X-ray images with medical ontology concepts, we hope to obtain better and accurate image retrieval.