High demand of building construction has been taking places in the major city of Malaysia. However, despite this magnificent development, the lack of proper maintenance has caused a large portion of these properties deteriorated over time. The implementation of the project - Automated Detection of Physical Defect via Computer Vision - is a low cost system that helps to inspect the wall condition using Kinect camera. The system is able to classify the types of physical defects -crack and hole - and state its level of severity.The system uses artificial neural network as the image classifier due to its reliability and consistency. The validity of the system is shown using experiments on synthetic and real image data. This automated physical defect detection could detect building defect early, quickly, and easily, which results in cost saving and extending building life span.