Nowadays, text readability is of great importance. Simple but very often ignored, readability statistics can provide information about the level of difficulty of the readability of particular documents and increase an evaluator’s credibility. Hence, this research aims to examine the readability index of the test instrument for BIPA (Bahasa Indonesia Untuk Penutur Asing), evaluate the student's reading ability, and analyze the relevance readability index and cloze test result. The study was carried out through an experiment involving 21 international students in several Muhammadiyah Universities. The students were provided with three sets of reading instruments from BIPA-1 to BIPA-3. Data analysis was carried out with correlation analysis. The result showed that the reading instrument difficulty was at a fairly easy to moderate level. Gunning Fog score and Automated Readability Index were the most relevant tool to the student's test achievement. This research implies the importance of test instrument evaluation. Assessment of the readability of the text is vital in the process of developing an appropriate test instrument.