Classification accuracy of diagnostic methods: A simulation study

The objectives of this Monte Carlo simulation study were two-fold. One objective was to investigate classification accuracy of selected diagnostic methods ̶ Wainer augmented method (AUG), the multidimensional three-parameter logistic model (M3PL), and the log-linear cognitive diagnosis model (LCDM) under various conditions. The second objective is to evaluate whether the LCDM parameter estimates are population invariant. Subscale length, attribute correlations, and average examinee ability were systematically manipulated. The results suggested that AUG and M3PL performed similarly with regards to classification accuracy in most cases. LCDM had lower percent agreement, kappa, and sensitivity when compared to AUG or M3PL, especially when the examinee abilities were higher than what the items targeted. However, LCDM had better specificity when the examinee abilities were higher than the items targeted. The results of this study also suggested that the LCDM estimates may not be population invariant. In conclusion, based on this study, AUG may be advised over M3PL or LCDM in practice where the conditions are similar to those simulated by this study due to both its classification accuracy performance and its ease of implementation.