Bayesian treatment of non-standard problems in test analysis

Abstract

This paper extends the methods of [10] in an attempt to handle non-standard problems in test analysis. The approach is based on a Bayesian framework where test characteristics are treated as random parameters for which posterior probability assessments are available. The generality of the approach permits straightforward analyses of problems that may be difficult using standard classical test theory and standard item response theory. We first illustrate the methods on aviation test scores where the test outcomes are not dichotomous (i.e. correct and incorrect responses). Instead, the approach is modified to handle questions with answers on a five-point ordinal scale. The second problem addresses the complication of the assessment of instructors in addition to the assessment of test questions and students.