ATP’s Graduate Student Research Award allows students the opportunity to present innovative research with practical applications in assessments among graduate student researchers, and to provide opportunities for graduate researchers to network with leaders in the assessment industry.
ATP is thrilled to announce the recipients of this year’s Graduate Student Research Award. Join this session and learn more about each graduate student’s research.
In remote, low-stakes assessments, examinees’ true performance may be confounded with low motivation. Several analytic methods exist to remove the effect of low effort from observed data by filtering examinees who exhibit low effort based on total test-taking time. Other methods filter responses identified as non-effortful using the effort-moderated IRT model. Both types of methods require the selection of a threshold to separate effortful and non-effortful responses. For comparison, six combinations of methods (examinee-based filtering methods or response-based filtering methods) and thresholds (conservative or liberal) were applied to estimate the score means of examinees assessed remotely in 2021. The score means, estimated six ways, were compared to score means of examinees assessed in person in 2016-2020. The results support the use of motivation filtering regardless of which method is used. However, the selection of specific methods and thresholds may depend on the assessment context.
Speaker: Sarah Alahmadi, James Madison University
Test cheating has a great impact on the test validity of high-stake assessments and is a significant source of concern for many testing companies. While many studies develop statistical methods to identify fraud in testing, the comparison of their applications to empirical data is rarely discussed. The current study was designed to apply four quantitative methods to exam results data and compare their ability to identify known cheaters and exposed items. Preliminary results demonstrated some implications and limitations for current psychometric methods. Further analysis to improve the method performance is discussed.
Speaker: Jiaying Xiao, University of Washington