Speakers
Description
In performance-based language assessment, consistency is challenging because rating is interpretive; scores reflect raters’ severity, scale use, and the aspects of performance they attend to. Rater training can support consistent rubric application in line with test developers’ intentions. However, it is often difficult to determine which aspects of training influenced raters’ decision-making. In this study, we used Many-Facet Rasch Measurement (MFRM) to guide follow-up interviews in a novel approach to rater training program evaluation.
Abstract section 1: Relevance
Without a follow-up mechanism to determine why raters’ scoring behavior changes as a result of training, improvements to a training program might be limited or reduced to guesswork. In the current study, we used MFRM-identified rater patterns to select raters for follow-up interviews and to tailor training follow-up questions to each rater’s statistical profile, allowing observed statistical change to be examined in relation to raters’ own accounts of how they engaged with the training. This approach made it possible to distinguish between surface-level statistical alignment and deeper shifts in how raters understood and applied the assessment criteria.
Abstract section 2: Contribution/research questions
We aimed to contribute develop a structured approach through which language researchers and program administrators can use a combination of MFRM analysis and follow-up interviews to improve rater training programs and, in turn, performance-based language assessment.
1. Can MFRM analysis be used to identify raters with unique insight as to how training programs might be improved?
2. How can statistically identified changes in rater behavior following training be explained through rater-specific accounts of training uptake?
Abstract section 3: Content/method
Ten raters scored the spoken performances of 12 university students using a 5-level analytic rubric. The raters scored the performances before and after completing a training program centered on the target construct. MFRM was used to identify changes in rater severity, fit, and scoring patterns. Based on these results, six raters with differing profiles were selected for interviews designed to clarify their interpretation of the rubric and their engagement with the training.
Abstract section 4: Outcomes/results
MFRM analysis showed differential rater change post-training: some raters demonstrated severity shifts or changes in fit, while others showed minimal change. Interview accounts with some raters confirmed these patterns in that engagement with the training reflected changes identified in the MFRM results (e.g., raters associated with post-test leniency shifts or reduced underfit described corresponding changes in their approach). However, not all accounts matched as predicted, including cases where raters expressed uncertainty about whether training had helped despite measurable shifts in their post-training patterns. Overall, the combination of MFRM and follow-up interviews appears useful for training program evaluation and improvement.
| Title | Examining Rater Behavior Using MFRM-Informed Interviews |
|---|---|
| Teaching Context | College and university education |