Judging texts with rubrics and comparative judgement: Taking into account reliability and time investment

Subtitle:een afweging van betrouwbaarheid en tijdsinvestering

Writing researchers and practitioners both aim for reliable judgements with a minimum investment of time. This study focuses on two judgement methods, rubrics and comparative judgement. For each method, we studied how long it took to complete a judgement per text. Moreover, we examined how the reliability evolves in relation to the time spent judging. Judges were randomly attributed to the rubrics condition or the comparative judgement condition. In each condition, the same 35 texts were judged and time was tracked during this process. Results show that, when reliability is operationalized as the stability of the rank order, both methods require a comparable time investment to reach a stable rank order. Future research on the reliability and time investment should take into account the time needed for developing the rubric and to set up a comparative judgement assessment. Further research should also clarify whether the findings can be generalized to other texts and rubrics.

Publication year:2017

Keywords:A1 Journal article

BOF-keylabel:yes

BOF-publication weight:0.1

CSS-citation score:1

Authors from:Higher Education

Accessibility:Open